Classroom audio analysis and identification method based on two-stage adaptive clustering

Abstract

The invention discloses a classroom audio analysis and identification method based on dual-stage adaptive clustering, and the method comprises the following steps: S10, carrying out audio collection and preprocessing: carrying out the processing of an original classroom record, and outputting a standardized audio waveform and audio quality parameters; s20, voice detection is carried out; s30, performing embedding extraction: for each voice segment, outputting a 64-dimensional uniform embedding vector embedding of each voice segment; s40, performing first-stage clustering: taking embedding vectors of all events and segment-level energy signal-to-noise ratio data as input, and outputting a corresponding human voice group index and an environment voice group index; s50, carrying out second-stage clustering: carrying out further dimensionality reduction on an embedding vector of the human voice group and a corresponding voice segment, and outputting human voice fine-grained cluster tags for distinguishing different human voices for subsequent role recognition and event-level combination; s60, performing role identification, and outputting role allocation results of teachers and students; and S70, performing event fusion and result output.

Description

Technical Field

[0001] This invention belongs to the field of audio signal processing technology, specifically relating to a classroom audio analysis and recognition method based on two-stage adaptive clustering. Background Technology

[0002] With the development of educational informatization, classroom recording and intelligent teaching analysis have become important means of evaluating teaching quality. Currently, classroom audio analysis usually relies on manual transcription or the use of traditional speech recognition systems to extract teaching speech information. However, in real classroom environments, due to complex factors such as multiple speakers, noise interference, echoes, and equipment differences, existing methods have low accuracy in speaker separation, role recognition (teacher / student differentiation), and event classification.

[0003] Specifically, existing classroom speech processing mainly suffers from the following technical problems: (1) Insufficient ability to distinguish between speech and ambient sound: Existing classroom audio analysis systems are mostly based on fixed threshold speech activity detection or single-stage clustering algorithms, which cannot accurately identify different types of audio events, and are prone to misclassification and omission, leading to distortion in subsequent analysis. (2) Insufficient clustering stability and robustness: Traditional methods usually use static clustering parameters, which significantly reduce performance under different signal-to-noise ratios, microphone distances, or multi-person discussion scenarios, resulting in unstable clustering results and difficulty in accurately distinguishing the speaking roles of teachers and multiple students. (3) Lack of dynamic adaptive mechanism: Existing technologies fail to adaptively adjust clustering strategies according to audio content features, event density, and speaker distribution, resulting in low recognition accuracy in complex scenarios with multiple background sound sources. (4) Inability to achieve intelligent analysis at the classroom behavior level: Traditional systems mostly remain at the low-level speech detection or simple speaker separation stage, lacking the ability to perform high-level semantic modeling of specific classroom teaching behaviors, which limits their application value in education quality assessment. (5) Poor generalization and deployability of algorithms: Some deep learning-driven source separation algorithms perform well under laboratory conditions, but rely on a large amount of labeled data for training. Their performance drops significantly after being transferred to real classroom scenarios, and they have high hardware computing power requirements, making them unsuitable for independent operation on teaching terminals or school servers. Summary of the Invention

[0004] In view of the above problems, this invention provides a classroom audio analysis and recognition method based on two-stage adaptive clustering. By introducing speech activity detection and event-level embedding and density adaptive clustering, it realizes automatic analysis and recognition of events such as teacher and student speech and noise in classroom audio. It is suitable for automatic analysis of classroom audio and intelligent identification of people in educational scenarios.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A classroom audio analysis and recognition method based on two-stage adaptive clustering includes the following steps: S10 performs audio acquisition and preprocessing: it processes the original classroom recordings by including uniform sampling rate, mono conversion, and lightweight noise reduction, and performs amplitude normalization and short-time energy statistics to obtain the quality level of the audio segment, and then outputs standardized audio waveforms and audio quality parameters. S20, Perform speech detection: Receive the preprocessed normalized audio waveform and frame features, input them into a pyannote.audio neural network, automatically construct speech segments based on the frame intervals with consecutive 1s in the mask, generate start and end timestamps for each segment, and output a frame-level speech confidence sequence and a binary speech mask obtained by thresholding it; or use the DummyVAD speech activity detection model to adaptively filter effective speech segments based on multiple indicators including energy and spectral features, and output a list of start and end timestamps for speech segments and corresponding metadata including segment-level energy and signal-to-noise ratio; S30, perform embedding extraction: For each speech segment, read the metadata of its audio sample's energy and signal-to-noise ratio, and perform short-time spectral subtraction and normalization on the segment signal; on this basis, extract the self-supervised embedding and fuse it with the representation based on traditional acoustic features; then, use principal component analysis (PCA) to reduce the dimensionality of the fused vector to 64 dimensions and perform L2 normalization on the dimensionality-reduced vector to output a 64-dimensional unified embedding vector for each speech segment; S40, Perform the first stage of clustering: Take the embedding vectors of all events and the segment-level energy signal-to-noise ratio data as input, use the HDBSCAN clustering algorithm to cluster the events, and make a comprehensive judgment on the spectral distribution, energy statistics and signal-to-noise ratio characteristics of the cluster centers at the cluster level to divide the events into human voice groups and ambient sound groups, and output the corresponding human voice group index and ambient sound group index to provide initial grouping constraints for the subsequent fine-grained clustering and role recognition stages; S50, perform the second stage of clustering: further reduce the dimensionality of the embedding vector of the human voice group and the corresponding speech segment, and then use the adaptive KMeans or hierarchical clustering algorithm for the human voice group events. Under the constraints of internal evaluation indicators such as silhouette coefficient or heuristic rules, the number of clusters k is automatically selected, and fine-grained human voice cluster labels are output to distinguish different human voices for subsequent role recognition and event-level merging. S60, perform role recognition: using the second-stage clustering results and segment-level statistical features, calculate the total speaking time, average length and speaking frequency of each voice cluster, construct a weighted behavior scoring model to automatically identify the teacher cluster, and number the remaining voice clusters as students according to the order of behavior characteristics, and output the role allocation results of teachers and students. S70 performs event fusion and outputs results: the voice event sequences with completed role annotations are arranged in chronological order, adjacent voice segments with the same role are merged, abnormal short segments are filtered out, smooth temporal voice events are generated, a time-series table after role separation is obtained, and corresponding audio slices are output as needed for subsequent analysis and application.

[0006] In one possible implementation, S10 specifically includes: Read in the original classroom recording and standardize the sampling rate: audio_resampled = resample(input_audio, target_sr); The sampling rate used, audio_resampled, is calculated as the number of samples per second using the resample function and the target sampling rate, target_sr. Then, it is converted to mono: audio_mono = mean(audio_resampled, axis=channels); The `audio_mono` parameter represents the processed result, and `channels` is the number of channels in the existing audio. Next, primary denoising is performed using a lightweight spectral subtraction or noisereduce library. First, the stillness segment is estimated, and the frame with the lowest energy within a certain time frame is used as the noise spectrum estimation benchmark. Amplitude normalization, short-time energy, root mean square energy (RMS), and zero-crossing rate (ZCR) are then performed on the audio signal for subsequent adaptive threshold estimation. Finally, audio quality is estimated, including signal-to-noise ratio (SNR) estimation and spectral entropy calculation, outputting the audio quality level `quality`, with values ​​ranging from `quality∈{low,medium,high}`. Here, `low` indicates that the SNR is below the set lower limit, and the spectral entropy or zero-crossing rate is abnormal, indicating severe ambient noise or significant signal distortion. `medium` indicates that the SNR and other indicators are in the normal middle range, with moderate recording quality that meets most subsequent analysis needs. `high` indicates that the SNR is significantly higher than the threshold, the spectral entropy is low, and the signal is clear and stable, indicating good recording quality suitable for fine-grained speech processing. The formula for amplitude normalization of the input waveform is as follows: ; This represents the instantaneous amplitude value of the original audio signal at time t. This represents the maximum absolute value of the original audio signal. It uses tiny positive numbers to prevent the denominator from being 0, and the amplitude normalization processing formula is used to eliminate volume differences caused by different recording devices and speaking distances.

[0007] In a possible implementation manner, in step S20, the enhanced DummyVAD is used to screen out the effective speech segments, which includes performing adaptive threshold calculation: DummyVAD uses a comprehensive judgment that combines the feature joint determination of the adaptive threshold, the lower bound of the zero-crossing rate, and the upper bound of the spectral entropy. The formula for the adaptive threshold is as follows: vad_threshold = max(global_threshold_config, percentile(RMS, 25)) Where, vad_threshold is the core threshold for voice activity detection; global_threshold_config refers to the global energy threshold, with a range between 0.01 and 0.1; percentile(RMS, 25) refers to the 25th percentile of the root mean square energy RMS sequence of the audio signal, which reflects the "lower quartile" level of the RMS distribution and is used to capture the statistical characteristics of the signal energy, helping to distinguish weak speech from noise.

[0008] In a possible implementation manner, it further includes multi-feature joint determination, which is used to introduce two constraints of the zero-crossing rate and spectral entropy on the basis of the energy threshold, and jointly define the voice active frames. The specific expression is: mask=(RMS>vad_threshold)&(ZCR>zcr_min)&(spec_entropy<spec_max); Where, vad_threshold is the core threshold for voice activity detection, RMS is the root mean square of the audio signal, and RMS>vad_threshold is used to screen out the frames with strong enough energy; the zero-crossing rate ZCR reflects the activity degree of the audio frequency components, the range of zcr_min is 0.005 - 0.02, and ZCR>zcr_min means that the zero-crossing rate is required to be higher than the minimum value; spec_entropy is the spectral entropy, and spec_entropy<spec_max means screening out the spectral entropy lower than the maximum value, which is more inclined to the low spectral entropy feature of speech.

[0009] In a possible implementation manner, it further includes speech segment construction and segment-level filtering: the frames with continuous mask = 1 are merged into one segment: calculate the segment duration, average energy, and segment-level signal-to-noise ratio seg_snr; then apply two filtering conditions including minimum segment length filtering and minimum signal-to-noise ratio filtering to filter the speech segments: The minimum segment length filtering is to discard if seg_duration<min_speech_sec where seg_duration is the length of the speech segment, min_speech_sec is the minimum length of the speech segment, and min_speech_sec is automatically adjusted according to the audio quality and set in the AudioProcessor, generally ranging from 0.3 to 1.0 seconds; The minimum signal-to-noise ratio filtering is to discard if seg_snr < snr_min. where seg_snr is the signal-to-noise ratio of the speech segment, snr_min is the minimum signal-to-noise ratio, ranging from 3 to 10 dB, and it can vary with the quality level.

[0010] In a possible implementation, it further includes boundary smoothing processing: using a short-time cross-window median filter and a 20 - 50 ms boundary extension at the speech segment boundary, outputting a list of preliminary speech / silence segments with timestamps, and attaching the start and end sampling positions, average energy, estimated signal-to-noise ratio, and number of frames at each speech segment level.

[0011] In a possible implementation, S30 includes: Using the start and end points at the audio segment level output in S10, directly cutting out the original waveform corresponding to each speech event from the normalized audio and the speech segment-level attributes corresponding to each speech event, including average energy, duration, and estimated signal-to-noise ratio, performing short-time spectral subtraction on a single segment of audio, and retaining parameters such as the FFT size fft_size and the noise reduction ratio prop_decrease; Performing segment amplitude normalization to ensure consistent amplitude scales. The calculation method is as follows: seg = seg / (max_abs(seg) + ) Outputting the feature seg, which is obtained by adding a minimum value to the absolute maximum value max_abs(seg) of the calculated feature to avoid a minimum value for division by zero and ensure consistent amplitude scales; Performing self-supervised embedding feature calculation: calling a self-supervised embedding model and returning a vector with an output dimension between 512 and 1024; Performing traditional feature supplementation: If the embedding is unstable, that is, the similarity fluctuation between consecutive speech segment embedding vectors exceeds the threshold, or the signal-to-noise ratio of the audio segment is low, that is, the segment signal-to-noise ratio is lower than the set threshold, then use the fallback method to supplement traditional acoustic features: calculate the MFCC mean and variance MFCC-stat, spectral centroid, ZCR, RMS-stat, and form an auxiliary feature vector; Fusion is performed, and the fusion strategy is as follows: The participation ratio of traditional features is adaptively determined according to the segment-level SNR. The fusion weight α is set, and its range is between 0 and 1. If it is weak speech with a low SNR, that is, the segment signal-to-noise ratio is lower than the set threshold, then α is increased to strengthen the role of traditional features; if it is clear speech with a high SNR, then α is decreased to focus on self-supervised embeddings. Dimensionality reduction and normalization are performed: First, a normalization operation is performed on the fused vector, and then it is reduced to 64 dimensions using PCA. An L2 normalization operation is performed on the vector, and the embedding representing each segment with 64 dimensions is output, along with metadata such as the original vector dimension, energy, signal-to-noise ratio, and duration.

[0012] In a possible implementation manner, the S40 specifically includes: For the low-dimensional embeddings obtained in the embedding extraction of all events, the corresponding segment-level metadata includes energy, signal-to-noise ratio, and length. First, the density clustering algorithm HDBSCAN is used to perform density clustering on the embeddings. The HDBSCAN algorithm outputs cluster_labels and cluster stability, where -1 in cluster_labels represents noise, and the cluster stability is represented by cluster_probabilities; the statistical features of each cluster are used to calculate the metrics including the mean cepstral coefficient MFCC, mean spectral centroid, mean zero-crossing rate, and speech ratio within the cluster. According to the threshold, the cluster category is judged. If mean_spectral_centroid < thr_sc and mean_zcr < thr_zcr and speech_ratio > thr_sr, then it is determined as the human voice cluster; otherwise, it is determined as the environmental sound cluster; for the clusters determined as environmental sounds, they are marked and excluded from the next stage or refined separately, and two types of partition indices including the segment indices corresponding to the human voice cluster and the segment indices of the environmental sound cluster are output.

[0013] In a possible implementation manner, the S50 specifically includes: Based on the first stage processing that identifies the human voice portion, the input includes the embeddings of the human voice group and the signal-to-noise ratio, energy, and length information of the human voice segment metadata. For cases where the human voice group characteristics are more complex and the distribution is more similar, PCA is first performed on the human voice group for further dimensionality reduction to improve stability. If HDBSCAN has already generated multiple clusters in the human voice group and the cluster quality score exceeds the threshold, the HDBSCAN algorithm clustering result is used directly; otherwise, adaptive KMeans clustering is used: that is, the optimal number of clusters k is automatically selected based on the silhouette score: by searching k∈[2, min(max_clusters, N-1)] through the silhouette coefficient, the largest silhouette score is selected. If the number of samples is small or the silhouette has no obvious peak, a heuristic k based on speech turns is used, such as setting k equal to the square root of the number of speech segments and rounding up to reduce cluster label jumps. The clustering result is optimized for temporal smoothing techniques such as neighborhood majority voting and simple linear chain conditional random fields, and the output is the final human voice cluster number with smoothing.

[0014] In one possible implementation, S60 specifically includes: The voice segment clusters after the two-stage clustering were statistically analyzed and identified. For each cluster, information including cumulative speaking time, first and last appearance times, number of times spoken, and average speaking length was collected. Weighted behavioral features were used to identify teacher roles. score=w 1* duration+w 2* turns+w 3* avg_len `duration` represents the speaking time, `turns` represents the number of times a speaker speaks, `avg_len` represents the average speaking length, and `w1`, `w2`, and `w3` represent their respective weights, each ranging from 0.2 to 0.6. The determination strategy is as follows: if the cumulative duration difference, i.e., the difference in cumulative speaking time among multiple speakers exceeds 40%, then the cluster with the longest duration is Teacher; if the difference is not significant, then a composite feature of "turns plus avg_len" is introduced; the remaining clusters are labeled as Student1, Student2, ... Student in order of appearance. n .

[0015] The present invention offers the following advantages: by introducing adaptive thresholding, event-level embedding fusion, and a two-stage clustering mechanism, it effectively solves the problems of unstable clustering and inaccurate role recognition in existing technologies. Specifically, it includes the following aspects: (1) Adaptive VAD with multi-feature fusion: It adopts a multi-dimensional judgment mechanism of RMS, zero-crossing rate and spectral entropy, and combines the signal-to-noise ratio to dynamically adjust the threshold, so as to achieve higher speech segment detection accuracy than the single threshold method. (2) Event-level audio embedding fusion: Combining self-supervised speech embedding with traditional statistical features such as MFCC and spectral centroid, and dynamically adjusting the fusion weights according to the segment-level SNR, significantly improving the clustering discrimination under weak sound segments and noisy conditions; (3) Two-stage adaptive clustering: Through the two-stage mechanism of "density clustering algorithm (HDBSCAN) + adaptive KMeans", human voice and ambient sound are first distinguished, and then teachers and students are distinguished in fine granularity in the subspace, so as to realize automatic determination of cluster number and noise removal; (4) Statistical role determination strategy: Based on the clustering results, the total speaking time, the number of speaking times and the average speaking time are comprehensively considered to dynamically infer the teacher's role and effectively avoid misjudgment caused by a single duration criterion; (5) Cross-device robustness: Through energy standardization and audio quality assessment mechanisms, the system maintains consistent processing performance under different recording devices and recording conditions; (6) Scalability and embedded adaptation: The system can replace PyAnnote VAD with lightweight models such as Silero, and can be deployed in teaching recording equipment or embedded analysis systems, with good versatility and scalability.

[0016] (7) Experimental verification shows that the present invention improves the accuracy of teacher / student speech separation by about 15%–25% in typical classroom scenarios compared with traditional solutions, and still maintains a clustering stability of more than 0.45 in noisy scenarios. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the steps of a classroom audio analysis and recognition method based on two-stage adaptive clustering, according to an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Reference Figure 1 The diagram shows a flowchart of a classroom audio analysis and recognition method based on two-stage adaptive clustering according to an embodiment of the present invention, including the following steps: S10 performs audio acquisition and preprocessing: the original classroom recording, which can be in WAV or MP3 format, is processed including uniform sampling rate, mono conversion, lightweight noise reduction, amplitude normalization and short-time energy statistics to obtain the quality level of the audio segment, and then outputs standardized audio waveforms and audio quality parameters. The above processing can reduce the impact of equipment and distance differences.

[0020] S20, Perform speech detection: Receive the preprocessed normalized audio waveform and frame features, input them into a pyannote.audio-based neural network, automatically construct speech segments based on the frame intervals with consecutive 1s in the mask, generate start and end timestamps for each segment, and output a frame-level speech confidence sequence and a binary speech mask obtained by thresholding it; or utilize an enhanced speech activity detection DummyVAD model to adaptively filter effective speech segments based on multiple indicators including energy and spectral features, and output a list of start and end timestamps for speech segments and corresponding metadata including segment-level energy and signal-to-noise ratio; S30, perform embedding extraction: For each speech segment, read the metadata of its audio sample's energy and signal-to-noise ratio, and perform short-time spectral subtraction and normalization on the segment signal; on this basis, extract the self-supervised embedding and fuse it with the representation based on traditional acoustic features; then, use principal component analysis (PCA) to reduce the dimensionality of the fused vector to 64 dimensions and perform L2 normalization on the dimensionality-reduced vector to output a 64-dimensional unified embedding vector for each speech segment; S40, Perform the first stage of clustering: Take the embedding vectors of all events and the segment-level energy signal-to-noise ratio data as input, use the HDBSCAN clustering algorithm to cluster the events, and make a comprehensive judgment on the spectral distribution, energy statistics and signal-to-noise ratio characteristics of the cluster centers at the cluster level to divide the events into human voice groups and ambient sound groups, and output the corresponding human voice group index and ambient sound group index to provide initial grouping constraints for the subsequent fine-grained clustering and role recognition stages; S50, perform the second stage of clustering: further reduce the dimensionality of the embedding vector of the human voice group and the corresponding speech segment, and then use the adaptive KMeans or hierarchical clustering algorithm for the human voice group events. Under the constraints of internal evaluation indicators such as silhouette coefficient or heuristic rules, the number of clusters k is automatically selected, and fine-grained human voice cluster labels are output to distinguish different human voices for subsequent role recognition and event-level merging. S60, perform role recognition: using the second-stage clustering results and segment-level statistical features, calculate the total speaking time, average length and speaking frequency of each voice cluster, construct a weighted behavior scoring model to automatically identify the teacher cluster, and number the remaining voice clusters as students according to the order of behavior characteristics, and output the role allocation results of teachers and students. S70 performs event fusion and outputs results: the voice event sequences with completed role annotations are arranged in chronological order, adjacent voice segments with the same role are merged, abnormal short segments are filtered out, smooth temporal voice events are generated, a time-series table after role separation is obtained, and corresponding audio slices are output as needed for subsequent analysis and application.

[0021] In a specific application example, S10 specifically includes: Read in the original classroom recording and standardize the sampling rate: audio_resampled = resample(input_audio, target_sr) The sampling rate used, audio_resampled, is calculated as the number of samples per second using the resample function and the target sampling rate, target_sr. Then, it is converted to mono: audio_mono = mean(audio_resampled, axis=channels) The audio_mono parameter represents the processed result, and channels is the number of channels in the existing audio. Next, primary denoising is performed using a lightweight spectral subtraction or noisereduce library. First, the quiet segment is estimated by taking the frame with the lowest energy within a certain time frame at the beginning, such as the frame with the lowest energy within the first 0.5 seconds, as the noise spectrum estimation benchmark. Amplitude normalization, short-time energy, root mean square energy (RMS), and zero-crossing rate (ZCR) are then performed on the audio signal for subsequent adaptive threshold estimation. Finally, audio quality is estimated, including signal-to-noise ratio (SNR) estimation and spectral entropy calculation, outputting an audio quality level, with values ​​ranging from low to medium to high. Low indicates that the SNR is below the set lower limit, and the spectral entropy or zero-crossing rate is abnormal, indicating severe ambient noise or significant signal distortion. Medium indicates that the SNR and other indicators are in the normal middle range, with moderate recording quality that meets most subsequent analysis needs. High indicates that the SNR is significantly higher than the threshold, the spectral entropy is low, and the signal is clear and stable, indicating good recording quality suitable for fine-grained speech processing. The formula for amplitude normalization of the input waveform is as follows:

[0022] This represents the instantaneous amplitude value of the original audio signal at time t. This represents the maximum absolute value of the original audio signal. It is a tiny positive number to prevent the denominator from being 0. The amplitude normalization processing formula is used to eliminate the volume difference caused by different recording devices and speaking distances.

[0023] In a specific application example, in S20, an enhanced DummyVAD is used to screen out the effective speech segments, including performing adaptive threshold calculation: DummyVAD uses a comprehensive judgment that combines feature joint determination of the adaptive threshold, the lower bound of the ZCR, and the upper bound of the spectral entropy. Among them, the adaptive threshold calculation formula is as follows: vad_threshold = max(global_threshold_config, percentile(RMS, 25)) Among them, vad_threshold is the core threshold of voice activity detection; global_threshold_config refers to the global energy threshold, with a range between 0.01 and 0.1; percentile(RMS, 25) refers to the 25% quantile of the root mean square energy RMS sequence of the audio signal, reflecting the "lower quartile" level of the RMS distribution, used to capture the statistical characteristics of the signal energy, and helps to distinguish weak speech from noise.

[0024] Based on the adaptive threshold calculation, it further includes multi-feature joint determination, which is used to introduce two constraints of the zero crossing rate and spectral entropy on the basis of the energy threshold, and jointly define the voice active frames. The specific expression is: mask=(RMS>vad_threshold)&(ZCR>zcr_min)&(spec_entropy<spec_max); Among them, vad_threshold is the core threshold of voice activity detection, RMS is the root mean square of the audio signal, and RMS>vad_threshold is used to screen frames with strong enough energy; the zero crossing rate ZCR reflects the activity degree of the audio frequency components, and the range of zcr_min is 0.005 - 0.02, and ZCR>zcr_min means that the zero crossing rate is required to be higher than the minimum value; spec_entropy is the spectral entropy, and spec_entropy<spec_max means screening the spectral entropy lower than the maximum value, which is more inclined to the low spectral entropy feature of speech.

[0025] Based on the multi-feature joint determination, it further includes speech segment construction and segment-level filtering: frames with continuous mask = 1 are merged into one segment: calculate the segment duration, average energy, and segment-level signal-to-noise ratio seg_snr; then apply two filtering conditions including minimum segment length filtering and minimum signal-to-noise ratio filtering to filter the speech segments: The minimum segment length filtering is to discard if seg_duration<min_speech_sec where seg_duration is the length of the speech segment, min_speech_sec is the minimum length of the speech segment, and min_speech_sec is automatically adjusted according to the audio quality and set in AudioProcessor, generally ranging from 0.3 to 1.0 seconds; The minimum signal-to-noise ratio filtering is to discard if seg_snr < snr_min. where seg_snr is the signal-to-noise ratio of the speech segment, snr_min is the minimum signal-to-noise ratio, ranging from 3 to 10 dB, which can vary with the quality level.

[0026] On the basis of speech segment construction and segment-level filtering, it further includes boundary smoothing processing: using a short-time cross-window median filter and a 20 - 50 ms boundary extension at the speech segment boundary, outputting a preliminary list of speech / silence segments with timestamps, and attaching the start and end sampling positions, average energy, estimated signal-to-noise ratio, and number of frames at each speech segment level.

[0027] In a specific application example, S30 includes: Using the start and end points at the audio segment level output in S10, directly cutting out the original waveform corresponding to each speech event from the normalized audio and the attributes at the speech segment level corresponding to each speech event, including average energy, duration, and estimated signal-to-noise ratio, performing short-time spectral subtraction on single-segment audio, and retaining parameters such as the FFT size fft_size and the noise reduction ratio prop_decrease; Performing segment amplitude normalization to ensure consistent amplitude scales. The calculation method is as follows: seg = seg / (max_abs(seg) + ) Outputting the feature seg, which is obtained by adding a minimum value to the absolute maximum value max_abs(seg) of the calculated feature to avoid division by zero, ensuring consistent amplitude scales; Performing self-supervised embedding feature calculation: calling a self-supervised embedding model and returning a vector with an output dimension between 512 and 1024; Performing traditional feature supplementation: If the embedding is unstable, that is, the similarity between consecutive speech segment embedding vectors fluctuates beyond the threshold, or the signal-to-noise ratio of the audio segment is low, that is, the segment signal-to-noise ratio is lower than the set threshold, then use the fallback method to supplement traditional acoustic features: calculate the MFCC mean and variance MFCC-stat, spectral centroid, ZCR, RMS-stat, and form an auxiliary feature vector; Fusion is performed, and the fusion strategy is as follows: The participation ratio of traditional features is adaptively determined according to the segment-level SNR. The fusion weight α is set, and its range is between 0 and 1. If it is weak speech with a low SNR, that is, the segment signal-to-noise ratio is lower than the set threshold, then α is increased to strengthen the role of traditional features; if it is clear speech with a high SNR, then α is decreased to focus on self-supervised embeddings. Dimensionality reduction and normalization are performed: First, a normalization operation is performed on the fused vector, and then it is reduced to 64 dimensions using PCA. An L2 normalization operation is performed on the vector, and the embedding of the 64-dimensional vector representation of each segment is output, as well as the corresponding original vector dimensions, energy, signal-to-noise ratio, and duration metadata, that is, metadata (orig_dim, energy, snr, length).

[0028] In a specific application example, S40 specifically includes: For the low-dimensional embeddings obtained in the embedding extraction of all events, the corresponding segment-level metadata includes energy, signal-to-noise ratio, and length. First, the density clustering algorithm HDBSCAN is used to perform density clustering on the embeddings. For example, the minimum number of clusters is set to 3 (min_cluster_size = 3), and the minimum number of samples is 1 (min_samples = 1). The HDBSCAN algorithm outputs cluster_labels and cluster stability. Among them, -1 in cluster_labels represents noise, and the cluster stability is represented by cluster_probabilities; the statistical features of each cluster are used to calculate the indicators including the mean cepstral coefficient MFCC, mean spectral centroid, mean zero-crossing rate, and speech ratio within the cluster. According to the threshold, the cluster category is judged. If mean_spectral_centroid < thr_sc and mean_zcr < thr_zcr and speech_ratio > thr_sr, it is judged as a human voice cluster; otherwise, it is judged as an environmental sound cluster; for the clusters judged as environmental sounds, they are marked and excluded or separately refined in the next stage, and two types of partition indexes including the segment indexes corresponding to the human voice clusters and the segment indexes of the environmental sound clusters are output.

[0029] In a specific application example, S50 specifically includes: Based on the first stage processing that identifies the human voice portion, the input includes the embeddings of the human voice group and the signal-to-noise ratio, energy, and length information of the human voice segment metadata. For cases where the human voice group characteristics are more complex and the distribution is more similar, PCA is first performed on the human voice group for further dimensionality reduction to improve stability. If HDBSCAN has already generated multiple clusters in the human voice group and the cluster quality score exceeds the threshold, the HDBSCAN algorithm clustering result is used directly; otherwise, adaptive KMeans clustering is used: that is, the optimal number of clusters k is automatically selected based on the silhouette score: by searching k∈[2, min(max_clusters, N-1)] through the silhouette coefficient, the largest silhouette score is selected. If the number of samples is small or the silhouette has no obvious peak, a heuristic k based on speech turns is used, such as setting k equal to the square root of the number of speech segments and rounding up to reduce cluster label jumps. The clustering result is optimized for temporal smoothing techniques such as neighborhood majority voting and simple linear chain conditional random fields, and the output is the final human voice cluster number with smoothing.

[0030] In a specific application example, S60 specifically includes: The voice segment clusters after the two-stage clustering were statistically analyzed and identified. For each cluster, information including cumulative speaking time, first and last appearance times, number of times spoken, and average speaking length was collected. Weighted behavioral features were used to identify teacher roles. score=w 1* duration+w 2* turns+w 3* avg_len `duration` represents the speaking time, `turns` represents the number of times a speaker speaks, `avg_len` represents the average speaking length, and `w1`, `w2`, and `w3` represent their respective weights, each ranging from 0.2 to 0.6. The determination strategy is as follows: if the cumulative duration difference, i.e., the difference in cumulative speaking time among multiple speakers exceeds 40%, then the cluster with the longest duration is Teacher; if the difference is not significant, then a composite feature of "turns plus avg_len" is introduced; the remaining clusters are labeled as Student1, Student2, ... Student in order of appearance. n .

[0031] In a specific application example, the S70 merges adjacent segments with the same role and filters abnormally short segments by: inputting a role tag sequence and a timestamp sequence, merging adjacent segments with the same role in chronological order, and merging them into a single segment if the two segments have the same role and the interval is less than 0.2 seconds; for segments that are too short (less than 0.15 seconds), they are filtered or absorbed into a neighboring tag to ensure the continuity of the role's time sequence.

[0032] The classroom audio analysis and recognition method based on two-stage adaptive clustering described above can process classroom recordings in real time, distinguish between various event types such as "teacher speaking," "student speaking," "discussion," and "ambient noise," and dynamically adjust clustering parameters to cope with different audio densities and signal-to-noise ratios, providing highly stable speaker separation results. This provides data support for educational behavior analysis, classroom interaction assessment, and teaching quality diagnosis. By introducing event segmentation based on speech activity detection and combining self-supervised embedding features with a two-stage density adaptive clustering strategy, the method achieves automatic separation and event recognition of teacher and student speech in classroom audio, thus solving the problems of low recognition accuracy, poor clustering robustness, and insufficient adaptability in existing technologies.

[0033] It should be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A method for classroom audio analysis and recognition based on two-stage adaptive clustering, characterized in that, Comprising the following steps: S10, audio acquisition and preprocessing: the original classroom recording is processed to include unified sampling rate, single channel, light noise reduction, amplitude normalization and short-time energy statistics, to obtain the quality level of the audio segment, and then the standardized audio waveform and audio quality parameters are output; S20, speech detection: the normalized audio waveform and frame features obtained by preprocessing are input into the pyannote.audio neural network, and the speech segments are automatically constructed according to the frame interval with consecutive 1 in the mask, and the start and end time stamps of each segment are generated, and the frame-level speech confidence sequence and the binary speech mask obtained by thresholding are output; or use the DummyVAD model of voice activity detection, and adaptively select the effective speech segment based on multiple indexes including energy and spectral features, and output the start and end time stamp list of the speech segment and the corresponding metadata including segment-level energy and signal-to-noise ratio; S30, embedding extraction: for each speech segment, read the energy and signal-to-noise ratio metadata of its audio sample, and perform short-time spectral subtraction and normalization on the segment signal; On this basis, self-supervised embedding is extracted and fused with the representation based on traditional acoustic features; Then, use principal component analysis PCA to reduce the fusion vector to 64 dimensions and do L2 normalization on the reduced vector, output the 64-dimensional unified embedding vector embedding of each speech segment; S40, first stage clustering: use HDBSCAN clustering algorithm to cluster events using embedding vectors embedding and segment-level energy and signal-to-noise ratio data as input, and judge the spectral distribution, energy statistics and signal-to-noise ratio features of the cluster center at the cluster level, and divide the events into vocal group and environmental sound group, and output the corresponding vocal group index and environmental sound group index, which provides initial grouping constraints for subsequent fine-grained clustering and role recognition stage; S50, second stage clustering: further reduce the embedding vector embedding of the vocal group and the corresponding speech segment, then use adaptive KMeans or hierarchical clustering algorithm for the vocal group events, automatically select the number of clusters k according to internal evaluation indicators such as silhouette coefficient or heuristic rules, output the vocal fine-grained cluster label for distinguishing different vocal sounds, and use it for subsequent role recognition and event-level merging; S60, role recognition: use the second stage clustering result and segment-level statistical features to calculate the total speaking time, average length and speaking frequency of each vocal cluster, build a weighted behavior scoring model to automatically identify the teacher cluster, and the remaining vocal clusters are numbered as students according to the behavior characteristics, and output the role allocation results of the teacher and the student; S70, event fusion and result output: arrange the speech event sequence with completed role labeling in chronological order, merge adjacent speech segments with consistent roles, filter abnormal short segments, generate smooth time-sequential speech events, get the time-sequential timetable after role separation, and output the corresponding audio slices as needed for subsequent analysis and application.

2. The dual-stage self-adaptive clustering based classroom audio analysis and recognition method of claim 1, wherein, The S10 specifically includes: Read the original classroom recording and unify the sampling rate: audio_resampled = resample(input_audio, target_sr); The sampling rate used by the resample function, target_sr, calculates the number of samples per second. Then the mono processing is performed: audio_mono = mean(audio_resampled, axis=channels); The audio_mono parameter represents the processed result, and channels is the number of channels in the existing audio. The primary denoising is performed using the lightweight spectral subtraction or noisereduce library. First, estimate the noise segment, take the frame with the lowest energy in a certain time at the beginning as the noise spectrum estimation reference. Perform amplitude normalization, short-time energy, root mean square energy RMS, and zero-crossing rate ZCR statistics on the audio signal for subsequent adaptive threshold estimation. Finally, estimate the audio quality, including signal-to-noise ratio estimation and spectral entropy calculation, and output the audio quality level quality, with a value range of quality ∈ {low, medium, high}. Low indicates that the signal-to-noise ratio is below the set lower limit, and the spectral entropy or zero-crossing rate is abnormal, indicating that the recording environment noise is severe or the signal distortion is obvious. Medium indicates that the signal-to-noise ratio and other indicators are in the normal middle interval, and the recording quality is moderate, which can meet most subsequent analysis requirements. High indicates that the signal-to-noise ratio is significantly higher than the threshold, the spectral entropy is low, and the signal is clear and stable, indicating that the recording quality is good and suitable for fine speech processing. The formula for amplitude normalization of the input waveform is as follows: ； represents an instantaneous amplitude value of the original audio signal at time t, represents an absolute value maximum of the original audio signal, is a tiny positive number to prevent the denominator from being zero, and the processing formula for amplitude normalization is used to eliminate the volume gap caused by different recording devices and speaking distances.

3. The dual-stage self-adaptive clustering based classroom audio analysis and recognition method of claim 1, wherein, The S20 uses an enhanced DummyVAD to screen out valid speech segments, including adaptive threshold calculation: DummyVAD uses a comprehensive judgment that combines feature joint decision adaptive threshold, ZCR lower bound, and spectral entropy upper bound. The adaptive threshold calculation formula is as follows: vad_threshold = max(global_threshold_config, percentile(RMS, 25)) Where vad_threshold is the core threshold for speech activity detection; global_threshold_config is the global energy threshold, ranging from 0.01 to 0.1; percentile(RMS, 25) is the 25th percentile of the RMS sequence of the audio signal, reflecting the "lower quartile" level of RMS distribution, which helps to distinguish weak speech from noise.

4. The dual-stage self-adaptive clustering based classroom audio analysis and recognition method of claim 3, wherein, Further including multi-feature joint decision, which is used to introduce zero-crossing rate and spectral entropy constraints on the basis of energy threshold, to jointly define speech active frames, with the specific expression as follows: mask = (RMS > vad_threshold) & (ZCR > zcr_min) & (spec_entropy < spec_max) Wherein, vad_threshold is the core threshold of voice activity detection, RMS is the root mean square of the audio signal, RMS>vad_threshold is used to screen the frame with sufficient energy; The zero-crossing rate ZCR reflects the degree of activity of the audio frequency component, zcr_min ranges from 0.005 to 0.02, and ZCR>zcr_min indicates that the zero-crossing rate is higher than the minimum value; The spec_entropy is the spectral entropy, and spec_entropy<spec_max means that the spectral entropy is lower than the maximum value, which is more inclined to the low spectral entropy feature of the voice.

5. The dual-stage self-adaptive clustering based classroom audio analysis and recognition method of claim 4, wherein, Further comprising voice segment construction and segment-level filtering: merging the frames with continuous mask=1 into a segment: calculating the segment duration, average energy and segment-level signal-to-noise ratio seg_snr; Then, the two filtering conditions including minimum segment length filtering and minimum signal-to-noise ratio filtering are applied to the voice segment for filtering: The minimum segment length filtering is to discard if seg_duration < min_speech_sec Wherein, seg_duration is the voice segment length, min_speech_sec is the minimum voice segment length, and min_speech_sec is automatically adjusted according to the audio quality, which is set in AudioProcessor, and the general range is 0.3-1.0 seconds; The minimum signal-to-noise ratio filtering is to discard if seg_snr < snr_min Wherein, seg_snr is the signal-to-noise ratio of the voice segment, snr_min is the minimum signal-to-noise ratio, and the range is 3-10 dB, which can change with the quality level.

6. The dual-stage self-adaptive clustering based classroom audio analysis and recognition method of claim 5, wherein, Further comprising boundary smoothing processing: using a short-time cross-window median filter, a boundary expansion of 20-50 ms, and outputting a preliminary voice / mute segment list with timestamps, and accompanying each voice segment level start and end sample position, average energy, estimated signal-to-noise ratio and frame number.

7. The two-stage self-adaptive clustering based classroom audio analysis and recognition method of claim 1, wherein, The S30 has the following steps: Using the start and end of the audio segment level output in S10, the original waveform corresponding to each voice event and the attributes of each voice event corresponding to the voice segment level including average energy, duration and estimated signal-to-noise ratio are directly cut out from the normalized audio, and short-time spectral subtraction is performed on the single segment audio, and parameters such as FFT size fft_size and noise reduction proportion prop_decrease are retained; Performing segment amplitude normalization to ensure consistent amplitude scale, and the calculation method is as follows: seg = seg / (max_abs(seg) + ) The output feature seg is computed by adding a minimum value to the absolute maximum value max_abs(seg) of the feature to avoid a division by zero and to ensure a consistent amplitude scale. Performing self-supervised embedding feature calculation: calling a self-supervised embedding model to return a vector with an output dimension of 512-1024; Performing traditional feature supplement: if the embedding is unstable, i.e., the similarity between consecutive voice segment embedding vectors fluctuates beyond a threshold, or the signal-to-noise ratio of the audio segment is low, i.e., the segment signal-to-noise ratio is lower than a set threshold, then the fallback method is used to supplement the traditional acoustic features: calculating the MFCC mean and variance MFCC-stat, spectral centroid, ZCR, RMS-stat, and constructing an auxiliary feature vector; Fusion is performed, and the fusion strategy is: the participation proportion of traditional features is adaptively determined according to the segment-level SNR, and the fusion weight alpha is set, which ranges from 0 to 1; if it is weak speech with low SNR, that is, the segment SNR is lower than the set threshold, alpha is increased to strengthen the role of traditional features; if it is clear speech with high SNR, alpha is reduced to focus on self-supervised embedding; Dimension reduction and standardization are performed: the vector after fusion is first subjected to standardization operation, and then subjected to PCA dimension reduction to 64 dimensions, and L2 normalization operation is performed on the vector, to output the embedding represented by the 64-dimensional vector of each segment and the corresponding original vector dimension, energy, SNR and time length and other metadata.

8. The dual-stage self-adaptive clustering based classroom audio analysis and recognition method of claim 1, wherein, The S40 specifically includes: The low-dimensional embedding obtained in the embedding extraction of all events, and the corresponding segment-level metadata including energy, SNR, length are processed: first, the embedding is subjected to density clustering by using the density clustering algorithm HDBSCAN algorithm, the HDBSCAN algorithm outputs cluster_labels and cluster stability, wherein -1 in cluster_labels represents noise, and cluster stability is represented by cluster_probabilities; the statistical features of each cluster are used to calculate the indexes including average mel-frequency cepstral coefficient MFCC, average spectral centroid mean_spectral_centroid, average zero-crossing rate mean_zcr and speech ratio speech_ratio in the cluster, and the cluster category is judged according to the threshold value; if mean_spectral_centroid<thr_sc and mean_zcr<thr_zcr and speech_ratio>thr_sr, it is determined as a human voice cluster; otherwise, it is determined as an environmental sound cluster; for the cluster determined as environmental sound, it is labeled and excluded or separately refined from the next stage, and two types of partition indexes including the segment indexes corresponding to the human voice cluster and the segment indexes of the environmental sound cluster are output.

9. The two-stage self-adaptive clustering based classroom audio analysis and recognition method of claim 1, wherein, The S50 specifically includes: According to the first stage processing, the part determined as human voice is inputted, the embeddings of the human voice group and the signal-to-noise ratio, energy, length information of the human voice segment metadata are inputted; for the case that the human voice group characteristics are more complex and the distribution is closer, PCA is further performed on the human voice group to improve stability; if HDBSCAN has generated multiple clusters in the human voice group and the cluster quality score exceeds the threshold, the HDBSCAN algorithm clustering result is directly used; otherwise, adaptive KMeans clustering is used: that is, the optimal cluster number k is automatically selected through the silhouette score: the maximum silhouette score is selected by searching k∈[2, min(max_clusters, N-1)] through the silhouette coefficient; if the number of samples is small or the silhouette has no obvious peak value, the k based on the heuristic of the number of speech turns is used, for example, k is set to be equal to the square root of the number of speech segments, and is rounded up, so as to reduce the clustering label jump; the clustering result is optimized through time sequence smoothing technology such as neighborhood majority voting and simple linear chain conditional random field, and the final human voice cluster number with smoothing is outputted.

10. The dual-stage self-adaptive clustering based classroom audio analysis and recognition method of claim 1, wherein, The S60 specifically includes: The voice segment cluster after two-stage clustering is counted and identified, the information of each cluster including cumulative speaking time, first and last occurrence time, speaking frequency, average speaking length is counted, and the weighted behavior feature is used to identify the teacher role: score = w 1* duration + w 2* turns + w 3* avg_len duration is the speaking time, turns is the number of speaking times, avg len is the average speaking length, w1, w2, w3 represent the respective weights, the weight range is 0.2-0.6, and the determination strategy is as follows: if the cumulative time difference, that is, the difference ratio of the cumulative speaking time length between multiple speakers exceeds 40%, the longest cluster is the teacher; if the difference is not obvious, the "turns plus avg len" composite feature is introduced; the rest of the clusters are marked as students Student1, Student2,... Student n .

Images