Snore blocking part classification method based on audio abstract

Abstract

The invention provides a snore blocking part classification method based on an audio abstract, aims to solve the problems of data redundancy and classification accuracy in existing snore analysis, and provides reliable technical support for diagnosis of sleep apnea syndrome. Key snore segments are extracted from all-night audio data through five core modules, and blocked parts are recognized. Firstly, preprocessing such as pre-emphasis and sliding window framing is carried out on all-night audio signals recorded by a microphone, and a Mel spectrogram is generated. Secondly, based on a pre-trained openL3 network, the module receives a Mel spectrogram generated by preprocessing as input, and feature extraction is carried out through a multi-layer convolutional network; thirdly, clustering and importance evaluation of snore segments are achieved through an unsupervised learning method on the basis of acoustic depth features. Fourthly, a voice endpoint detection algorithm based on a Gaussian mixture model is included, and non-silent snore segments are extracted. And fifthly, a deep learning network pre-trained on the public data set MPSSC is adopted to achieve snore blocking part recognition.

Description

Technical Field

[0001] The invention relates to a method for classifying snoring obstruction parts based on audio summaries, and belongs to the technical field of biomedical signal analysis and sleep sound recognition. Background Art

[0002] Obstructive sleep apnea-hypopnea syndrome (OSAHS) is a common and serious sleep disorder, which is mainly manifested by apnea and hypopnea events that occur during sleep. Snoring is one of the important symptoms of OSAHS, and its formation process is closely related to the anatomical structure of the upper respiratory tract and airflow changes. When the airway is partially or completely blocked, changes in airflow will cause changes in snoring characteristics. Compared with the traditional PSG method, the method based on snoring analysis has the advantages of low cost, simple operation, and high patient acceptance, and can provide a non-invasive auxiliary tool for clinical diagnosis.

[0003] The acoustic characteristics of snoring are closely related to different obstruction sites in the upper respiratory tract. The analysis of snoring can provide support for the diagnosis and treatment of OSAHS. Existing snoring analysis methods mainly rely on machine learning and deep learning technologies for classification, but due to limited data volume and uneven sample distribution, it is often difficult to obtain ideal classification results. Taking the currently public MPSSC database as an example, although the database contains snoring data from different obstruction sites, the number of samples is small and there is a class imbalance problem, making it difficult to effectively apply traditional learning methods.

[0004] For snoring audio data, existing research is mostly focused on the field of speech, and lacks in-depth research on the acoustic features unique to snoring. Compared with speech, snoring lacks semantic information and relies only on acoustic features, which brings greater challenges to feature extraction. With the accumulation of all-night snoring data, audio data presents the characteristics of high temporal redundancy and acoustic feature similarity. A single patient's all-night snoring record usually contains hundreds to thousands of snoring events, which are highly similar in time-frequency features but also have subtle differences. Traditional supervised learning methods require a large number of manual expert annotations, which not only consumes a lot of manpower and material resources, but also often show limitations when faced with the diversity of snoring between different patients. Summary of the invention

[0005] The present invention proposes a snoring summary and obstruction site classification method based on unsupervised learning, which realizes the extraction of key snoring segments from the whole night audio data and the identification of obstruction sites through five core modules. The present invention directly processes the whole night audio signal, extracts acoustic features through audio preprocessing and OpenL3 network, realizes the screening of representative snoring segments based on unsupervised learning, uses Gaussian mixture model to detect snoring events, and finally completes the classification of obstruction sites by using transfer learning of pre-trained deep learning network.

[0006] Step 1, the audio preprocessing module, preprocesses and extracts features from the entire night audio signal recorded by the microphone. First, the original signal is preprocessed by pre-emphasis, framing and windowing, and then the signal is resampled and short-time Fourier transformed to obtain a spectrum array. Finally, a standardized Mel spectrum map is generated through Mel band filtering and logarithmic transformation as the input for subsequent feature extraction.

[0007] Step 2, deep feature extraction module, builds an acoustic feature extraction module based on the pre-trained OpenL3 network. This module receives the pre-processed mel-spectrogram as input and extracts features through a multi-layer convolutional network. The network adopts a step-by-step multiplication filter design (from 64 to 512 channels), combined with batch normalization layers and maximum pooling operations to gradually extract and reduce the dimension of feature representation. Finally, through the global pooling operation, the features of each 1-second audio clip are compressed into a 512-dimensional vector. The entire deep feature extraction module uses pre-trained parameters on the AudioSet dataset.

[0008] Step 3, the audio summary module, based on acoustic deep features, implements clustering and importance assessment of snoring segments through unsupervised learning methods. This module first standardizes the extracted 512-dimensional feature vectors to ensure the consistency of feature distribution, and then uses the MiniBatchKMeans algorithm to perform cluster analysis on the standardized features, and improves computational efficiency through iterative optimization of small batches of data. Based on the clustering results, the TF-IDF method is used to construct feature representation to capture the similarities and uniqueness between segments, and principal component analysis is used to reduce the dimension and retain the most representative feature dimensions. Finally, the Euclidean distance and confidence score are calculated based on the reduced-dimensional features, and the most representative segment is selected to generate a snoring summary.

[0009] Step 4, the snoring event detection module uses the VAD technology based on the Gaussian mixture model to identify and extract snoring segments from the whole night sleep audio. The module first performs 25ms frame length framing and short-time Fourier transform on the input audio to obtain the spectrum features; then the spectrum signal is divided into six sub-bands (80Hz to 4kHz) and the logarithmic energy is calculated, and the speech and noise probabilities of each sub-band are calculated based on the pre-trained Gaussian mixture model; through the log-likelihood ratio judgment and the dynamic management of the two-end data container, when the proportion of the sound segment exceeds 90%, it is determined that the snoring event has begun, and the snoring segment can be accurately detected and extracted.

[0010] Step 5, the snoring obstruction site classification module uses a pre-trained deep learning network to identify the snoring obstruction site. The present invention takes the CAM++ (Context-Aware Masking Plus Plus) network as an example. This module extracts the Mel-spectrogram features of the voiced segment after VAD processing, performs feature extraction and multi-scale fusion through a deep neural network, and finally outputs the probability distribution of each obstruction site. The model uses the angular additive marginal Softmax loss function to pre-train on the MPSSC dataset and supports migration to the local dataset for multi-label classification.

[0011] The present invention realizes the automatic summary analysis of the whole night audio, ensures the accuracy of the classification of the obstruction site based on transfer learning, and provides reliable technical support for the clinical diagnosis of sleep apnea syndrome. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 The overall flow chart of the invention

[0013] Figure 2 Flowchart for deep feature extraction

[0014] Figure 3 This is the structure diagram of the clustering-based all-night snoring summary algorithm

[0015] Figure 4 Flowchart of snoring event detection based on GMM

[0016] Figure 5 Overall flow chart for classifying the obstruction site of snoring

[0017] Figure 6 CAM++ model structure diagram DETAILED DESCRIPTION

[0018] The present invention provides a snoring obstruction site classification method based on audio summaries, aiming to solve the problems of data redundancy and classification accuracy in existing snoring analysis, and provide reliable technical support for the diagnosis of sleep apnea syndrome.

[0019] To make the purpose, technical solution and advantages of the present invention clearer, the specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the described embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0020] like Figure 1 As shown, the method for classifying snoring obstruction parts based on audio summaries provided by the present invention includes the following technical solutions:

[0021] The audio preprocessing module performs pre-emphasis and sliding window framing on the audio signal recorded by the microphone all night long, and generates a Mel spectrum.

[0022] The deep feature extraction module is based on the pre-trained OpenL3 network. This module receives the pre-processed mel-spectrogram as input and performs feature extraction through a multi-layer convolutional network.

[0023] The clustering audio summarization module realizes the clustering and importance assessment of snoring segments based on acoustic deep features through unsupervised learning methods.

[0024] The voice activity detection module includes a voice endpoint detection algorithm based on a Gaussian mixture model to extract non-silent snoring segments.

[0025] The snoring obstruction site classification module uses a deep learning network pre-trained on the public dataset MPSSC to identify the snoring obstruction site.

[0026] Step 1, the audio preprocessing module, performs pre-emphasis and sliding window framing on the audio signal recorded by the microphone all night long, and generates a Mel spectrum, which specifically includes the following steps:

[0027] Step 1-1: Perform basic preprocessing on the recorded whole night sleep audio, including pre-emphasis, framing and windowing.

[0028] Step 1-2, pre-emphasis is processed by a first-order high-pass filter and is implemented by the following formula:

[0029] y(n)=x(n)-αx(n-1)

[0030] Where x(n) is the original signal at time n, y(n) is the pre-emphasized signal at time n, α is the pre-emphasis coefficient, and the value range is 0-1. Here, α=0.95

[0031] Step 1-3, the audio is segmented using a standard frame length of 1 second;

[0032] In step 1-4, the windowing adopts the Hamming window function to avoid spectrum leakage when the audio signal period is extended.

[0033]

[0034] In steps 1-5, the whole night audio signal is resampled to 16 kHz and converted to a single precision format to ensure that the audio signal meets the subsequent processing requirements.

[0035] In step 1-6, the audio signal is processed using short-time Fourier transform (STFT), specifically using a 10-ms Hanning window and a 5-ms frame shift, and a 512-point discrete Fourier transform (DFT) to convert the audio signal into a 257×L spectrum array, where 257 represents the number of frequency bins in a single-sided spectrum, and L represents the number of time frames of the audio signal.

[0036] Step 1-7, extract the amplitude information of the complex spectrum, process it through a filter bank containing 128 Mel frequency bands, accumulate the amplitude values ​​in each frequency band, and obtain a Mel frequency spectrum of size 128×L.

[0037] Steps 1-8 perform logarithmic transformation on the Mel-spectrogram to enhance the matching degree with human ear perception.

[0038] In steps 1-9, the logarithmic Mel spectrogram is divided into multiple overlapping segments, each segment contains 199 spectrogram frames, and the size of each frame is 128×1, and finally a four-dimensional array of 128×199×1×K is generated as preprocessing, where K represents the number of generated spectrograms.

[0039] In steps 1-10, the processed Mel-spectrogram is input into the pre-trained OpenL3 network to extract deep features, providing a basis for subsequent analysis.

[0040] Step 2, deep feature extraction module, such as Figure 2 As shown in the figure, an acoustic feature extraction module is constructed based on the pre-trained OpenL3 network, which specifically includes:

[0041] Step 2-1, take the 128×199 logarithmic Mel-spectrogram output by the preprocessing module as input. Each input corresponds to a 1-second audio clip and is regarded as a grayscale image and input into the audio subnet of the OpenL3 network for processing.

[0042] Step 2-2, feature extraction is performed through the multi-layer convolution structure of the OpenL3 network. The network adopts a series of convolution blocks, each of which contains two 3×3 convolution layers. The stacking design of small convolution kernels expands the receptive field while maintaining a small number of parameters. The number of filters between adjacent convolution blocks is multiplied (64, 128, 256 and 512), and the feature extraction capability is improved by gradually increasing the number of feature channels.

[0043] In step 2-3, batch normalization and ReLU activation function are applied after each convolution operation to enhance the nonlinear expression ability of the model and accelerate the training process. A 2×2 maximum pooling layer (with a step size of 2 and no padding) is used between convolution blocks to gradually reduce the spatial resolution of the feature map and achieve dimensionality reduction and extraction of features.

[0044] Step 2-4, in the final stage of feature extraction, all spatial positions are pooled through a global maximum pooling operation to compress the multi-dimensional feature map into a 512-dimensional feature vector as a deep feature representation of the audio clip.

[0045] Steps 2-5 use the network parameters pre-trained on the AudioSet large-scale dataset. The pre-training process enables the model to learn rich audio feature representations.

[0046] Steps 2-6 repeat the above feature extraction process for each 1-second audio clip input, and finally obtain a time series feature sequence, in which each time point corresponds to a 512-dimensional feature vector, which will be used as the input for subsequent clustering and snoring summary.

[0047] Step 3, audio summary module, such as Figure 3 As shown in the figure, based on the deep acoustic features extracted by the pre-trained OpenL3 network, feature projection and clustering are performed through unsupervised learning methods, and TF-IDF vector representation and principal component analysis are combined to achieve importance assessment and summary generation of snoring audio clips, including:

[0048] Step 3-1: Use the pre-trained OpenL3 model to extract high-dimensional deep acoustic feature representation from the original snoring audio. Take the 512-dimensional feature vector output by the preprocessing module as input and perform feature extraction on each 1-second audio clip.

[0049] Step 3-2, feature clustering preprocessing, standardizes the extracted feature vectors to ensure that all features are within the same scale range.

[0050] Step 3-3, K-means clustering, uses the MiniBatchKMeans algorithm to perform K-means clustering on the extracted deep acoustic features. The algorithm aims to minimize the following objective function:

[0051]

[0052] Where K is the number of clusters, S i is the i-th cluster, μ i is the centroid of the ith cluster, and x is the data point. The specific process includes:

[0053] Step 3-3-1, specify the K value and determine the number of clusters K. The K value determines the number of final clusters and the number of initial centroids.

[0054] Step 3-3-2, initialize the centroid, use the K-means++ method to randomly select K samples from the data points as the initial centroid, and select points that are far away from the existing centroid as the new centroid to accelerate the convergence of the algorithm and improve the clustering quality.

[0055] Step 3-3-3, randomly sample small batches of data, and randomly extract a small batch of data subsets (mini-batch) with batch_size of 100 from the dataset.

[0056] Step 3-3-4, for each data point x assigned to cluster j, update the centroid according to the following incremental update rule:

[0057]

[0058] in is the centroid of the j-th cluster at the t-th iteration, is the number of data points assigned to the j-th cluster in the t-th iteration.

[0059] Step 3-3-5, repeat steps 3-3-3 and 3-3-4 until the convergence condition is met: the average change of all centroids is less than a preset threshold (such as 1e-4) or the maximum number of iterations is reached.

[0060] Step 3-4, TF-IDF feature extraction, applies the TF-IDF method to the clustering results for vector representation, specifically including:

[0061] Step 3-4-1, calculate the word frequency TF, for each cluster label t in the audio segment d, calculate its word frequency:

[0062]

[0063] Among them, TF(t, d) is the term frequency of word t in document d, N(t, d) represents the number of times word t appears in document d, and N(d) represents the total number of words in document d.

[0064] Step 3-4-2, calculate the inverse document frequency IDF. For each cluster label t, calculate its inverse document frequency in all segments:

[0065]

[0066] Among them, IDF(t) is the inverse document frequency of word t, D represents the total number of documents, and DF(t) represents the number of documents containing word t.

[0067] Step 3-4-3, calculate the TF-IDF value:

[0068] TF-IDF(t,d)=TF(t,d)×IDF(t)

[0069] Among them, TF-IDF((t,d) is the term frequency minus the inverse document frequency of word t for document d.

[0070] Step 3-5, principal component analysis dimensionality reduction, PCA dimensionality reduction is performed on the TF-IDF matrix, specifically including:

[0071] Step 3-5-1, calculate the covariance matrix.

[0072] Step 3-5-2, calculate the eigenvalues ​​and eigenvectors.

[0073] Step 3-5-3, select the first four principal components so that the cumulative variance contribution rate reaches 85%.

[0074] Step 3-5-4, project the original TF-IDF matrix into the selected principal component space.

[0075] Step 3-6, calculate the confidence score, based on the Euclidean distance between the TF-IDF vector of each segment and the overall audio feature vector, specifically including:

[0076] Step 3-6-1, calculate the overall audio feature vector, that is, the average value of the TF-IDF vectors of all segments.

[0077] Step 3-6-2, calculate the overall audio feature vector v mean , which is the mean of all segment TF-IDF vectors:

[0078]

[0079] Where N is the total number of audio clips, v i is the TF-IDF vector of the i-th segment.

[0080] Step 3-6-2, calculate the Euclidean distance between each fragment vector and the overall feature vector:

[0081]

[0082] where d i is the distance metric of the i-th segment, and D is the dimension of the feature vector.

[0083] Step 3-6-3, convert the Euclidean distance to a confidence score, using a normalized distance metric:

[0084]

[0085] where s i is the confidence score of the ith segment, and its value range is (0, 1]. The smaller the distance, the closer the score is to 1, indicating that the segment is more representative.

[0086] Step 4, snoring event detection module, such as Figure 4 As shown, the VAD (Voice Activity Detection) technology based on the Gaussian mixture model is used to identify and extract the voiced segments of snoring sleep audio, which specifically includes the following steps:

[0087] Step 4-1, preprocessing and framing the input snoring sleep audio, using a 25 millisecond frame length and a 10 millisecond frame shift for framing, and performing a 512-point short-time Fourier transform on each frame signal to obtain spectral features.

[0088] Step 4-2: Create a class object based on the framed signal to store and manage audio segment data.

[0089] Step 4-3, perform sub-band energy analysis, divide each frame signal into six sub-bands according to the frequency range: 80-250 Hz, 250-500 Hz, 500 Hz-1 kHz, 1-2 kHz, 2-3 kHz and 3-4 kHz, and calculate the logarithmic energy value of each sub-band.

[0090] Step 4-4, when the total frame energy exceeds the trigger threshold, the speech probability P(X|H1) and the noise probability P(X|H0) are calculated for each sub-band, where H1 represents the speech hypothesis and H0 represents the noise hypothesis. The probability calculation is based on the pre-trained Gaussian mixture model.

[0091] Step 4-5, calculate the log-likelihood ratio. For each sub-band, calculate its log-likelihood ratio according to the formula:

[0092]

[0093] Step 4-6, determine the sound segment. If the likelihood ratio of any sub-band exceeds the preset threshold, or the sum of the likelihood ratios of all sub-bands exceeds the overall threshold, the frame is determined to be a sound segment, otherwise it is determined to be a silent segment.

[0094] Step 4-7, initialize a dual-end data container for temporarily storing the audio clips to be processed.

[0095] Step 4-8, perform voiced segment collection. When the voiced segments in the data container account for more than 90%, write the current container data into the result list, and continue this process until the silent segments account for more than 90%.

[0096] Step 4-9, outputting the sequence of voiced segments processed by VAD as input data for subsequent snoring analysis.

[0097] The VAD module described above realizes accurate extraction of effective signals from snoring audio by combining frequency domain analysis with statistical models, laying the foundation for subsequent feature analysis.

[0098] Step 5, snoring obstruction location classification module, such as Figure 5 As shown in FIG. 1 , a pre-trained deep learning network is used to construct a snoring obstruction location recognition system. The present invention takes the CAM++ (Context-Aware Masking Plus Plus) network as an example, and the network structure diagram is shown in FIG. Figure 6As shown in the figure, the network shows excellent classification performance in cross-validation of public data sets. At the same time, the classification module design in the present invention has good scalability and supports replacement with other deep learning models with similar network structures and excellent pre-training performance, providing flexible model selection space for different application scenarios. Specifically, the following steps are included:

[0099] Step 5-1, perform feature preprocessing on the voiced segment after VAD processing, and extract Mel-spectrogram features according to the method of step 1 as input data of the CAM++ network.

[0100] In step 5-2, the input features are sent to the front-end convolution module (FCM), and the time-frequency domain acoustic features are extracted through multiple two-dimensional convolution blocks with residual connections. The feature map is flattened along the channel and frequency dimensions and then input into the D-TDNN backbone network.

[0101] In step 5-3, three series-connected D-TDNN (Dense Time Delay Neural Network) blocks are used to construct a feature extraction backbone network. Each D-TDNN block contains a densely connected layer and an improved context-aware masking module.

[0102] In step 5-4, a context-aware masking (CAM) module is embedded in each D-TDNN block to assign dynamic attention weights to the features output by the TDNN layer and achieve effective aggregation of context information at different levels through multi-granularity pooling.

[0103] In step 5-5, a dense connection mechanism is used to fuse the features of the outputs of each D-TDNN block, concatenate the current block output with the features of all previous layers, and integrate the global and local context information through a multi-granularity pooling layer.

[0104] In step 5-6, classification prediction is performed on the fused feature vector, feature transformation is performed through the fully connected layer, and the probability distribution of each blocked part is calculated using the softmax function.

[0105] In steps 5-7, model pre-training is performed on the Munich Passau Snore Sound Corpus (MPSSC) public dataset, using the angular additive margin Softmax loss function (AAM-Softmax Loss) as the optimization objective, and the network parameters are updated through back propagation.

[0106] Step 5-8: deploy the pre-trained CAM++ model to the local data set, load the pre-trained parameters, classify the input snoring sound segment according to the blocked part, and output the final recognition result.

Claims

1. A method for classifying snoring obstruction sites based on audio summaries, characterized by: The following steps are involved: S1. Preprocess and extract features of the audio signal recorded by the microphone all night long to generate a Mel-spectrogram; S2. Extract acoustic features using the pre-trained OpenL3 network; S3. Clustering and importance evaluation of snoring segments based on unsupervised learning methods; S4. Use VAD technology based on Gaussian mixture model to identify and extract snoring sound segments; S5. Classify the obstruction sites of snoring clips using a pre-trained deep learning network.

2. The method according to step S1 of claim 1, characterized in that: The whole night audio signal recorded by the microphone is preprocessed and features are extracted. First, the original signal is preprocessed by pre-emphasis, framing and windowing. Then the signal is resampled and short-time Fourier transformed to obtain a spectrum array. Finally, a standardized Mel spectrum map is generated through Mel band filtering and logarithmic transformation as the input for subsequent feature extraction.

3. The method according to step S2 of claim 1, characterized in that: An acoustic feature extraction module is built based on the pre-trained OpenL3 network. The module receives the pre-processed mel-spectrogram as input and extracts features through a multi-layer convolutional network. The network adopts a step-by-step doubling filter design from 64 to 512 channels, combined with batch normalization layers and maximum pooling operations to gradually extract and reduce the dimensionality of feature representations. Finally, through a global pooling operation, the features of each 1-second audio clip are compressed into a 512-dimensional vector. The entire module uses pre-trained parameters on the AudioSet dataset.

4. The method according to step S3 of claim 1, characterized in that: Based on acoustic deep features, the clustering and importance evaluation of snoring fragments are achieved through unsupervised learning methods. The module first standardizes the extracted 512-dimensional feature vectors to ensure the consistency of feature distribution, and then uses the MiniBatchKMeans algorithm to perform cluster analysis on the standardized features. The computational efficiency is improved through iterative optimization of small batch data. Based on the clustering results, the TF-IDF method is used to construct feature representation to capture the similarity and uniqueness between fragments, and the principal component analysis is used to reduce the dimension and retain the most representative feature dimensions. Finally, the Euclidean distance and confidence score are calculated based on the features after dimensionality reduction, and the most representative fragment is selected to generate a snoring summary.

5. The method according to step S4 of claim 1, characterized in that: VAD technology based on Gaussian mixture model is used to identify and extract snoring fragments from the whole night sleep audio. The module first divides the input audio into 25ms frames and performs short-time Fourier transform to obtain spectral features, then divides the spectral signal into six sub-bands and calculates the logarithmic energy. The speech and noise probabilities of each sub-band are calculated based on the pre-trained Gaussian mixture model. Through log-likelihood ratio judgment and dynamic management of two-end data containers, when the proportion of sound fragments exceeds 90%, it is determined that the snoring event has started, thereby achieving accurate detection and extraction of snoring fragments.

6. The method according to step S5 of claim 1, characterized in that: A deep learning network pre-trained on the public dataset MPSSC is used to identify the obstruction site of snoring. Taking CAM++ as an example, the module extracts the Mel-spectrogram features of the sound segment after VAD processing, performs feature extraction and multi-scale fusion through a deep neural network, and finally outputs the probability distribution of each obstruction site. The model is pre-trained on the MPSSC dataset using the angular additive marginal Softmax loss function, and supports migration to the local dataset for multi-label classification.

Images