All-night snore detection method based on frequency domain attention and self-attention pooling double-fusion mechanism

Abstract

The invention discloses an all-night snore detection method based on a frequency domain attention and self-attention pooling double-fusion mechanism, and belongs to the field of sound event detection. The method comprises the following steps: designing a waveform feature extraction module based on a frequency domain self-attention sub-module, a Mel-cepstrum feature extraction module based on a deformable convolution residual sub-module, a self-attention pooling double-fusion module and a decoder based on a multilayer perceptron; a deep learning snore event detection model based on a frequency domain attention and self-attention pooling double-fusion mechanism is constructed by using the modules. The model can detect and position snoring sound in all-night sleep sound data in a point-by-point framing detection mode by loading trained parameters, and the starting and ending time of a snoring event in all-night time and the corresponding snoring sound type are obtained. The model can rapidly and accurately detect snore segments in the whole night sleep period, and is helpful for assisting medical staff in rapidly screening patients with sleep breathing disorder in the early stage.

Description

Technical Field

[0001] The present invention relates to the field of sound event detection, and in particular to a method for detecting snoring all night long based on a dual fusion mechanism of frequency domain attention and self-attention pooling. Background Art

[0002] Reasonable sleep time is crucial to human health. In today's society, more and more people suffer from poor memory due to poor sleep quality, which reduces work and study efficiency and even affects people's normal life. One of the major culprits for poor sleep quality is sleep apnea syndrome (OSAHS). According to The Lancet Respiratory Medicine, as of 2019, there were more than one billion patients with sleep apnea worldwide, and with the change of lifestyle, the prevalence has increased year by year, and the incidence rate of adult males has increased from 3.3% in the early 1980s to 38%. OSAHS refers to the direct disappearance or significant reduction of nasal airflow during sleep (reduced by 90% or more than the basic level) and lasting for 10 seconds or more. Symptoms of OSAHS include shortness of breath, wheezing, uneven snoring, body twitching, respiratory arrest, and even shock during sleep. OSAHS seriously affects people's quality of life. It may not only cause problems such as fatigue, depression, memory impairment and bad temper, but also become the main cause of road traffic accidents, work injuries and sudden death. Due to long-term chronic intermittent hypoxia, OSAHS can also cause a series of serious cardiovascular and cerebrovascular diseases and metabolic diseases including hypertension, coronary heart disease, stroke, and diabetes, and cause a series of socioeconomic problems. However, most OSA patients do not realize their condition until it becomes serious, thus missing the opportunity for early treatment. In this case, early diagnosis and early warning of OSAHS is an issue worthy of attention.

[0003] There are currently two main methods for identifying snoring throughout the night. One is to use a wearable intelligent system to collect the wearer's turning information during sleep, and the other is to monitor the changes in the sleep position of the monitored person through infrared video monitoring. Both main methods have certain shortcomings. The former wearable device will affect the patient's sleep state, cause discomfort to the patient, and the electrode is prone to fall off. The latter will infringe on the patient's privacy. Therefore, the sleep sound data of snoring patients throughout the night is collected through a non-contact sleep sound detection system, and the snoring characteristics are analyzed for detection and screening. A variety of methods such as mathematical statistics, machine learning, and pattern recognition are used to achieve a classification study of the sleep position of snoring patients throughout the night. This method can not only ensure the stability of the patient's sleep state, but also ensure the patient's privacy.

[0004] Commonly used snoring detection methods are mainly divided into two categories: single-stage method and multi-stage method. The specific difference between these two methods is that the former needs to go through multiple stages of processing and analysis to achieve the purpose of snoring detection, while the latter can achieve end-to-end detection in one step. The multi-stage method divides the snoring detection task into multiple processes, which can be mainly divided into three stages: sound segment detection, feature extraction, and snoring detection. Although this method can achieve snoring detection, its process is complicated and difficult to achieve widespread application. Since the beginning of the 21st century, a large number of advanced methods have emerged for snoring detection. Traditional and classic machine learning algorithms, such as K-nearest neighbor (KNN), support vector machine (SVM), Gaussian mixture model (GMM), linear regression, logistic regression and other classification algorithms have been applied. In addition to single classifiers, the ensemble learning classifier Adaboost of multiple classifiers has also been applied in the task of snoring detection. Recently, the rapid development of deep learning technology has greatly promoted the progress of research in the field of artificial intelligence, and many deep learning technologies have also been used in snoring detection. Recurrent neural networks and artificial neural networks are widely used in snoring detection, but convolutional neural networks, which have made great achievements in the field of images, have not been widely used in the field of snoring detection. Therefore, how to apply the object detection technology based on convolutional neural networks to the field of snoring detection is still a challenging and meaningful task.

[0005] At present, the research on automatic classification of OSAHS still has problems such as large amount of data annotation, heavy expert annotation work, and small amount of existing annotation data, and the research is still very challenging. The present invention designs a waveform feature extraction module based on the frequency domain self-attention submodule to enhance the frequency bands in the waveform data that are effective for detection and classification; constructs a Mel-cepstral feature extraction module based on the deformable convolution residual submodule to improve the encoder module's ability to extract complex audio pattern features; and designs a dual fusion module based on self-attention pooling to achieve bidirectional fusion of audio features and Mel-cepstral features; finally, a decoder based on a multi-layer perceptron module is constructed to achieve classification and snoring time position regression, forming the entire detection network model FDSNet. The model has a better function of extracting snoring characteristics, and can quickly and accurately detect snoring fragments during a full night of sleep. On the one hand, it can effectively alleviate the existing problems of PSG detection resource shortage in hospitals, reduce the workload of medical workers, and relieve work pressure to assist in diagnosis. On the other hand, the diagnostic screening of simple snoring and mild to moderate OSAHS patients can be extended to the community and even the family, which is of great significance for the prevention, screening and postoperative monitoring of OSAHS in the snoring patient group. Summary of the invention

[0006] In view of the problems in the prior art such as low snoring detection accuracy and missed detection, the present invention provides a whole-night snoring detection method based on a dual fusion mechanism of frequency domain attention and self-attention pooling, aiming to quickly and accurately extract all snoring events in long-term whole-night sleep audio files.

[0007] The present invention designs a waveform feature extraction module based on the frequency domain self-attention submodule, and uses the frequency domain attention mechanism to enhance the frequency bands in the waveform data that are effective for detection and classification; constructs a Mel-cepstral feature extraction module based on the deformable convolution residual submodule to improve the ability to extract complex audio pattern features; and designs a dual fusion module based on self-attention pooling to achieve bidirectional fusion of audio features and Mel-cepstral features; finally, constructs a decoder based on the multi-layer perceptron module to achieve snoring event location and classification, forming the entire detection network model FDSNet. The entire network outputs four-layer multi-scale prediction regression results and classification results, which can effectively reduce missed detection events and improve detection accuracy.

[0008] In order to achieve the above object, the present invention provides a method for detecting snoring all night long based on a dual fusion mechanism of frequency domain attention and self-attention pooling, comprising the following steps:

[0009] Step S10: Dataset construction

[0010] The audio files of all-night sleep sounds recorded by a high-sensitivity microphone are divided into frames to create a mixed dataset of snoring events and non-snoring events. The sample frame length of the dataset is set to T seconds, and the corresponding sound event types of the obtained data are marked as: non-snoring events (throat clearing, coughing, door pushing, collision, talking, thunder), snoring events (normal snoring and obstructed snoring); the dataset is normalized to the range of [0, 1] according to formula (1):

[0011]

[0012] Where x is one-dimensional audio waveform data.

[0013] The Mel-frequency cepstrum of the normalized waveform is calculated according to equations (2) and (3); the data set required for the model is constructed, including T seconds of audio waveform data X, Mel-frequency cepstrum M, and the corresponding label L, and finally a data set consisting of {X, M, L} is obtained for network training.

[0014]

[0015]

[0016] where |X(n)| is the spectrum amplitude, H m(k, n) is the response of the mth Mel filter spectrum to the spectral component X(n) at frequency index k, log_S(m, k) is the logarithmic magnitude spectrum, MFCC c (m) is the c-th Mel-cepstral coefficient of the m-th frame, K is the number of Mel filter banks, and c is the index of the Mel-cepstral coefficient.

[0017] Step S20: construct an FSDNet snoring detection network model, which consists of four parts: audio waveform encoder (Audio WaveEncoder, AWE), Mel Cepstrum Encoder (Mel Cepstrum Encoder, MCE), multi-domain bidirectional fusion module (Multi-domain Bidirectional Fusion Module, MDBFM) and decoder.

[0018] The input of the audio waveform encoder (AWE) is one-dimensional audio waveform data. The audio waveform encoder is composed of four levels of waveform feature extraction modules connected in series. The audio data is downsampled and features are extracted step by step to obtain four layers of multi-scale audio features. Each waveform feature extraction module includes three frequency domain self-attention sub-modules (FASM). The frequency domain self-attention sub-module (FASM) adds a frequency domain self-attention layer FA (Frequency Attention, FA) between the one-dimensional convolution layer and the Transformer encoding layer, which can enhance the frequency bands in the waveform data that are effective for detection and classification, and can better extract the inherent features of snoring for easy detection.

[0019] The input of the Mel Cepstral Encoder (MCE) is a Mel Cepstral graph, which includes a Mel Cepstral preprocessing module and four Mel Cepstral feature extraction modules. Similar to the audio waveform encoder, the Mel Cepstral Encoder also connects four levels of Mel Cepstral feature extraction modules in series to downsample the Mel Cepstral coefficient graph. The output of the MCE is four layers of feature graphs of different scales. Each level of Mel Cepstral feature extraction module includes three Deformable Conv Residual Sub-Modules (DCRSM), which use deformable convolution layers to adaptively select effective feature patterns of the Mel Cepstral to extract features from the Mel Cepstral, thereby improving the encoder module's ability to extract complex audio pattern features.

[0020] The multi-domain bidirectional fusion (MDBFM) can realize the fusion of audio waveform features and Mel cepstrum features so that the network can learn the characteristics of snoring more accurately. It includes four levels of self-attention pooling and double fusion modules (SAPDF), which can obtain four layers of audio multimodal representation data of different scales. Each level of SAPDF module consists of a self-attention pooling layer (SAP) and two feature fusion layers (Fusion Block); the input of the self-attention pooling layer is the four layers of different scale feature maps output by the Mel cepstrum encoder, and the resolution feature maps of the other three scales are used as K, Q, and V to calculate the multi-scale attention, which can combine the contextual information of different scales and fuse with the pixel features of the current scale, which can reduce the loss of features during downsampling; the input of the two feature fusion layers is the output of SAP and the output of the waveform feature extraction module of the corresponding level, respectively, and the features calculated by the weights of the audio waveform features and the weights calculated by the Mel cepstrum features are obtained, and the outputs of the two feature fusion layers are added to obtain the output of the final SPADF module.

[0021] The decoder is composed of four multi-layer perceptron (MLP) modules, each of which contains three fully connected layers. The input of this module is the multimodal representation of the audio data obtained by the output of the corresponding level SAPDF module. Based on the output of the four multi-layer perceptron modules, the entire model is optimized and regressed to obtain a one-dimensional predicted regression box and category. The predicted regression box contains the center point of the sound clip, the audio length offset and the confidence. The loss function is obtained by weighting the CIOU loss, confidence loss and classification loss of the predicted regression box.

[0022] Step S30: Use the data set constructed in S10 to train the FDSNet network model constructed in S20. Set hyperparameters such as learning rate, learning rate change mode, sample batch size, and acoustic signal anchor box size and number during training.

[0023] Among them, the anchor frame parameters can be determined by unsupervised clustering. The specific method is: the K-means clustering algorithm is used to obtain the size of the anchor frame, the starting point information of all samples in the data set is converted into duration, and the K-means clustering algorithm is used to randomly generate k frames, and the similarity between each frame and the original frame is calculated and the similarity is 1-IOU. After all the frames are divided into k categories, the k frames are updated using the mean of the classification, and the similarity is repeatedly calculated and updated. The algorithm is stopped when the change value between the new frame and the original frame is small.

[0024] According to the set hyperparameters, the FDSNet network model is trained using the data set, and the model is optimized through the reverse transfer algorithm until the model training error is less than the preset threshold L threshold Or the number of iterations is greater than the given value T m .

[0025] Step S40: Use the trained optimal model to detect the whole night audio file, divide the whole night audio file to be predicted into frames, and cut it into multiple audio segments of T seconds in length. Use formula (1) to perform normalization preprocessing on the segmented segments, and then use formula (2) and formula (3) to calculate the Mel-frequency cepstrum of the normalized segments. The model loads the optimal parameters after training, and sends the audio waveform data and the corresponding Mel-frequency cepstrum to form the data pair to be predicted into the model to obtain the prediction result of each frame of data. Then post-process the results, merge the segments that are divided into different snoring segments but actually belong to the same snoring segment, and finally obtain the time interval and event category of all snoring in the whole night audio file.

[0026] The audio waveform encoder described in step S20 is composed of four levels of waveform feature extraction modules of different scales connected in series in ascending order of scale. The input of the first waveform feature extraction module is a one-dimensional audio signal of length L×1 (L is the length of the audio data, 1 is the number of channels), and the output is an audio feature A1 of L / 40×C, where C represents the number of channels; the input of the remaining three waveform feature extraction modules is the output of the previous waveform feature extraction module; the corresponding outputs A2 to A4 are L / 80×C, L / 160×C, and L / 320×C, respectively.

[0027] Each waveform feature extraction module is composed of three frequency domain attention submodules (FASM) connected in series. Each frequency domain attention submodule is composed of a one-dimensional convolutional layer, a frequency domain self-attention layer and a Transformer encoding layer connected in series, which is used to extract audio features; its main feature is the addition of a frequency domain self-attention layer (FA), which specifically includes the following steps: cutting the features into C components according to the number of channels (C is the number of channels), performing a fast Fourier transform (FFT) on each component according to formula (4) to obtain the frequency domain component and splicing it on the channel, then passing the fully connected layer and the Sigmoid output as the channel weight coefficient, and finally multiplying it with the original feature on the channel to obtain the attention-weighted one-dimensional feature.

[0028]

[0029] The Mel cepstral encoder described in step S20 is correspondingly composed of four levels of Mel cepstral feature extraction modules of different scales connected in series. The input of the Mel cepstral encoder is an image with a resolution of H×W×1 (H and W are the height and width of the Mel cepstral image, respectively, and 1 is the number of channels). First, it passes through the Mel cepstral preprocessing module, and the resolution becomes H / 4×W / 4×C; then the first level Mel cepstral feature extraction module without downsampling is input, and its output is a feature spectrum with a resolution of H / 4×W / 4×C. The remaining Mel cepstral feature extraction modules are all downsampled and the input is the output of the previous Mel cepstral feature extraction module; finally, the four-layer multi-scale resolution feature maps are obtained, respectively: H / 4×W / 4×C, H / 8×W / 8×2C, H / 16×W / 16×4C, H / 32×W / 32×8C, where C is the number of channels.

[0030] The Mel-cepstral preprocessing module consists of two two-dimensional convolutional layers with a convolution kernel of 5×5, a deformable convolutional layer with a convolution kernel of 3×3, a batch normalization layer, an activation layer and a pooling layer connected in series.

[0031] Each Mel-cepstrum feature extraction module is composed of three deformable convolution residual submodules connected in series. The main feature of the deformable convolution residual submodule is that it uses a deformable convolution layer, which can adaptively select patterns from the Mel-cepstrum that are helpful for downstream audio detection and classification as feature extraction, and improve the robustness of the representation of complex and diverse snoring cepstrum characteristics. The specific construction method of the deformable convolution residual submodule is:

[0032] f rbc (x) = ReLU(b(ζ(x))) (5)

[0033] y = ReLU(b(g(f rbc2 (f rbc1 (x))))+Δx) (6)

[0034] ReLU(x)=max(0,x) (7)

[0035] Where x is the input of the deformable convolution residual submodule, y is the output of the DCRSM module, Δx is the residual term, ζ is the deformable convolution, b is batch normalization, ReLU is the rectified linear unit, and f rbc1 and f rbc2 They are respectively formula (5), where the step sizes of the deformable convolution are 1 and 2 respectively.

[0036] The multi-domain bidirectional fuser described in step S20 is composed of four separate waveform domain and Mel cepstral domain bidirectional fusion modules SAPDF based on the self-attention pooling mechanism. Each SAPDF module is used to fuse the one-dimensional audio features and Mel cepstral features of the same level; the output of the four-level Mel cepstral feature extraction module is adjusted to the same number of channels to obtain F1~F4: H / 4×W / 4×C, H / 8×W / 8×C, H / 16×W / 16×C, H / 32×W / 32×C, and the input of each SPADF module is F1, F2, F3, F4 and the audio feature A output by the i-th waveform feature extraction module. i , the output is the feature after fusion of the current scale, and finally the output of the four-level multi-scale SAPDF module is Z1~Z4; its main feature is the design based on the self-attention pooling layer and the feature fusion layer, which can capture the audio features more accurately, including the following steps:

[0037] The input of the self-attention pooling layer SAP is F1~F4, and the resolution feature maps of the other three scales are used as K, Q, and V to calculate the multi-scale attention; the V value is spliced ​​on the channel after global average pooling and global maximum pooling, and V′ is obtained after convolution. The results of global average pooling of K value and global maximum pooling of Q value are spliced ​​on the channel through the convolution layer, and the output after Sigmoid is multiplied by V' value. The result is Softmaxed and combined with the current scale feature map F i Multiply and add F i Add together to get the final output M i , the calculation formula of the self-attention pooling layer is as follows:

[0038] f(x,y)=g(GAP(x)||MAP(x)) (8)

[0039]

[0040]

[0041]

[0042] Where || represents concatenation on the channel, g is the convolution layer, and σ is the Sigmoid function.

[0043] When calculating M1, Q = F2, K = F3, V = F4;

[0044] When calculating M2, Q = F1, K = F3, V = F4;

[0045] When calculating M3, Q = F1, K = F2, V = F4;

[0046] When calculating M4, Q = F1, K = F2, V = F3;

[0047] Based on the output M of the self-attention pooling layer i After global average pooling, it is obtained through the convolution layer With the same feature resolution as Ai at the current scale, and A i Input two feature fusion layers, add the outputs of the two feature fusion layers to get the output of the SPADF module, where the formula of the feature fusion layer is formula (1.12):

[0048]

[0049]

[0050] Among them, Both ρ and μ are one-dimensional convolutions with a kernel of 1, and N is a normalization factor.

[0051] In step S20, the decoder includes four multi-layer perceptron (MLP) modules, each of which includes three fully connected layers, and the activation function is ReLU, and its input is the multimodal representation of the audio data obtained by the output of the corresponding level SAPDF module; the formula of the MLP module is as follows:

[0052] f mlp =ReLU(ψ(x)) (14)

[0053] O(x)=f mlp3 (f mlp2 (f mlp1 (x))) (15)

[0054] The module input is x, ψ is the linear layer, and ReLU is the rectified linear unit.

[0055] The entire model is optimized and regressed based on the output of the decoder. The total loss of the entire FDSNet network model is the weighted sum of classification loss, confidence loss and prediction regression loss. This can reduce the missed detection of short snoring sounds and accurately and quickly detect long snoring sounds. The specific steps include:

[0056] The four-layer multi-scale feature mapping obtains a one-dimensional prediction regression box {x, l, con} and a classification result. The prediction regression box contains the center point x of the sound clip, the audio length offset l, and the confidence con; the classification loss L class With confidence loss L con The selected loss function is the binary cross entropy loss function, and the calculation formula is (16) and (17). The L of the one-dimensional prediction box and the one-dimensional target box is calculated. CIOU The loss is calculated as shown in equations (18) to (22):

[0057]

[0058] Where N is the number of samples, y represents the true label of the sample, x represents the predicted value of the sample, and σ represents the Sigmoid function.

[0059]

[0060] Where N is the number of samples, y represents the true confidence of the sample, x represents the predicted confidence of the sample, and α represents the Sigmoid function.

[0061] L CIOU (T, Y) = 1-IOU (T, Y) + d (T, Y) + αv (T, Y) (18)

[0062]

[0063]

[0064]

[0065] α=v / (1-IOU+ν) (22)

[0066] Where IOU is the intersection-over-union ratio of two boxes, T e and T s is the end point and starting point of the real box T, Y e and Y s is the end point and starting point of the prediction box Y, eps is 10 -16 , αν and l(T,Y) can further adjust the loss.

[0067] The total loss is calculated as (23), which is the classification loss L class , confidence loss L con , prediction box and target box loss L CIOU The weighted sum, where the values ​​of λ1, λ2, and λ3 range from 0 to 1 and their sum is 1.

[0068] loss=λ1L class +λ2L con +λ3L CIOU (twenty three)

[0069] Step S30 specifically includes the following steps:

[0070] The FDSNet network model constructed by S20 is trained using the dataset constructed by S10. Hyperparameters such as optimizer, learning rate, learning rate change mode, sample batch size, and acoustic signal anchor box size and number are set during training.

[0071] The learning rate changes in a cosine dynamic transformation, and the overall logic is that the coefficients maintain linear growth before the warm-up period and during the warm-up period t warm Then the corresponding cosine function is used for transformation, and the calculation formula is (24):

[0072]

[0073] where t c is the current training cycle, t warm is the preheating period, t all is the total training cycle, N c is the number of cycles.

[0074] Among them, the anchor frame parameters can be determined by unsupervised clustering. The specific method is: the K-means clustering algorithm is used to obtain the size of the anchor frame, the starting point information of all samples in the data set is converted into duration, and the K-means clustering algorithm is used to randomly generate k frames, and the similarity between each frame and the original frame is calculated and the similarity is 1-IOU. After all the frames are divided into k categories, the k frames are updated using the mean of the classification, and the similarity is repeatedly calculated and updated. The algorithm is stopped when the change value between the new frame and the original frame is small.

[0075] According to the set hyperparameters, the FDSNet network model is trained using the data set, and the model is optimized through the reverse transfer algorithm until the model training error is less than the preset threshold L threshold Or the number of iterations is greater than the given value T m The training evaluation indicator is the mean average precision (mAP) of all classes, and the model with the highest mAP is selected as the optimal model.

[0076] Step S40 specifically includes the following steps:

[0077] Use the trained optimal model to detect the whole night audio file, divide the whole night audio file to be predicted into frames, and cut it into multiple audio segments of T seconds in length. Use formula (1) to normalize the segmented segments, and then use formula (2) and formula (3) to calculate the Mel cepstrum of the normalized segment. The model loads the optimal parameters after training, and sends the audio waveform data and the corresponding Mel cepstrum to form the data pair to be predicted into the model, and the prediction result Y of each frame of data can be obtained. pad ∈{t s , t e ,c,y p}, where t s is the starting point of the audio segment being measured, t e is the end point, c is the confidence level, y p For category.

[0078] Then, the detection results are post-processed to first filter out all the detection segments that are classified as snoring and have a confidence level greater than c. threshold The audio detection results are then merged into different snoring segments but actually belong to the same snoring segment. The splicing and fusion criteria include: (1) Both audio segments are snoring and the confidence level exceeds c threshold , that is, the result of formula (25) is 1; (2) the time difference Δt between the two snoring segments is less than the threshold value t0, that is, the result output of formula (26) is 1; if the above two criteria are met, that is, the results of formula (25) and formula (26) are both 1, then they are regarded as the same snoring segment and spliced ​​into a complete snoring segment, otherwise they are still regarded as two independent snoring segments, and finally the starting points of all snoring segments in the whole night audio file are obtained.

[0079]

[0080]

[0081] Compared with the prior art, the advantages of the present invention are: a waveform feature extraction module based on a frequency domain self-attention submodule is designed to enhance the frequency bands in the waveform data that are effective for detection and classification, thereby better extracting the inherent features of snoring and improving the performance of downstream detection tasks; a Mel-cepstrum feature extraction module based on a deformable convolution residual submodule is constructed, and the Mel-cepstrum is extracted by using an effective feature method of adaptively selecting Mel-cepstrum by multi-level deformable convolution, thereby improving the encoder module's ability to extract complex audio pattern features; a dual fusion module based on self-attention pooling is designed to achieve bidirectional fusion based on audio features and Mel-cepstrum features, and finally obtain a multimodal representation of the input audio data, thereby improving the feature fusion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 This is the overall structure diagram of the FDSNet network;

[0083] Figure 2 It is a schematic diagram of FASM module;

[0084] Figure 3 It is the schematic diagram of DCRSM module;

[0085] Figure 4 This is the schematic diagram of the SAPDF module;

[0086] Figure 5 This is the test result of a 10-minute snoring segment; Figure 6 It is a diagram of the invention summary; DETAILED DESCRIPTION

[0087] In order to enable persons skilled in the art to better understand the solutions of the embodiments of the present invention, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and implementation modes.

[0088] The present invention comprises the following steps:

[0089] The whole-night sleep audio files of a middle-aged female patient with mild OSAHS and a young male patient with simple snoring were selected to create a dataset. The audio file format is bin, the sampling rate is 16k, and the data type is Int16; the whole-night audio data is pre-emphasized to increase the weight of the high-frequency components in the entire frequency band, and the pre-emphasized filter coefficient is 0.97. The sample frame length of the dataset is 5 seconds, and the corresponding sound segment labels are obtained by marking the obtained data, including sound event categories: non-snoring events (clearing throat, coughing, door pushing, collision, speaking, thunder), snoring events (normal snoring and obstructed snoring), and the label L includes the starting point and category of the sound segment. The snoring event category is marked as 1, and the non-snoring event category is marked as 0. The total number of final datasets is 4458.

[0090] After the data set is normalized according to formula (1), the Mel-cepstrum features of each audio waveform data are extracted according to formulas (2) and (3). The Mel-cepstrum with 128 Mel filters is extracted from the 5-second audio sample. The size of the obtained Mel-cepstrum feature is 128×157. The data set required for the model is constructed, including 5-second audio waveform data, Mel-cepstrum, and corresponding labels. Finally, the audio waveform data-Mel-cepstrum-label data set is obtained for network training.

[0091] Construct the FSDNet snoring detection network model, which consists of four parts: Audio Wave Encoder (AWE), Mel Cepstrum Encoder (MCE), Multi-domain Bidirectional Fusion Module (MDBFM) and Decoder. The specific steps include:

[0092] The audio waveform encoder is composed of four levels of waveform feature extraction modules of different scales connected in series from small to large scale. The input of the first waveform feature extraction module is a one-dimensional audio signal of length 80000×1 (80000 is the audio data length, 1 is the number of channels), and the output is an audio feature A1 of 2000×256; the input of the remaining three waveform feature extraction modules is the output of the previous waveform feature extraction module; the corresponding outputs A2~A4 are: 1000×256, 500×256, and 250×256 respectively. These four audio features are used as the output of the audio waveform encoder and input into the four SPADF modules respectively.

[0093] Each waveform feature extraction module is composed of three frequency domain attention submodules (FASM). Each frequency domain attention submodule is composed of a one-dimensional convolutional layer, a frequency domain self-attention layer and a Transformer encoding layer in series, which is used to extract audio features; its main feature is the addition of a frequency domain self-attention layer (FA), which specifically includes the following steps: cutting the features into 256 components according to the number of channels, performing a fast Fourier transform (FFT) on each component according to formula (4) to obtain the frequency domain component and splicing it on the channel, then passing the fully connected layer and Sigmoid output as the channel weight coefficient, and finally multiplying it with the original feature on the channel to obtain the attention-weighted one-dimensional feature.

[0094] The Mel Cepstral Encoder is composed of four levels of Mel Cepstral feature extraction modules of different scales connected in series. The input of the Mel Cepstral Encoder is an image with a resolution of 128×157×1. First, it passes through the Mel Cepstral preprocessing module, and the resolution becomes 32×40×256; then it is input into the first level Mel Cepstral feature extraction module without downsampling, and its output is a feature spectrum with a resolution of 32×40×256. The remaining Mel Cepstral feature extraction modules are all downsampled and the input is the output of the previous Mel Cepstral feature extraction module; finally, the four layers of multi-scale resolution feature maps are obtained, which are: 32×40×256, 16×20×512, 8×10×1024, and 4×5×2048.

[0095] The Mel-cepstral preprocessing module consists of two two-dimensional convolutional layers with a convolution kernel of 5×5, a deformable convolutional layer with a convolution kernel of 3×3, a batch normalization layer, an activation layer and a pooling layer connected in series.

[0096] Each Mel-cepstrum feature extraction module is composed of three deformable convolution residual sub-modules connected in series. The main feature of the deformable convolution residual sub-module is that it uses a deformable convolution layer, which can adaptively select patterns from the Mel-cepstrum that are helpful for downstream audio detection and classification as feature extraction, thereby improving the robustness of the representation of complex and diverse snoring cepstrum characteristics. The construction formulas are Equation (5), Equation (6) and Equation (7).

[0097] The multi-domain bidirectional fuser consists of four separate self-attention pooling dual fusion modules SAPDF. Each SAPDF module is used to fuse the one-dimensional audio features and Mel cepstral features of the same level; the output of the four-level Mel cepstral feature extraction module is adjusted to the same number of channels to obtain F1~F4: 32×40×256, 16×20×256, 8×10×256, 4×5×256. The input of each SPADF module is F1, F2, F3, F4 and the audio feature A output by the i-th waveform feature extraction module. i The output is the feature after fusion of the current scale. Finally, the output of the four-level multi-scale SAPDF module is Z1~Z4: 2000×256, 1000×256, 500×256, and 250×256.

[0098] The specific steps include:

[0099] The input of the self-attention pooling layer SAP is F1~F4, and the resolution feature maps of the other three scales are used as K, Q, and V to calculate the multi-scale attention; the V value is spliced ​​on the channel after global average pooling and global maximum pooling, and V′ is obtained after convolution. The results of global average pooling of K value and global maximum pooling of Q value are spliced ​​on the channel through the convolution layer, and the output after Sigmoid is multiplied by V' value. The result is Softmaxed and combined with the current scale feature map F i Multiply and add F i Add together to get the final output M i 'The calculation formula of the self-attention pooling layer is from equation (8) to equation (11). When calculating M1, Q = F2, K = F3, V = F4; when calculating M2, Q = F1, K = F3, V = F4; when calculating M3, Q = F1, K = F2, V = F4; when calculating M4, Q = F1, K = F2, V = F3; the outputs of M1 to M4 are: 32×40×256, 16×20×256, 8×10×256, 4×5×256.

[0100] Based on the output M of the self-attention pooling layer i After global average pooling, it is obtained through the convolution layer A with the current scale i With the same feature resolution, and A i Input two feature fusion layers, add the outputs of the two feature fusion layers to get the output of the SPADF module, where the formula of the feature fusion layer is formula (1.12).

[0101] The decoder includes four multi-layer perceptron modules MLP. The input of each MLP module is the multimodal representation of the audio data output by the corresponding level SAPDF module. The formulas of the MLP modules are equations (14) and (15).

[0102] The entire model is optimized and regressed based on the output of the decoder. The total loss of the entire FDSNet network model is the weighted sum of the classification loss, confidence loss and predicted regression box, which can reduce the missed detection of short snoring and accurately and quickly detect long snoring. The specific steps include: four-layer multi-scale feature mapping to obtain a one-dimensional predicted regression box {x, l, con} and a classification result. The predicted regression box contains the sound clip center point x, audio length offset l, and confidence con; classification loss L class With confidence loss L con The selected loss function is the binary cross entropy loss function, and the calculation formula is (16) and (17). The L of the one-dimensional prediction box and the one-dimensional target box is calculated. CIOU The loss is calculated as shown in equations (18) to (22). The total loss is calculated as loss = 0.3L class +0.2L con +0.5L CIOU .

[0103] The data set constructed in step S10 is input into the FDSNet network constructed in step S20 for parameter training. The optimizer is set to SGD during training, the initial learning rate is 1e-10, the learning rate change mode is cosine dynamic transformation, and the sample batch size is 32.

[0104] The learning rate change mode is cosine dynamic transformation, and the calculation formula is (24). The warm-up cycle is 100 and the total training cycle is 350.

[0105] The anchor box parameters can be determined by unsupervised clustering. The sizes of the 12 prior boxes obtained by the K-means clustering method are: 79999, 64480, 45526, 36906, 28713, 24820, 22194, 19652, 16754, 13332, 10111, 6430.

[0106] According to the set hyperparameters, the FDSNet network model is trained using the data set, and the model is optimized by the reverse transfer algorithm until the model training error is less than the preset threshold of 0.1 or the number of iterations is greater than the given value of 300. The training evaluation indicator is the mean average precision (mAP) of all classes, and the model with the highest mAP is selected as the optimal model.

[0107] Table 1: Model Optimal Performance Indicators

[0108]

[0109] Use the trained optimal model to detect the 10-minute audio file, divide the entire night audio file to be predicted into frames, and cut it into multiple 5-second audio segments. Use formula (1) to normalize the segmented segments, and then use formulas (2) and (3) to calculate the Mel cepstrum of the normalized segment. The model loads the trained optimal parameters, and the audio waveform data and the corresponding Mel cepstrum are sent to the model to form the data pair to be predicted, and the prediction result Y for each frame of data can be obtained. pad ∈{t s , t e ,c,y p}, where t s is the starting point of the audio segment being measured, t e is the end point, c is the confidence level, y p For category.

[0110] The detection results are then post-processed. First, the audio detection results with the snoring category and a confidence level greater than 0.9 are screened out from all the detection segments. Then, the audio detection results that are classified into different snoring segments but actually belong to the same snoring segment are fused. The splicing and fusion criteria include: (1) The categories of the two audio segments are both snoring and the confidence level is greater than 0.9, that is, the result of formula (25) is 1; (2) The time difference Δt between the two snoring segments is less than the threshold value t0=0.05s, that is, the result of formula (26) is 1; if the above two criteria are met, that is, the results of formula (25) and formula (26) are both 1, then they are regarded as the same snoring segment and spliced ​​into a complete snoring segment; otherwise, they are still regarded as two independent snoring segments. Finally, the starting points of all snoring segments in the 10-minute audio file are obtained. Figure 5 This is a visualization image of the detection results. The result is assigned a value of 1 for a snoring segment, and 0 for a non-snoring segment. Comparing the actual snoring segments with the detection results, the FDSNet model has high accuracy and good reliability when used to detect snoring segments in long-term sleep audio.

Claims

1. This patent proposes a method for detecting snoring all night long based on the dual fusion of frequency domain attention and self-attention pooling. This method is mainly implemented through the FDSNet (FASM-DCRSM-SAPDF-Net) network, which can realize the continuous detection of snoring in long-term sleep sound data all night long. It is characterized by: The specific steps include: S10: Dataset construction: The audio files of the whole night's sleep sounds recorded by the high-sensitivity microphone are divided into frames to produce a mixed data set of snoring events and non-snoring events. The sample frame length of the data set is T seconds, and the corresponding sound event types of the obtained data are marked as: non-snoring events (throat clearing, coughing, door pushing, collision, talking, thunder), snoring events (normal snoring and obstructed snoring); the data set is normalized to the range of [0, 1] according to formula (1): Where x is one-dimensional audio waveform data; The Mel cepstrum of the normalized waveform is calculated according to equations (2) and (3); the data set required for the model is constructed, including T seconds of audio waveform data X, Mel cepstrum M, and the corresponding mark L, and finally a data set consisting of {X, M, L} is obtained for network training; where |X(n)| is the spectrum amplitude, H m (k, n) is the response of the mth Mel filter spectrum to the spectral component X(n) at frequency index k, log_S(m, k) is the logarithmic magnitude spectrum, MFCC c (m) is the c-th Mel-cepstral coefficient of the m-th frame, K is the number of Mel filter banks, and c is the index of the Mel-cepstral coefficient; S20: Construct a whole-night snoring event detection and localization model FDSNet (FASM-DCRSM-SAPDF-Net) based on dual fusion of frequency domain attention and self-attention pooling; the network consists of four parts: Audio Wave Encoder (AWE), Mel Cepstrum Encoder (MCE), Multi-domain Bidirectional Fusion Module (MDBFM) and Decoder; The input of the audio waveform encoder (AWE) is one-dimensional audio waveform data. The audio waveform encoder is composed of four levels of waveform feature extraction modules connected in series. The audio data is downsampled and features are extracted step by step to obtain four layers of multi-scale audio features. Each waveform feature extraction module includes three frequency domain self-attention sub-modules (FASM). The frequency domain self-attention sub-module (FASM) adds a frequency domain self-attention layer FA (Frequency Attention, FA) between the one-dimensional convolution layer and the Transformer encoding layer, which can enhance the frequency bands in the waveform data that are effective for detection and classification, and can better extract the inherent features of snoring for easy detection. The input of the Mel Cepstral Encoder (MCE) is a Mel Cepstral graph, which includes a Mel Cepstral preprocessing module and four Mel Cepstral feature extraction modules. Similar to the audio waveform encoder, the Mel Cepstral Encoder also connects four levels of Mel Cepstral feature extraction modules in series to downsample the Mel Cepstral coefficient graph. The output of the MCE is four layers of feature graphs of different scales. Each level of the Mel Cepstral feature extraction module includes three Deformable Conv Residual Sub-Modules (DCRSM), which use deformable convolution layers to adaptively select effective feature patterns of the Mel Cepstral to extract features from the Mel Cepstral, thereby improving the encoder module's ability to extract complex audio pattern features. The multi-domain bidirectional fuser (MDBFM) can realize the fusion of audio waveform features and Mel-cepstral features so that the network can learn the characteristics of snoring more accurately. It includes four levels of self-attention pooling and double fusion modules (SAPDF), which can obtain four layers of audio multimodal representation data of different scales; each level of SAPDF module consists of a self-attention pooling layer (SAP) and two feature fusion layers (Fusion Block); the input of the self-attention pooling layer is the four-layer feature map of different scales output by the Mel-cepstral encoder, and the resolution feature maps of the other three scales are used as K, Q, and V to calculate the multi-scale attention, which can combine the contextual information of different scales and fuse with the pixel features of the current scale, which can reduce the loss of features during downsampling; The inputs of the two feature fusion layers are the output of SAP and the output of the waveform feature extraction module of the corresponding level. The features fused after the weights are calculated by the audio waveform features and the weights are calculated by the Mel-frequency cepstrum features. The outputs of the two feature fusion layers are added to obtain the output of the final SPADF module. The decoder is composed of four multi-layer perceptron (MLP) modules, each of which contains three fully connected layers. The input of this module is the multimodal representation of the audio data obtained by the output of the corresponding level SAPDF module. Based on the output of the four multi-layer perceptron modules, the entire model is optimized and regressed to obtain a one-dimensional predicted regression box and category. The predicted regression box contains the center point of the sound clip, the audio length offset and the confidence. The loss function is obtained by weighting the CIOU loss, confidence loss and classification loss of the predicted regression box. S30: Use the data set constructed in S10 to train the FDSNet network model constructed in S20; set hyperparameters such as learning rate, learning rate change mode, sample batch size, and acoustic signal anchor box size and number during training; The anchor frame parameters can be determined by unsupervised clustering. The specific method is as follows: the size of the anchor frame is obtained by using the K-means clustering algorithm, the starting point information of all samples in the data set is converted into duration, k frames are randomly generated using the K-means clustering algorithm, the similarity between each frame and the original frame is calculated and the similarity is 1-IOU, all frames are divided into k categories, and the k frames are updated using the mean of the classification, the similarity is repeatedly calculated and the k frames are updated, and the algorithm is stopped when the change value between the new frame and the original frame is small; According to the set hyperparameters, the FDSNet network model is trained using the data set, and the model is optimized through the reverse transfer algorithm until the model training error is less than the preset threshold L threshold Or the number of iterations is greater than the given value T m ; S40: Use the trained optimal model to detect the whole night audio file, divide the whole night audio file to be predicted into frames, and cut it into multiple audio segments of T seconds in length; use formula (1) to perform normalization preprocessing on the segmented segments, and then use formula (2) and formula (3) to calculate the Mel-frequency cepstrum of the normalized segments; load the model with the optimal parameters after training, and send the audio waveform data and the corresponding Mel-frequency cepstrum to form the data pair to be predicted into the model to obtain the prediction result of each frame of data; then post-process the results, merge the segments that are divided into different snoring segments but actually belong to the same snoring segment, and finally obtain the time interval and event category of all snoring in the whole night audio file.

2. According to claim 1, a method for detecting snoring all night long based on dual fusion of frequency domain attention and self-attention pooling is mainly implemented through the FDSNet network, and the waveform data in the all night long sleep sound can be extracted by using an audio waveform encoder, which is characterized by: The audio waveform encoder described in step S20 is composed of four levels of waveform feature extraction modules of different scales connected in series in ascending order of scale; the input of the first waveform feature extraction module is a one-dimensional audio signal of length L×1 (L is the audio data length, 1 is the number of channels), and the output is an audio feature A1 of L / 40×C, where C represents the number of channels; the input of the remaining three waveform feature extraction modules is the output of the previous waveform feature extraction module; the corresponding outputs A2 to A4 are: L / 80×C, L / 160×C, L / 320×C, respectively; Each waveform feature extraction module is composed of three frequency domain attention submodules (FASM) connected in series; each frequency domain attention submodule is composed of a one-dimensional convolutional layer, a frequency domain self-attention layer and a Transformer encoding layer connected in series to extract audio features; its main feature is the addition of a frequency domain self-attention layer (Frequency Attention, FA), which specifically includes the following steps: The feature is cut into C components according to the number of channels (C is the number of channels), and each component is subjected to Fast Fourier Transform (FFT) according to formula (4) to obtain the frequency domain component and concatenate it on the channel. Then, it is output as the channel weight coefficient through the fully connected layer and Sigmoid, and finally multiplied with the original feature on the channel to obtain the attention-weighted one-dimensional feature.

3. According to claim 1, a method for detecting snoring all night long based on dual fusion of frequency domain attention and self-attention pooling is mainly implemented through the FDSNet network, and the Mel cepstrum encoder can be used to extract features from the Mel cepstrum data in the sleep sound all night long, and its characteristics are: The Mel-cepstral encoder described in step S20 is correspondingly composed of four levels of Mel-cepstral feature extraction modules of different scales connected in series in sequence; the input of the Mel-cepstral encoder is an image with a resolution of H×W×1 (H and W are the height and width of the Mel-cepstral image, respectively, and 1 is the number of channels), which first passes through a Mel-cepstral preprocessing module, and the resolution becomes H / 4×W / 4×C; then the first-level Mel-cepstral feature extraction module without downsampling is input, and its output is a feature spectrum with a resolution of H / 4×W / 4×C, and the remaining Mel-cepstral feature extraction modules are all downsampled and the input is the output of the previous Mel-cepstral feature extraction module; finally, four layers of multi-scale resolution feature maps are obtained, namely: H / 4×W / 4×C, H / 8×W / 8×2C, H / 16×W / 16×4C, and H / 32×W / 32×8C, where C is the number of channels; The Mel-Cepstral preprocessing module is composed of two 2D convolutional layers with 5×5 convolution kernels, a deformable convolutional layer with 3×3 convolution kernels, a batch normalization layer, an activation layer, and a pooling layer connected in series. Each Mel-cepstrum feature extraction module is composed of three deformable convolution residual submodules connected in series. The main feature of the deformable convolution residual submodule is that it uses a deformable convolution layer, which can adaptively select patterns from the Mel-cepstrum that are helpful for downstream audio detection and classification as feature extraction, and improve the robustness of the representation of complex and diverse snoring cepstrum characteristics. The specific construction method of the deformable convolution residual submodule is: f rbc (x)=ReLU(b(ζ(x))) (5) y=ReLU(b(g(f rbc2 (f rbc1 (x))))+Δx) (6) ReLU(x)=max(0,x) (7) Where x is the input of the deformable convolution residual submodule, y is the output of the DCRSM module, Δx is the residual term, ζ is the deformable convolution, b is batch normalization, ReLU is the rectified linear unit, and f rbc1 and f rbc2 They are respectively formula (5), where the step sizes of the deformable convolution are 1 and 2 respectively.

4. According to claim 1, a method for detecting snoring all night based on dual fusion of frequency domain attention and self-attention pooling is mainly implemented through the FDSNet network, and a multi-domain bidirectional fuser can be used to realize the fusion of audio domain waveform features and frequency domain Mel cepstrum features, and its characteristics are: In step S20, the multi-domain bidirectional fuser is composed of four separate waveform domain and Mel cepstral domain bidirectional fusion modules SAPDF based on the self-attention pooling mechanism; each SAPDF module is used to fuse the one-dimensional audio features and Mel cepstral features of the same level; the output of the four-level Mel cepstral feature extraction module is adjusted to the same number of channels to obtain F1~F4: H / 4×W / 4×C, H / 8×W / 8×C, H / 16×W / 16×C, H / 32×W / 32×C, and the input of each SPADF module is F1, F2, F3, F4 and the audio feature A output by the i-th waveform feature extraction module i , the output is the feature after fusion of the current scale, and finally the output of the four-level multi-scale SAPDF module is Z1~Z4; its main feature is the design based on the self-attention pooling layer and the feature fusion layer, which can capture the audio features more accurately, including the following steps: The input of the self-attention pooling layer SAP is F1~F4, and the resolution feature maps of the other three scales are used as K, Q, and V to calculate the multi-scale attention; the V value is spliced ​​on the channel after global average pooling and global maximum pooling, and V′ is obtained after convolution. The results of global average pooling of K value and global maximum pooling of Q value are spliced ​​on the channel through the convolution layer, and the output after Sigmoid is multiplied by V' value. The result is Softmaxed and combined with the current scale feature map F i Multiply and add F i Add together to get the final output M i , the calculation formula of the self-attention pooling layer is as follows: f(x,y)=g(GAP(x)||MAP(x)) (8) Where || represents splicing on the channel, g is the convolution layer, and σ is the Sigmoid function; When calculating M1, Q = F2, K = F3, V = F4; When calculating M2, Q = F1, K = F2, V = F4; When calculating M3, Q = F1, K = F2, V = F4; When calculating M4, Q = F1, K = F2, V = F3; Based on the output M of the self-attention pooling layer i After global average pooling, it is obtained through the convolution layer A with the current scale i With the same feature resolution, and A i Input two feature fusion layers, add the outputs of the two feature fusion layers to get the output of the SPADF module, where the formula of the feature fusion layer is formula (1.12): Among them, Both ρ and μ are one-dimensional convolutions with a kernel of 1, and N is a normalization factor.

5. According to claim 1, a method for detecting snoring all night based on dual fusion of frequency domain attention and self-attention pooling is mainly implemented through the FDSNet network, and the output of the decoder can be used to realize snoring event location and classification, and its characteristics are: In step S20, the decoder includes four multi-layer perceptron (MLP) modules, each of which includes three fully connected layers, and the activation function is ReLU, and its input is the multimodal representation of the audio data obtained by the output of the corresponding level SAPDF module; the formula of the MLP module is as follows: f mlp =ReLU(ψ(x)) (14) O(x)=f mlp3 (f mlp2 (f mlp1 (x))) (15) The module input is x, ψ is the linear layer, and ReLU is the rectified linear unit; The entire model is optimized and regressed based on the output of the decoder. The total loss of the entire FDSNet network model is the weighted sum of classification loss, confidence loss and prediction regression loss. This can reduce the missed detection of short snoring sounds and accurately and quickly detect long snoring sounds. The specific steps include: The four-layer multi-scale feature mapping obtains a one-dimensional prediction regression box {x, l, con} and a classification result. The prediction regression box contains the center point x of the sound clip, the audio length offset l, and the confidence con; the classification loss L class With confidence loss L con The selected loss function is the binary cross entropy loss function, and the calculation formula is (16) and (17). The L of the one-dimensional prediction box and the one-dimensional target box is calculated. CIOU The loss is calculated as shown in equations (18) to (22): Where N is the number of samples, y represents the true label of the sample, x represents the predicted value of the sample, and σ represents the Sigmoid function; Where N is the number of samples, y represents the true confidence of the sample, x represents the predicted confidence of the sample, and σ represents the Sigmoid function; L CIOU (T,Y)=1-IOU(T,Y)+d(T,Y)+αν(T,Y) (18) α=ν / (1-IOU+v) (22) Where IOU is the intersection-over-union ratio of two boxes, T e and T s is the end point and starting point of the real box T, Y e and Y s is the end point and starting point of the prediction box Y, eps is 10 -16 , αν and l(T,Y) can further adjust the loss; The total loss is calculated as (23), which is the classification loss L class , confidence loss L con , prediction box and target box loss L CIOU The weighted sum, where the values ​​of λ1, λ2, and λ3 range from 0 to 1 and their sum is 1. loss=λ1L class +λ2L con +λ3L CIOU (23) 6. According to claim 1, a method for detecting snoring all night based on dual fusion of frequency domain attention and self-attention pooling is mainly implemented through the FDSNet network, and the model parameters are optimized by obtaining a priori frames and setting appropriate hyperparameters, characterized in that: The step S30 specifically includes: Use the dataset constructed by S10 to train the FDSNet network model constructed by S20; set hyperparameters such as optimizer, learning rate, learning rate change method, sample batch size, and acoustic signal anchor box size and number during training; The learning rate changes in a cosine dynamic transformation, and the overall logic is that the coefficients maintain linear growth before the warm-up period and during the warm-up period t warm Then the corresponding cosine function is used for transformation, and the calculation formula is (24): where t c is the current training cycle, t warm is the preheating period, t all is the total training cycle, N c is the number of cycles; Among them, the anchor frame parameters can be determined by unsupervised clustering; the specific method is: using the K-means clustering algorithm to obtain the size of the anchor frame, converting the starting point information of all samples in the data set into duration, using the K-means clustering algorithm to randomly generate k frames, calculating the similarity between each frame and the original frame and the similarity is 1-IOU, dividing all frames into k categories and using the mean of the classification to update the k frames, repeatedly calculating the similarity and updating the k frames, and stopping the algorithm when the change value between the new frame and the original frame is small; According to the set hyperparameters, the FDSNet network model is trained using the data set, and the model is optimized through the reverse transfer algorithm until the model training error is less than the preset threshold L threshold Or the number of iterations is greater than the given value T m The training evaluation indicator is the mean Average Precision (mAP) of all classes, and the model with the highest mAP is selected as the optimal model.

7. According to claim 1, a method for detecting snoring all night based on dual fusion of frequency domain attention and self-attention pooling, characterized in that: The step S40 specifically includes the following steps: Use the trained optimal model to detect the whole night audio file, divide the whole night audio file to be predicted into frames, and cut it into multiple audio segments with a duration of T seconds; use formula (1) to perform normalization preprocessing on the segmented segments, and then use formulas (2) and (3) to calculate the Mel cepstrum of the normalized segments; load the model with the optimal parameters after training, and send the audio waveform data and the corresponding Mel cepstrum to form the data pair to be predicted into the model, and the prediction result Y of each frame of data can be obtained. pad ∈{t s , t e ,c,y p }, where t s is the starting point of the audio segment being measured, t e is the end point, c is the confidence level, y p For categories; Then, the detection results are post-processed to first filter out all the detection segments that are classified as snoring and have a confidence level greater than c. threshold The audio detection results are then merged into different snoring segments but actually belong to the same snoring segment. The splicing and fusion criteria include: (1) Both audio segments are snoring and the confidence level exceeds c threshold , that is, the result of formula (25) is 1; (2) the time difference Δt between the two snoring segments is less than the threshold value t0, that is, the result output of formula (26) is 1; if the above two criteria are met, that is, the results of formula (25) and formula (26) are both 1, then they are regarded as the same snoring segment and spliced ​​into a complete snoring segment, otherwise they are still regarded as two independent snoring segments, and finally the starting points of all snoring segments in the whole night audio file are obtained.

Images