A snoring sound detection method and device based on multi-model superposition
By combining a multi-model stacking method with an autoencoder model, the problem of false detection of snoring under background noise interference by a single detection model is solved, achieving higher detection accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHANGZHOU SOLEX SMART HOME CO LTD
- Filing Date
- 2023-03-10
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, single-detection models are easily affected by background noises such as music or human voices when identifying snoring, resulting in a high false detection rate and difficulty in accurately detecting snoring.
A multi-model overlay method is adopted. First, the background sound is detected as silent by a background sound multi-classification model. Then, the snoring sound is detected by a snoring sound binary classification model. The MFCC features are encoded and decoded by an autoencoder model to remove noise. The accuracy of detection is ensured by combining the decibel value and time interval.
It significantly reduced the false detection rate caused by background noise interference, improved the accuracy of snoring detection, and enhanced the recognition ability in complex environments.
Smart Images
Figure CN116312634B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of snoring detection technology, and in particular to a snoring detection method and device based on multi-model superposition. Background Technology
[0002] Obstructive sleep apnea-hypopnea syndrome (OSAHS) is a common condition that is easily overlooked. It is primarily caused by apnea-hypopnea or apnea during sleep, which can lead to daytime sleepiness and recurrent apnea. A prominent characteristic is often snoring during sleep; therefore, snoring detection during sleep can effectively identify potential OSAHS.
[0003] Existing technologies generally employ single-detection models for snoring identification. Reference document 1, "A Snoring Monitoring Method and System Based on Deep Learning Algorithm and Corresponding Electric Bed Control Method and System" (Application No. 202110803746.6), discloses using a pre-trained deep learning model for deep feature extraction and classification to categorize each audio segment into audio slices containing snoring and those not containing snoring. Reference document 1 directly uses a neural network to judge MFCC feature data without identifying background noise. Reference document 2, "A Snoring Signal Recognition Method" (Application No. 201910834050.2), discloses using a k-means clustering algorithm to identify snoring and non-snoring sounds. Both reference documents 1 and 2 exhibit a certain false detection rate. Summary of the Invention
[0004] To address the aforementioned technical problems, this invention provides a snoring detection method and apparatus based on multi-model superposition. Snoring detection is only performed after the background sound is detected as silent, reducing false detections caused by background sounds such as music or human voices, and improving the accuracy of detection.
[0005] The present invention adopts the following technical solution:
[0006] On the one hand, a snoring detection method based on multi-model superposition includes:
[0007] Acquire audio data including the current frame and the preset frame before the current frame as background audio data;
[0008] Extract MFCC features from background noise audio data;
[0009] The MFCC features are input into a trained background noise multi-classification model for detection, and the detection results are obtained.
[0010] If the detection result is silence, the MFCC feature is input into the trained snoring binary classification model to detect whether the current frame audio data includes snoring.
[0011] Preferably, acquiring audio data including the current frame and the preset frame preceding the current frame as background audio data specifically includes:
[0012] Get the audio data of the current frame;
[0013] The audio data of the current frame is concatenated with the audio data in the frame buffer queue and then inserted into the frame buffer queue; the frame buffer queue adopts the first-in-first-out principle and can store the audio data of the first preset frame;
[0014] Determine whether the cache queue contains audio data of the first preset frame. If so, use the audio data in the cache queue as background audio data.
[0015] Preferably, if the current frame of audio data is detected to include snoring, the method further includes:
[0016] Calculate the decibel value of the current frame's audio data;
[0017] Calculate the time interval between the current frame and the previous frame that was identified as snoring;
[0018] Update the information in the current storage list; the information includes timestamps and decibel values;
[0019] Determine whether the decibel value is within a preset decibel range and whether the time interval is within a preset time range. If both conditions are met, indicate that snoring has been detected.
[0020] Preferably, before inputting the MFCC features into the trained snoring binary classification model, the method further includes:
[0021] The MFCC features are input into a trained autoencoder model to encode and decode the features to obtain the reconstructed MFCC features.
[0022] The MFCC features are input into a trained snoring binary classification model to detect whether the current frame of audio data includes snoring, specifically including:
[0023] The reconstructed MFCC features are input into a trained snoring binary classification model to detect whether the current frame of audio data includes snoring.
[0024] Preferably, the autoencoder model is a neural network model comprising an input layer, an encoder layer, a decoder layer, and an output layer;
[0025] The autoencoder model uses the equal Euclidean distance as a loss metric, as follows:
[0026]
[0027] Where, x j This represents the raw data input to the model. This represents the prediction result output by the model.
[0028] Preferably, the encoder layer includes several interconnected convolutional layers and pooling layers, and the decoder layer includes several interconnected convolutional layers and upsampling layers.
[0029] Preferably, the background sound multi-classification model adopts the AudioLiteNet neural network model, which includes several interconnected blocks; the blocks are connected directly or by skipping layers; the direct connection is to perform data pipeline streaming calculation using predefined operators in the block; the skipping layer connection is to add features together.
[0030] Preferably, the background noise multi-classification model uses five frames of audio data as background noise audio data. The input of the background noise multi-classification model is the extracted MFCC features, and the output of the background noise multi-classification model is the confidence score of music, human voice and silence. The class with the highest confidence score is taken as the detection result.
[0031] Preferably, the snoring binary classification model adopts the AudioLiteNet neural network model, which includes several interconnected blocks; the blocks are connected directly or by skipping layers; the direct connection is to perform data pipeline streaming calculation using predefined operators in the block; the skipping layer connection is to add features together.
[0032] Preferably, the input to the snoring binary classification model is the extracted MFCC features, and the output of the snoring binary classification model is the confidence score of snoring and non-snoring, with the class with the highest confidence score being taken as the detection result.
[0033] Preferably, both the background noise multi-classification model and the snoring binary classification model employ a binary cross-entropy loss function, as follows:
[0034]
[0035] Where N represents the total number of nodes to be measured, i.e., the total number of categories, and y i This indicates that the actual labeled value of the i-th node represents the correct classification. This represents the classification value predicted by the neural network for the i-th node.
[0036] On the other hand, a snoring detection device based on multi-model superposition includes:
[0037] The background sound audio data acquisition module is used to acquire audio data including the current frame and the previous preset frame of the current frame as background sound audio data.
[0038] The MFCC feature extraction module is used to extract MFCC features from background noise audio data.
[0039] The background noise multi-classification model detection and acquisition module is used to input the MFCC features into the trained background noise multi-classification model for detection and to obtain the detection results.
[0040] The snoring detection module is used to input the MFCC features into a trained snoring binary classification model when the detection result is silence, and to detect whether the current frame audio data includes snoring.
[0041] The present invention has the following beneficial effects:
[0042] (1) The present invention detects the background sound type through a background sound multi-classification model. After detecting that the background sound is a silent sound, it uses a snoring binary classification model to detect snoring or non-snoring sounds, which can reduce false detections caused by background sounds being music or human voices and improve the detection accuracy.
[0043] (2) After detecting that the background sound is silent, the present invention uses an autoencoder model to encode and reconstruct the MFCC feature map, focusing attention on the feature map information associated with snoring, and removing irrelevant features as noise, which can achieve the effect of feature encoding on the feature map and removing certain noise.
[0044] (3) The background sound audio data used for detecting background sound in this invention is longer than the audio data used for detecting snoring, which can further increase the detection accuracy.
[0045] (4) After detecting snoring using the snoring binary classification model, the present invention further determines the range of decibel values of the current frame audio data and whether the time interval between the current frame and the previous frame that was identified as snoring is within a preset time range. If both conditions are met, snoring is detected, thus increasing the detection accuracy.
[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Attached Figure Description
[0047] Figure 1 This is a flowchart of a snoring detection method based on multi-model superposition according to an embodiment of the present invention;
[0048] Figure 2 This is a schematic diagram of audio data acquisition for the snoring binary classification model according to an embodiment of the present invention;
[0049] Figure 3 This is a schematic diagram of the feature extraction layer of the snoring binary classification model according to an embodiment of the present invention;
[0050] Figure 4 This is a schematic diagram of the output layer structure of the snoring binary classification model according to an embodiment of the present invention;
[0051] Figure 5 This is a schematic diagram of the overall structure of the snoring binary classification model according to an embodiment of the present invention;
[0052] Figure 6 This is a schematic diagram illustrating the connection relationship between two blocks in the snoring binary classification model of this invention.
[0053] Figure 7 This is a schematic diagram of audio data acquisition for a background sound multi-classification model according to an embodiment of the present invention;
[0054] Figure 8 This is a schematic diagram of the overall structure of the background sound multi-classification model according to an embodiment of the present invention;
[0055] Figure 9 This is a schematic diagram of the overall structure of the autoencoder model according to an embodiment of the present invention;
[0056] Figure 10 This is a schematic diagram of the autoencoder model in an embodiment of the present invention.
[0057] Figure 11 This is a schematic diagram of the structure of MFCC-AENet according to an embodiment of the present invention;
[0058] Figure 12 This is a diagram illustrating the downsampling process of MFCC-AENet in an embodiment of the present invention;
[0059] Figure 13 This is a diagram illustrating the upsampling process of MFCC-AENet in an embodiment of the present invention;
[0060] Figure 14 This is a detailed flowchart of single-frame snoring detection according to an embodiment of the present invention;
[0061] Figure 15 This is a structural block diagram of a snoring detection device based on multi-model superposition according to an embodiment of the present invention. Detailed Implementation
[0062] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0063] In the description of this invention, it should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0064] In the description of this invention, it should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0065] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the step identifiers S101, S102, S103, etc. are used only for convenience of description and do not indicate the execution order. The corresponding execution order can be adjusted.
[0066] See Figure 1 As shown, this snoring detection method based on multi-model superposition includes:
[0067] S101, acquire audio data including the current frame and the previous preset frame of the current frame as background sound audio data;
[0068] S102, extract MFCC features from background noise audio data;
[0069] S103, Input the MFCC features into the trained background noise multi-classification model for detection and obtain the detection results;
[0070] S104, if the detection result is silence, input the MFCC feature into the trained snoring binary classification model to detect whether the current frame audio data includes snoring.
[0071] The above-described S101-S104 steps of the snoring detection method based on multi-model superposition in this embodiment can be implemented using anti-snoring pillow devices and / or smart terminal devices, as long as these devices have sound acquisition functions (such as microphones) and / or processor functions (such as microcontrollers). Specifically, one terminal can be responsible for acquisition (such as the anti-snoring pillow) and another terminal can be responsible for processing (such as a mobile phone terminal), or both acquisition and processing can be performed by a single terminal. This embodiment does not impose any specific limitations. Furthermore, the smart terminal device can be a mobile phone, tablet computer, wearable device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc. This embodiment does not impose any limitations on the specific type of terminal device.
[0072] In this embodiment, the snoring binary classification model, the background noise multi-classification model, and the autoencoder model need to be trained first.
[0073] (1) Training of the snoring binary classification model.
[0074] (1.1) Training data collection.
[0075] Sleep audio was collected from individuals or subjects suspected of having OSAHS using the microphone on the device. The audio was then manually segmented and cut to separate audio data containing snoring from non-snoring sounds. The audio was then cut into 1-second segments and stored in batches to form a dataset for snoring audio classification.
[0076] In this embodiment, a 16kHz PCM mono signal source is acquired for 1 second each time, and the signal in this format is defined as "one frame", that is, one frame is equal to 1 second of audio data signal, and similarly, five frames are 5 seconds of audio data signal.
[0077] For details, see Figure 2 As shown, the snoring binary classification model uses a 1-second input. Therefore, all data in the dataset is segmented into 1-second segments. The segmentation mode is continuous segmentation with no intervals, based on a 1-second window length. For example, a 30-second audio segment can be segmented into 30 1-second audio segments. The acquired audio data format is a 16kHz single-channel PCM signal, which can be considered as 16,000 Int16 signal quantities per second. The array length of 16,000 can be used as the window length for truncating 1-second audio. Through the operations described above, a batch of 1-second audio files can be obtained as the dataset for the snoring binary classification model.
[0078] (1.2) Training data calibration.
[0079] After the dataset is obtained, the audio files need to be labeled. Specifically, the labeling method involves manually listening to the audio files and classifying and labeling the content heard.
[0080] The calibration targets are as follows:
[0081] Snoring sounds during sleep (including snoring or breathing sounds emanating from the nasal cavity, throat cavity, and other areas);
[0082] Non-snoring sounds (including human voices, talking sounds, event sounds, noises, and all other sounds other than snoring).
[0083] The manually selected and labeled snoring audio classification dataset is divided into training, validation, and test sets in an 8:1:1 ratio (the specific ratio may vary depending on needs). The validation set is used during the training process to iteratively validate the model, while the test set is used after training to test the model's accuracy.
[0084] (1.3) Design of a binary classification model for snoring
[0085] An AudioLiteNet neural network model is constructed. The snoring classification dataset divided in (1.2) is fed into the neural network model for iterative training, and a snoring binary classification algorithm model with 1 second as input is obtained after training.
[0086] In this embodiment, the network structure of the AudioLiteNet neural network model includes a large number of stacked convolutional layers (such as 16 block layers) for channel splitting and channel fusion. The purpose is to reduce the amount of computation and improve the inference speed while ensuring accuracy when the model is implemented on a smart terminal device.
[0087] See Figure 3 As shown, the feature extraction layers of the network structure are all composed of deep-wise convolution, point-wise convolution, and residual modules, which are stacked together. Before performing deep convolution, a 1x1 convolution kernel is used to perform dimensionality increase on the feature channels to achieve the effect of channel expansion.
[0088] The final output layer of the network structure uses a 1x1 convolutional layer for channel mapping and dimensionality reduction, followed by flattening and then regression using a fully connected layer. Finally, a Softmax activation function is applied to map the classification confidence, where n is the number of output nodes (i.e., the number of categories), and x... i For the input value of the i-th node, the denominator is... The numerator represents the sum of all classification mapping results. Let represent the classification mapping result of the current i-th node. This formula can be used to map the classification result to the interval [0~1], and the summation is 1, thus better conforming to the probability distribution. The Softmax activation function is as follows:
[0089]
[0090] See the schematic diagram of the output layer after combination. Figure 4 As shown.
[0091] The AudioLiteNet snoring binary classification model takes MFCC feature maps as input and outputs binary classification confidence scores. For the overall network composition, see [link to full description]. Figure 5 As shown.
[0092] The AudioLiteNet network structure consists of blocks, and there are two main types of connections between blocks: direct connections and layer-skipping connections. Direct connections utilize predefined operators (ops) within the block to perform pipelined computation, where data flows through adjacent layers to obtain the computation result. See also Figure 6 The diagram illustrates the connection relationship between Blocks 4 and 5 in the overall network structure. Analyzing Block 5 reveals that the direct connection operator alters the number of feature channels in the original data. Skip-layer connections essentially add features together, so maintaining consistent dimensionality between the two connected data is crucial. Therefore, the direct connection operator uses a pointwise operator to convolve the altered data channel count back to the original input dimensionality, thus fulfilling the condition for skip-layer connections. Skip-layer connections represent an identity transformation pattern. When the model has many layers, this method ensures feature reuse and reduces the gradient vanishing problem during backpropagation in training.
[0093] The loss function used to calculate the error between the binary classification training result and the true result during training is the Binary Cross-Entropy Loss Function, as follows:
[0094]
[0095] Where N represents the total number of nodes to be measured, i.e., the total number of categories, and y i This indicates that the actual labeled value of the i-th node represents the correct classification. This represents the classification value predicted by the neural network for the i-th node.
[0096] After 150 rounds of iterative training, the snoring binary classification model achieved high accuracy when tested with pre-defined tests.
[0097] (2) Training of background noise multi-classification model.
[0098] (2.1) Training data collection.
[0099] A large number of datasets matching sleep scenarios were collected and the audio was manually segmented and trimmed to select audio classification datasets containing music, speech, and silence. The audio was then trimmed into 5-second segments and stored in batches to form a background sound classification dataset.
[0100] In this embodiment, a 16kHz PCM mono signal source is acquired for 1 second each time, and the signal in this format is defined as "one frame", that is, one frame is equal to 1 second of audio data signal, and similarly, five frames are 5 seconds of audio data signal.
[0101] For details, see Figure 7 As shown, the background noise multi-classification model uses a 5-second input. To achieve a certain accuracy, the background noise data requires extracting longer speech information features. To ensure data reusability and data volume, the segmentation mode does not use continuous segmentation; instead, a sliding window is used to cross-segment the array storing PCM signals. The length of the sliding window is 16000 * 5 samples, i.e., 5 seconds of data length, and the sliding step is 16000 * 2.5 samples, i.e., 2.5 seconds of data length. Through the operations described above, a batch of 5-second audio files can be obtained as the dataset for the background noise multi-classification model.
[0102] (2.2) Training data calibration.
[0103] After the dataset is obtained, the audio files need to be labeled. Specifically, the labeling method involves manually listening to the audio files and classifying and labeling the content heard.
[0104] The calibration targets are as follows:
[0105] Music audio (external music releases, including music-related audio such as opera, pop music, rock music, and sleep aid music);
[0106] Human voices (including speaking, reading aloud, discussing, telephone calls, and other voice-related activities);
[0107] Silence (sounds in a quiet scene, or sounds in a scene below a certain decibel level).
[0108] The background audio classification dataset, after manual screening and labeling, is divided into training, validation, and test sets in a 9:0.5:0.5 ratio (the specific ratio may vary depending on needs). The validation set is used during the training process to iteratively validate the training results, while the test set is used after training to test the model's accuracy.
[0109] (2.3) Design of a multi-classification model for background noise
[0110] An AudioLiteNet neural network model is constructed. The background sound audio classification dataset divided in (2.2) is fed into the neural network model for iterative training, and a trained background sound multi-classification model with 5 seconds as input can be obtained.
[0111] In this embodiment, the background noise multi-classification model is consistent with the snoring binary classification model in (1). Both are trained using AudioLiteNet to build a neural network model. Unlike the snoring binary classification model, the background noise multi-classification model uses a 5-second background noise dataset as input. Therefore, the input of the network structure needs to extract and transform the audio signal value using 500x39 MFCC features, while the output of the network structure outputs 3 neurons as classification confidence. The function for mapping the classification confidence is softmax. The network structure of the background noise multi-classification model is shown in [reference]. Figure 8 As shown.
[0112] After 200 rounds of iterative training, the background sound multi-classification model achieved high accuracy when tested on a pre-divided test set.
[0113] (3) Autoencoder model training.
[0114] See Figure 9 As shown in the figure, in this embodiment, an autoencoder model MFCC-AENet is also built to reconstruct the MFCC features of snoring and perform noise reduction processing.
[0115] MFCC-AENet is derived from a neural network based on an unsupervised data dimension encoding / decoding structure. It is trained by extracting MFCC feature maps from a large amount of snoring audio, enabling the model to encode and decode these feature maps for feature reconstruction. The goal is to focus the model's attention on feature map information associated with snoring and remove irrelevant features as noise, achieving the effect of feature encoding on the feature map while removing some noise. Specifically, it includes the following steps:
[0116] (3.1) Construct an autoencoder model and define the input, encoder / decoder, and output layers of the neural network. Since the MFCC feature map information is relatively refined, the hidden layer neural network such as the encoder and decoder does not need a very deep network structure. The encoder contains downsampling calculation units of EBlock blocks composed of 3 convolutional layers plus pooling layers, and the decoder contains upsampling layer calculation units of DBlock blocks composed of 3 convolutional layers plus sampling layers.
[0117] To ensure that the input and output data shapes of the autoencoder structure are consistent, the data size of the input layer needs to be defined as a 1:1 width-to-height ratio. In the case of the MFCC feature map shape for the aforementioned snoring binary classification, which is 39*100, the data needs to be preprocessed by padding before using the network. The padding values are all 0, and the width of the dimension 39 is padded to 100, thus forming a square data shape of 100*100.
[0118] In the output layer, the output data dimension is defined to match the shape of the padded MFCC data in the input layer. Then, a slicing method (Split) is used to cut the original data, which was padded from 39 widths to 100 dimensions, starting from index 40, thus restoring the encoded data to 39*100. See the padded diagram for details. Figure 10 As shown.
[0119] This embodiment constructs a neural network model with consistent input and output, forming a neural network capable of encoding and decoding. The placeholder parameters stored in each block of the network serve as learnable parameters that can be automatically adjusted by the training script during subsequent training. These parameters are automatically updated using conventional training methods such as gradient descent and backpropagation. By feeding in a certain amount of data, the network tries to find the optimal parameters during training, ultimately achieving the goal of inputting audio, feature encoding and decoding, and outputting restored and denoised audio.
[0120] See Figure 11 The diagram shown is a schematic of the MFCC-AENet structure according to an embodiment of the present invention. In the Encode module, convolutional layers are used for feature extraction while pooling layers are used for downsampling. This enriches the feature channel information while compressing the length and width dimensions of the feature map, thereby preserving feature information while removing redundant information and reducing the number of parameters. The pooling layer used in this structure is local max pooling (Max-Pool), which takes the maximum value in the sliding window as the parameter of the downsampling result. Its advantage is that it can filter out a lot of useless information, making the processed features more distinct. The downsampling process is described in [reference needed]. Figure 12 As shown.
[0121] In the Decode module, similar to the Encode module, while using convolutional layers, pooling layers are replaced with upsampling layers. The purpose is to expand the length and width dimensions of the feature map using interpolation, gradually approximating the information dimensions of the original input, thereby reconstructing the padded MFCC feature matrix of the original input. See [link to upsampling process] for details. Figure 13 As shown.
[0122] (3.2) The output data y is obtained after the input data x is processed by the model. ^ The corresponding relationships are as follows:
[0123] y ^ -x≈0
[0124] In other words, the input data should closely approximate the output data; that is, the input data needs to be reconstructed from the model to produce data consistent with the input. The error function for measuring the error between the two sets of data is the equal-separated Euclidean distance, as follows:
[0125]
[0126] Where, x j This represents the raw data input to the model. This represents the prediction result output by the model.
[0127] (3.3) Using mostly noise-free snoring data with clean background sounds for training, a high accuracy rate can be achieved after 30 training iterations.
[0128] In this embodiment, after training the snoring binary classification model, background noise multi-classification model, and autoencoder model, a pipeline execution flow is further established to connect the three models. Besides the calls to the algorithm models themselves, some business logic functions are added to connect the calls to each model. Since this embodiment uses a recording device to continuously record audio of the sleep scene in real time, the audio stream requests real-time business processing from the algorithm via the device. Specifically, the business processing involves segmenting the audio stream into "frames" and defining the data stream specifications with the device. Each time, the algorithm receives a 1-second 16kHz PCM mono signal source and defines this signal format as "one frame." Therefore, one frame equals one second of audio signal, and similarly, five frames equal five seconds.
[0129] See Figure 14 As shown, the smallest aspect of the algorithm flow is the execution flow for processing one frame of data, and the specific flow is as follows:
[0130] (9.1) Capture a frame from the device to get the current audio data stream. The data format is consistent with the fixed format of the device.
[0131] (9.2) Add the data of the current frame to the buffer queue. The total length of the queue is the length of 5 seconds of data to be stored.
[0132] (9.3) Determine if the data length of the buffer queue is greater than or equal to 5 seconds. If it is, proceed to the next step. If it is not, prepare to end the task of the current frame and proceed to the processing of the next frame.
[0133] (9.4) If the data length meets the requirements, extract the MFCC characteristic coefficients of the data and form a characteristic matrix for the next step of data calculation;
[0134] (9.5) Call the background noise multi-classification model, input the MFCC features for detection, and obtain the background noise classification results;
[0135] (9.6) Determine whether the background noise of the current audio belongs to the quietness that is consistent with the sleep scene based on the classification results of the detection. If it is, proceed to the next step. If it is noisy, prepare to end the task of the current frame and proceed to the processing of the next frame.
[0136] (9.7) Use an autoencoder model to autoencode the snoring sounds of the currently selected MFCC features that match the scenario. The reconstructed features can highlight the characteristics of snoring sounds more.
[0137] (9.8) Call the snoring binary classification model, input the reconstructed MFCC features for calculation, and obtain the snoring binary classification result;
[0138] (9.9) Determine the classification of the current audio as snoring based on the snoring binary classification result. If it is snoring, proceed to the next step. If it is not snoring, prepare to end the task of the current frame and proceed to the processing of the next frame.
[0139] (9.10) Calculate the decibel value of the audio segment as a basis for judging whether it is a snoring sound;
[0140] (9.11) Calculate the time interval between the audio segment and the previous frame that was judged as snoring, and use it as a basis for determining whether it is snoring.
[0141] (9.12) Update the information of the current storage list, including timestamps and decibel values, for use as a basis for judgment in subsequent frames;
[0142] (9.13) Determine whether the snoring sound meets the criteria for snoring in terms of loudness and time interval. The criteria are to determine whether the loudness of the snoring sound and the time interval of the previous snoring sound are within the preset range. If they are, proceed to the next step. If not, prepare to end the task of the current frame and proceed to the processing of the next frame. The loudness range of the snoring sound and the time interval of the previous snoring sound can be manually adjusted by the user or a default value can be set according to the user's sleep habits. The default loudness range is 48 to 78 decibels; the time interval range is 2 to 6 seconds.
[0143] (9.14) If the audio of the current frame is determined to be snoring and meets the above judgment conditions, a message indicating that snoring has been detected is sent to the smart terminal device through the interface, and the node for judging snoring can be terminated.
[0144] (9.15) Dequeue the data at the head of the queue to make room for the data of the next frame to enter the queue.
[0145] See Figure 15 As shown, this embodiment also discloses a snoring detection device based on multi-model superposition, including:
[0146] Background audio data acquisition module 1501 is used to acquire audio data including the current frame and the previous preset frame of the current frame as background audio data;
[0147] MFCC feature extraction module 1502 is used to extract MFCC features from background noise audio data;
[0148] Background noise multi-classification model detection and acquisition module 1503 is used to input the MFCC features into the trained background noise multi-classification model for detection and to obtain the detection results;
[0149] The snoring detection module 1504 is used to input the MFCC features into the trained snoring binary classification model when the detection result is silent sound, and to detect whether the current frame audio data includes snoring.
[0150] Furthermore, the device also includes a snoring confirmation module, specifically used to perform the following processing after the snoring detection module 1504 detects that the current frame audio data includes snoring:
[0151] Calculate the decibel value of the current frame's audio data;
[0152] Calculate the time interval between the current frame and the previous frame that was identified as snoring;
[0153] Update the information in the current storage list; the information includes timestamps and decibel values;
[0154] Determine whether the decibel value is within a preset decibel range and whether the time interval is within a preset time range. If both conditions are met, indicate that snoring has been detected.
[0155] Furthermore, the device also includes an MFCC feature reconstruction module, specifically used to input the MFCC features into a trained autoencoder model to encode and decode the features to obtain the reconstructed MFCC features.
[0156] Correspondingly, the MFCC features are input into the trained snoring binary classification model to detect whether the current frame audio data includes snoring, specifically including:
[0157] The reconstructed MFCC features are input into a trained snoring binary classification model to detect whether the current frame of audio data includes snoring.
[0158] Other specific implementations of a snoring detection device based on multi-model superposition are described in this embodiment, and will not be repeated for the same snoring detection method based on multi-model superposition.
[0159] This embodiment also includes a smart terminal device, which is equipped with a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, it implements steps S101 to S104 in the above embodiment of a snoring detection method based on multi-model superposition.
[0160] The terminal device can be a smart terminal device, such as a mobile phone, tablet computer, wearable device, laptop computer, super mobile personal computer, netbook, personal digital assistant, etc. This smart terminal device may include, but is not limited to, a processor and memory.
[0161] The processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.
[0162] In some embodiments, the memory may be an internal storage unit of the terminal device, such as a hard drive or RAM. In other embodiments, the memory may be an external storage device of the terminal device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory may include both internal and external storage units of the terminal device. The memory is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory can also be used to temporarily store data that has been output or will be output.
[0163] The above description is merely a preferred embodiment of the present invention; however, the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and its improved concepts, should be covered within the scope of protection of the present invention.
Claims
1. A snoring detection method based on multi-model superposition, characterized in that, include: Acquire audio data of the current frame and the previous preset frame during human sleep as background audio data; Extract MFCC features from background noise audio data; The MFCC features are input into a trained background noise multi-classification model for detection, and the detection results are obtained. If the detection result is silence, the MFCC feature is input into the trained snoring binary classification model to detect whether the current frame audio data includes snoring.
2. The snoring detection method based on multi-model superposition according to claim 1, characterized in that, Obtain the audio data of the current frame and the preset frame before the current frame as background audio data, specifically including: Get the audio data of the current frame; The audio data of the current frame is concatenated with the audio data in the frame buffer queue and then inserted into the frame buffer queue; the frame buffer queue adopts the first-in-first-out principle and can store the audio data of the first preset frame; Determine whether the cache queue contains audio data of the first preset frame. If so, use the audio data in the cache queue as background audio data.
3. The snoring detection method based on multi-model superposition according to claim 1, characterized in that, If the current frame of audio data is detected to include snoring, the method further includes: Calculate the decibel value of the current frame's audio data; Calculate the time interval between the current frame and the previous frame that was identified as snoring; Update the information in the current storage list; the information includes timestamps and decibel values; Determine whether the decibel value is within a preset decibel range and whether the time interval is within a preset time range. If both conditions are met, indicate that snoring has been detected.
4. The snoring detection method based on multi-model superposition according to claim 1, characterized in that, Before inputting the MFCC features into the trained snoring binary classification model, the following steps are also included: The MFCC features are input into a trained autoencoder model to encode and decode the features to obtain the reconstructed MFCC features. The MFCC features are input into a trained snoring binary classification model to detect whether the current frame of audio data includes snoring, specifically including: The reconstructed MFCC features are input into a trained snoring binary classification model to detect whether the current frame of audio data includes snoring.
5. The snoring detection method based on multi-model superposition according to claim 4, characterized in that, The autoencoder model is a neural network model that includes an input layer, an encoder layer, a decoder layer, and an output layer. The autoencoder model uses the equal Euclidean distance as a loss metric, as follows: ； in, This represents the raw data input to the model. This represents the prediction result output by the model.
6. The snoring detection method based on multi-model superposition according to claim 5, characterized in that, The encoder layer includes several interconnected convolutional layers and pooling layers, and the decoder layer includes several interconnected convolutional layers and upsampling layers.
7. The snoring detection method based on multi-model superposition according to claim 1, characterized in that, The background sound multi-classification model adopts the AudioLiteNet neural network model, which includes several interconnected blocks. The blocks are connected directly or by skipping layers. The direct connection uses predefined operators in the block to perform data pipeline streaming calculations. The skipping layer connection adds the features together.
8. The snoring detection method based on multi-model superposition according to claim 1, characterized in that, The background noise multi-classification model uses five frames of audio data as background noise audio data. The input of the background noise multi-classification model is the extracted MFCC features, and the output of the background noise multi-classification model is the confidence score of music, human voice and silence. The class with the highest confidence score is taken as the detection result.
9. The snoring detection method based on multi-model superposition according to claim 1, characterized in that, The snoring binary classification model uses the AudioLiteNet neural network model, which includes several interconnected blocks. The blocks are connected directly or by skipping layers. The direct connection uses predefined operators in the block to perform data pipeline streaming calculations. The skipping layer connection adds the features together.
10. The snoring detection method based on multi-model superposition according to claim 1, characterized in that, The input to the snoring binary classification model is the extracted MFCC features, and the output of the snoring binary classification model is the confidence score of snoring and non-snoring. The class with the highest confidence score is taken as the detection result.
11. The snoring detection method based on multi-model superposition according to claim 1, characterized in that, Both the background noise multi-classification model and the snoring binary classification model use the binary cross-entropy loss function, as follows: ； in, This represents the total number of nodes that need to be measured, i.e., the total number of categories. Indicates the first The correct classification is determined by the actual labeled value of each node. The first number predicted by the neural network The classification value of the node.
12. A snoring detection device based on multi-model superposition, characterized in that, include: The background sound audio data acquisition module is used to acquire audio data including the current frame and the previous preset frame of the current frame as background sound audio data. The MFCC feature extraction module is used to extract MFCC features from background noise audio data. The background noise multi-classification model detection and acquisition module is used to input the MFCC features into the trained background noise multi-classification model for detection and to obtain the detection results. The snoring detection module is used to input the MFCC features into a trained snoring binary classification model when the detection result is silence, and to detect whether the current frame audio data includes snoring.