A snoring detection method, device, equipment and readable storage medium

Abstract

The present application provides a snoring detection method, apparatus, device and readable storage medium. The method includes: calculating the Mel-frequency cepstral coefficients and linear prediction coefficients of the microphone sound signal respectively; performing feature splicing and dimensionality reduction processing on the Mel-frequency cepstral coefficients and linear prediction coefficients to obtain a target feature sequence; calculating a target feature vector corresponding to the micro-Doppler feature of the target body part of the user based on the radar echo signal; wherein, the microphone sound signal and the radar echo signal are signals synchronously collected in the user's sleep environment; the micro-Doppler feature is used to reflect the vital activity state of the target part; inputting the target feature sequence and the target feature vector into a preset fusion processing model respectively to output a snoring detection result. Through the implementation of the solution of the present application, by fusing the feature data collected by two sensors, the snoring recognition ability in the home scenario can be effectively improved, and the complexity of data processing and the dependence of data processing on the network can be reduced.

Description

Technical Field

[0001] This application relates to the technical field of sleep monitoring, and in particular, to a snoring detection method, device, equipment and readable storage medium. Background Art

[0002] Sleep is a biological activity initiated and controlled by the human brain. High-quality sleep is crucial for the health of the human body. The human body repairs and rebuilds itself during sleep, clearing metabolic waste accumulated during wakefulness. Sleep can also reorganize memories and support the formation of long-term memories. With the accelerating pace of life and increasing social pressure, the number of people suffering from insomnia and poor sleep is increasing, the incidence of sleep disorders is also rising day by day, and the group of sleep disorders is gradually getting younger; problems such as heart diseases and sudden deaths caused by sleep problems are also becoming increasingly serious. Therefore, human sleep monitoring has gradually become an important research topic. With the in-depth research, the technology of extracting physical sign signals has been continuously improved, and researchers have begun to expand the research focus to aspects such as multi-person respiratory rate estimation, sleep staging, sleeping posture recognition, and snoring detection.

[0003] Snoring is a physiological phenomenon related to breathing. Long-term snoring is related to various health problems. Snoring behavior is a symptom of diseases such as Obstructive Sleep Apnea Hypopnea Syndrome (OSAHS), systemic arterial hypertension, and coronary artery disease; among them, snoring is the earliest and most common symptom of OSAHS patients, with an incidence rate as high as 70%-95%, and as the severity of OSAHS increases, the intensity of snoring usually also increases. Therefore, snoring detection plays a very important role in the prevention and research of related diseases.

[0004] In related technologies, the methods of snoring detection usually include contact detection and non-contact detection. For contact detection, since the user needs to wear the detection device for a long time, it may cause discomfort to the user during sleep, resulting in a further decline in the user's sleep quality. Especially for people with sensitive skin, skin allergies, friction, and indentations are likely to occur. Non-contact detection can analyze the sleep state by collecting the user's voice, without the need for the device to be in direct contact with the user, which can improve the user's experience to a certain extent. However, the signal acquisition method of non-contact detection in related technologies is relatively single, and it is difficult to conduct in-depth and targeted analysis on the acquired data, with a high computational complexity and limited snoring recognition and detection efficiency. Summary of the Invention

[0005] Embodiments of this application provide a snoring detection method, device, equipment and readable storage medium, which can at least solve the problem that the accuracy of snoring detection in related technologies needs to be improved.

[0006] In a first aspect of the embodiments of the present application, a snoring detection method is provided, including:

[0007] Calculating the Mel-frequency cepstral coefficients and linear prediction coefficients of the microphone sound signal respectively;

[0008] Performing feature splicing and dimensionality reduction processing on the Mel-frequency cepstral coefficients and linear prediction coefficients to obtain a target feature sequence;

[0009] Based on the radar echo signal, calculating a target feature vector corresponding to the micro-Doppler feature of the target part of the user's body; wherein, the microphone sound signal and the radar echo signal are signals synchronously collected in the user's sleep environment; the micro-Doppler feature is used to reflect the vital activity state of the target part;

[0010] Inputting the target feature sequence and the target feature vector into a preset fusion processing model respectively to output a snoring detection result.

[0011] In a second aspect of the embodiments of the present application, a snoring detection device is provided, including:

[0012] A first calculation module, configured to calculate the Mel-frequency cepstral coefficients and linear prediction coefficients of the microphone sound signal respectively;

[0013] A splicing and dimensionality reduction module, configured to perform feature splicing and dimensionality reduction processing on the Mel-frequency cepstral coefficients and linear prediction coefficients to obtain a target feature sequence;

[0014] A second calculation module, configured to calculate a target feature vector corresponding to the micro-Doppler feature of the target part of the user's body based on the radar echo signal; wherein, the microphone sound signal and the radar echo signal are signals synchronously collected in the user's sleep environment; the micro-Doppler feature is used to reflect the vital activity state of the target part;

[0015] A fusion processing module, configured to input the target feature sequence and the target feature vector into a preset fusion processing model respectively to output a snoring detection result.

[0016] In a third aspect of the embodiments of the present application, an electronic device is provided, including: a memory and a processor, wherein the processor is configured to execute a computer program stored on the memory, and when the processor executes the computer program, each step in the snoring detection method provided in the first aspect of the embodiments of the present application is implemented.

[0017] In a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, each step in the snoring detection method provided in the first aspect of the embodiments of the present application is implemented.

[0018] As can be seen from the above, according to the snoring detection method, device, equipment and readable storage medium provided by the solution of the present application, the Mel frequency cepstral coefficients and linear prediction coefficients of the microphone sound signal are calculated respectively; feature splicing and dimensionality reduction processing are performed on the Mel frequency cepstral coefficients and linear prediction coefficients to obtain a target feature sequence; based on the radar echo signal, a target feature vector corresponding to the micro-Doppler feature of the user's body target part is calculated; wherein, the microphone sound signal and the radar echo signal are signals synchronously collected in the user's sleep environment; the micro-Doppler feature is used to reflect the vital activity state of the target part; the target feature sequence and the target feature vector are respectively input into a preset fusion processing model to output a snoring detection result. Through the implementation of the solution of the present application, feature splicing and dimensionality reduction processing are performed on the Mel frequency cepstral coefficients and linear prediction coefficients of the microphone sound signal, which can improve the effectiveness of data analysis while effectively reducing the calculation amount; by combining the data collected by the radar sensor and the microphone for processing, the fusion of multi-source features is realized, enabling the network to comprehensively utilize multi-modal information from different sensors, reducing the influence of environmental noise on snoring detection, improving the accuracy of snoring detection, and being able to non-contact, all-weather and real-time monitor the human sleep condition, avoiding the discomfort caused by contact devices, and being not affected by factors such as environmental light and clothing occlusion, which is suitable for long-term and continuous monitoring at home. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic flowchart of the basic process of a snoring detection method provided by the first embodiment of the present application;

[0020] Figure 2 It is a schematic structural diagram of the framework of a fusion processing model provided by the first embodiment of the present application;

[0021] Figure 3 It is a schematic detailed flowchart of a snoring detection method provided by the second embodiment of the present application;

[0022] Figure 4 It is a schematic diagram of the program modules of a snoring detection device provided by the third embodiment of the present application;

[0023] Figure 5 It is a schematic structural diagram of an electronic device provided by the fourth embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] To make the objects, features, and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0025] In the description of the embodiments of the present application, it should be understood that the orientation or positional relationship indicated by the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the embodiments of the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0026] In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, the meaning of "a plurality" is two or more, unless otherwise specifically defined.

[0027] In the embodiments of the present application, unless otherwise clearly defined and limited, the terms "mounted", "connected", "connected", "fixed", etc. should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or integrated; it may be a mechanical connection or an electrical connection; it may be directly connected, or indirectly connected through an intermediate medium, and it may be the internal communication of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to specific situations.

[0028] To solve the problem that the accuracy of snoring detection in related technologies needs to be improved, the first embodiment of the present application provides a snoring detection method, as Figure 1 is the basic process schematic diagram of the snoring detection method provided in this embodiment. The snoring detection method includes the following steps:

[0029] Step 101: Calculate the Mel-frequency cepstral coefficients and linear prediction coefficients of the microphone sound signal respectively.

[0030] Specifically, the sound information generated by the user during sleep is collected through a microphone, and the auditory perception features in the sound information are extracted by calculating the Mel Frequency Cepstral Coefficients (MFCCs) and Linear Prediction Coefficients (LPCs), which can lay a foundation for the real-time detection of snoring.

[0031] Step 102: Perform feature concatenation and dimensionality reduction on the Mel Frequency Cepstral Coefficients and Linear Prediction Coefficients to obtain a target feature sequence.

[0032] Specifically, performing feature concatenation on the Mel Frequency Cepstral Coefficients and Linear Prediction Coefficients can retain complete acoustic information, while dimensionality reduction can effectively reduce the load of subsequent calculations.

[0033] Step 103: Based on the radar echo signal, calculate the target feature vector corresponding to the micro-Doppler feature of the target body part of the user.

[0034] Specifically, the micro-Doppler feature is used to reflect the vital activity state of the target part. In this embodiment, the radar used to collect the echo signal can be a millimeter-wave radar. By integrating the fine motion perception ability of the millimeter-wave radar, the recognition ability of snoring and non-snoring (for example, other sounds generated by the user during sleep) under actual home conditions can be significantly improved. In this embodiment, the microphone sound signal and the radar echo signal are signals synchronously collected in the user's sleep environment. Specifically, for the acquisition of microphone and radar dual-modal synchronous data, software synchronization can be achieved through two steps: triggering by the power-on synchronization signal and timestamp marking. Since the initialization time of the radar RF module is relatively long, while the initialization time of the microphone module is relatively short, after the system powers on and completes initialization, it enters the "power-on synchronization signal triggering" stage. The radar module can send a synchronization signal to the microphone module through the GPIO port and immediately start data acquisition; after receiving the synchronization signal, the microphone module will first perform level jitter elimination processing to ensure signal stability, and then immediately start the data acquisition process.

[0035] Step 104: Input the target feature sequence and the target feature vector into a preset fusion processing model respectively to output the snoring detection result.

[0036] Specifically, in this embodiment, a fusion processing model with low complexity is designed to fuse the target feature sequence of the microphone and the target feature vector of the millimeter-wave radar. The network calculation burden is small, edge-side computing can be achieved, there is no need to rely on network upload, the impact of network fluctuations is small, and it can work without a network.

[0037] In an implementation manner of this embodiment, the step of calculating the Mel-frequency cepstral coefficients of the microphone sound signal includes: performing an FFT transformation on the microphone sound signal, and calculating the spectral line energy of each frame of spectral data obtained after the FFT transformation; calculating the Mel-frequency energy based on each spectral line energy; performing a discrete cosine transformation on the Mel-frequency energy to obtain the Mel-frequency cepstral coefficients.

[0038] Specifically, for the data collected by the microphone, the speech signal of the th frame , where [] represents the sampling point index, the steps for calculating the Mel-frequency cepstral coefficients are as follows:

[0039] (1) Perform an FFT transformation on each frame of the sound signal, expressed as:

[0040]

[0041] where, represents the result of the FFT transformation, represents the frequency index; and both represent the imaginary unit.

[0042] (2) Calculate the spectral line energy of the spectral data after the FFT for each frame:

[0043]

[0044] (3) Calculate the energy passing through the Mel filter:

[0045]

[0046]

[0047]

[0048]

[0049] where, represents the transfer function of the Mel filter bank, represents the filter index, represents the number of filters, represents the th filter's energy, represents the center frequency of each filter, represents converting the actual frequency to the perceived frequency in Mel units, represents the inverse conversion, represents the sampling frequency, represents the lowest frequency of the filter frequency range, Represents the highest frequency of the filter frequency range.

[0050] (4) Perform a discrete cosine transform on the Mel frequency energy to obtain Mel frequency cepstral coefficients:

[0051]

[0052] Wherein, Represents the index of the cepstral coefficient, Represents the order of the Mel frequency cepstral coefficient. Exemplarily, a 13-order Mel frequency cepstral coefficient can be adopted.

[0053] In an implementation manner of this embodiment, the above steps of calculating the linear prediction coefficients of the microphone sound signal include: constructing a linear prediction model of the microphone sound signal; calculating the autocorrelation function of the microphone sound signal based on the linear prediction model; constructing and solving a system of linear equations corresponding to the autocorrelation function to obtain the linear prediction coefficients of the microphone sound signal.

[0054] Specifically, in this embodiment, the extraction of the linear prediction coefficients relies on the linear prediction analysis method, which can simplify the sound signal into the output result of a linear prediction model and use the relevant parameters of the model to characterize the characteristics of the signal. Specifically, by establishing a linear prediction equation and solving this equation to obtain the linear prediction coefficients. In some specific implementation manners, taking the speech signal as an example, for the speech sampling value at a specific moment, it is possible to obtain a unique set of prediction coefficients in the way of linearly combining the speech sampling values at some past moments with the minimum mean square error to approximate the actual value as much as possible. This set of prediction coefficients is the linear prediction coefficients of the speech signal. The linear prediction coefficients have significant advantages in speech signal processing, can quickly calculate or estimate the linear feature rules, and can accurately characterize the time-domain and frequency-domain characteristics of the speech signal. In this embodiment, the transfer function of the linear prediction model can be expressed as:

[0055]

[0056] Wherein, Represents the order of the linear prediction model. Exemplarily, 13-order linear prediction coefficients can be selected, Represents the gain coefficient, Represents the linear prediction coefficients of the speech signal. In some specific implementation manners, the Levinson-Durbin algorithm can be used to solve the linear prediction coefficients. This algorithm has characteristics such as high computational efficiency and good numerical stability.

[0057] Further, in an implementation manner of this embodiment, the step of performing feature splicing and dimensionality reduction processing on the mel-frequency cepstral coefficients and linear prediction coefficients to obtain a target feature sequence includes: calculating a composite feature vector of the mel-frequency cepstral coefficients and linear prediction coefficients; performing dimensionality reduction processing on the composite feature vector to determine a feature set corresponding to all principal component features of the composite feature vector, where the cumulative contribution degree of the principal component features included in the feature set is greater than a preset threshold; performing feature serialization processing on the feature set in the time dimension to obtain a target feature sequence.

[0058] Specifically, first, two types of time-frequency spectrum features of the sound signal can be calculated in parallel: 13-dimensional mel-frequency cepstral coefficients (representing auditory perception characteristics) and 13-dimensional linear prediction coefficients (reflecting vocal tract resonance characteristics). Cross-domain information fusion is achieved through feature concatenation to form a 26-dimensional composite feature vector. Although this original splicing can retain complete acoustic information, the expansion of the feature space dimension will lead to a significant increase in the computational complexity of the subsequent classifier. Therefore, in this embodiment, dimensionality reduction processing is used to reconstruct the feature space. Specifically, the 26-dimensional composite feature can be projected onto the direction of the maximum variance through orthogonal transformation, and only the principal component features with a cumulative contribution degree greater than the preset threshold are retained. Exemplarily, the first 13 principal components with a cumulative contribution degree > 95% can be retained; at the same time, redundant feature components are removed, so that the 26-dimensional features are reduced to 13-dimensional main features, achieving the effect of significantly reducing the subsequent computational load. In some specific implementation manners, the 13-dimensional principal component features of consecutive time windows within each second can be arranged in chronological order to construct a 13-dimensional target feature sequence, which can be denoted as , in this way, both the dynamic change characteristics of the snoring sound are retained, and the ability of the classifier to capture the snoring rhythm pattern is enhanced through structured data expression.

[0059] In an implementation manner of this embodiment, the step of calculating a target feature vector corresponding to the micro-Doppler feature of the user's body target part based on the radar echo signal includes: determining the average range-Doppler spectrum of the radar echo signal, where the average range-Doppler spectrum is used to represent the difference in the movement amplitude exhibited by the user's body target part; extracting the micro-Doppler features of multiple target parts of the user's body based on the average range-Doppler spectrum; constructing a target feature vector corresponding to the multiple micro-Doppler features, where the target feature vector is expressed as: , represents the target feature vector, represents the micro-Doppler feature of the nth target part.

[0060] Specifically, during sleep, the body's vital activities show differences in the movement amplitude in local parts of the body, and these differences can be reflected by the average distance-Doppler spectrum. In this embodiment, by extracting effective and reliable local micro-Doppler features in the average distance-Doppler spectrum and combining and applying these extracted micro-Doppler features, the accuracy of snoring detection can be effectively improved.

[0061] Further, in an implementation manner of this embodiment, the radar echo signal is a discrete echo signal associated with a slow-time stack, a fast-time stack, and an antenna stack, and the number of antenna stacks is multiple; correspondingly, the steps of determining the average distance-Doppler spectrum of the radar echo signal include: performing an FFT transform on the radar echo signal in the fast-time dimension to obtain a distance-slow-time dimension-antenna dimension signal; performing distance dimension clutter suppression on the distance-slow-time dimension-antenna dimension signal to obtain a target multi-dimensional signal; performing a Doppler-FFT conversion on the target multi-dimensional signal in the slow-time dimension to obtain the distance-Doppler spectra corresponding to multiple antenna stacks; and performing an accumulation calculation on the distance-Doppler spectra corresponding to multiple antenna stacks to obtain the average distance-Doppler spectrum.

[0062] Specifically, the radar can continuously transmit electromagnetic wave signals into the sleep space, which are reflected by objects and then received by the radar receiver. After passing through a signal amplifier, a mixer, and ADC sampling, a discrete echo signal including a distance dimension, a velocity dimension, and an angle dimension is formed. In this embodiment, the echo signal received by the radar module can be expressed as , where represents the slow-time dimension, that is, the th linear frequency modulation continuous wave signal; represents the fast-time dimension, that is, the th sampling point; represents the antenna dimension, that is, the received signal of the th channel. Performing a fast FFT transform on the received radar echo signal in the fast-time dimension to obtain a distance-slow-time dimension-antenna dimension signal, denoted as , where, , represents distance unit sampling, is the number of FFT points; then performing distance dimension clutter suppression, and the clutter suppression can be calculated and processed using algorithms such as a sliding average algorithm and a subtraction average algorithm, which are not limited here. The multi-dimensional signal obtained after clutter suppression can be denoted as ; performing a Doppler-FFT on in the slow-time dimension to obtain the distance-Doppler spectrum, and accumulating the distance-Doppler spectra of multiple antenna stacks to obtain the average distance-Doppler spectrum , where, is the Doppler transform frequency.

[0063] Furthermore, in one implementation manner of this embodiment, the step of extracting the micro-Doppler features of multiple target parts of the user's body based on the average range-Doppler spectrum includes:

[0064] Based on the average range-Doppler spectrum, extract the micro-Doppler features of the user's neck. The calculation formula is as follows:

[0065]

[0066]

[0067]

[0068] Wherein, represents the first micro-Doppler feature of the neck, represents the second micro-Doppler feature of the neck, represents the first non-zero range cell, represents the second non-zero range cell, represents the range cell variable, and , represents the Doppler cumulative feature of the human body along the range change direction (i.e., the direction of radar beam propagation), represents the Doppler transform frequency, represents the average range-Doppler spectrum

[0069] Based on the average range-Doppler spectrum, extract the micro-Doppler features of the user's chest and abdomen. The calculation formula is as follows:

[0070]

[0071]

[0072] Wherein, represents the third micro-Doppler feature of the chest and abdomen, represents the fourth micro-Doppler feature of the chest and abdomen, represents the sequence the first peak of, represents the sequence the second peak of;

[0073] Based on the average range-Doppler spectrum, extract the micro-Doppler features of the user's head. The calculation formula is as follows:

[0074]

[0075]

[0076]

[0077]

[0078]

[0079]

[0080] Among them, represents the fifth micro-Doppler feature of the head, represents the sixth micro-Doppler feature of the head, represents the non-zero logical matrix of represents the size of the number of non-zeros of represents the decision threshold, represents the 0-norm calculation, represents the difference between the maximum and minimum values of the non-zero logic along the range image. The range image is a one-dimensional image generated by processing the fast-time stack data of the radar echo signal and reflecting the reflection intensity distribution of the target in the range direction;

[0081] It also includes: extracting the micro-Doppler features corresponding to the user's body movement index, and the calculation formula is as follows:

[0082]

[0083] Among them, represents the seventh micro-Doppler feature corresponding to the body movement index, represents the eighth micro-Doppler feature corresponding to the body movement index, represents the sampling time point corresponding to the slow-time stack, represents the body movement size at the moment of represents the upper limit value of the body movement index, represents the energy value corresponding to the highest energy point in the current range-Doppler spectrum, represents the minimum value among the highest energy points of all previous range-Doppler spectra.

[0084] Correspondingly, the target feature vector corresponding to the above multiple micro-Doppler features can be expressed as: .

[0085] It should be noted that during sleep, the movement amplitude differences between the snoring state and the non-snoring state are different. For example, when the snoring intensifies, the airway resistance increases, which will cause changes in the micro-Doppler feature parameters (that is, the amplitude of the micro-Doppler signal increases). At the same time, the throat tissue vibrates violently, causing the amplitude of the micro-Doppler signal corresponding to the neck to also increase accordingly; the head is connected to the cervical vertebra and thoracic vertebra. When the thoracic vertebra moves, it will also drive the head to move, thereby causing changes in the amplitude of the micro-Doppler signal.

[0086] For the same human body target, the periods of the subtle movements of various body parts caused by breathing are consistent. There will be differences in the echo energy and Doppler frequency of various body parts, that is, the micro-Doppler signal amplitudes of different body parts are different. Therefore, in this embodiment, the amplitude changes of different body parts are characterized by micro-Doppler features, and then the sleep state of the user can be evaluated by comprehensively considering the micro-Doppler features corresponding to different body parts of the user, which can effectively improve the accuracy and reliability of snore detection.

[0087] In an implementation manner of this embodiment, the fusion processing model includes an LSTM network layer, a Dropout layer, and a fully connected layer; correspondingly, the step of respectively inputting the target feature sequence and the target feature vector into a preset fusion processing model to output a snore detection result includes: inputting the target feature sequence into the LSTM network layer to extract the time dependence relationship of the target feature sequence and output the encoded high-order temporal features; concatenating the target output feature of the last time step corresponding to the high-order temporal features with the target feature vector to obtain a fusion feature; sequentially inputting the fusion feature into the Dropout layer and the fully connected layer to output a snore detection result.

[0088] Specifically, in this embodiment, the target feature sequence of the microphone sound signal has typical time series characteristics. This embodiment designs a low-complexity LSTM fusion detection network model, that is, the fusion processing model, and its framework structure is as Figure 2 shown. In this network architecture, first, the target feature sequence is input into a single-layer LSTM network layer with 16 hidden neurons. The LSTM network layer, relying on its unique gating mechanism, can effectively capture the time dependence relationship in the target feature sequence, deeply mine and extract the time features therein, and efficiently encode the time series information; subsequently, the feature output of the last time step of the LSTM network layer is concatenated with the target feature vector of the radar to achieve the fusion of multi-source features, enabling the network to comprehensively utilize multi-modal information from different sensors; the fused feature data obtained after concatenation is sequentially passed through the Dropout layer (that is, the random inactivation layer) and the fully connected layer; the Dropout layer can effectively avoid the overfitting of the model by randomly inactivating neurons, further improving the generalization ability of the model; the fully connected layer then performs weighted summation on the fused features and performs a non-linear transformation to finally output a snore detection result. By constructing such a lightweight classification network in this embodiment, compared with the traditional complex deep learning network model, the network calculation burden is small, edge-side computing can be achieved, there is no need to rely on network upload, the impact of network fluctuations is small, and it can work without a network.

[0089] During the verification test, in the actual home sleep scenario, sleep data (microphone sound signals and radar echo signals) of 31 nights were collected for different users, and the snoring detection method of this embodiment was used to obtain the snoring detection results. The statistical results show that the average accuracy rate of the snoring detection results can reach 94.7%. In addition, this detection method of this embodiment also has remarkable effects in non-snoring discrimination.

[0090] Based on the technical solutions of the embodiments of the present application above, the Mel frequency cepstral coefficients and linear prediction coefficients of the microphone sound signals are calculated respectively; feature splicing and dimensionality reduction processing are performed on the Mel frequency cepstral coefficients and linear prediction coefficients to obtain a target feature sequence; based on the radar echo signal, a target feature vector corresponding to the micro-Doppler feature of the target body part of the user is calculated; wherein, the microphone sound signal and the radar echo signal are signals synchronously collected in the user's sleep environment; the micro-Doppler feature is used to reflect the vital activity state of the target part; the target feature sequence and the target feature vector are respectively input into a preset fusion processing model to output the snoring detection result. Through the implementation of this embodiment, feature splicing and dimensionality reduction processing are performed on the Mel frequency cepstral coefficients and linear prediction coefficients of the microphone sound signals, which can improve the effectiveness of data analysis while effectively reducing the calculation amount; the data collected by the radar sensor and the microphone are combined for processing to realize the fusion of multi-source features, enabling the network to comprehensively utilize multi-modal information from different sensors, improving the accuracy of snoring detection, and capable of non-contact, all-weather, and real-time monitoring of the human sleep condition, avoiding the discomfort brought by contact devices, and being not affected by factors such as environmental light and clothing occlusion, and is suitable for long-term and continuous monitoring at home.

[0091] Figure 3 The method in [reference] provides a refined snoring detection method for the second embodiment of the present application. This snoring detection method includes:

[0092] Step 301: Calculate the Mel frequency cepstral coefficients and linear prediction coefficients of the microphone sound signals respectively.

[0093] Step 302: Calculate the composite feature vector of the Mel frequency cepstral coefficients and linear prediction coefficients.

[0094] Specifically, horizontal splicing of the Mel frequency cepstral coefficients and linear prediction coefficients can retain complete acoustic information.

[0095] Step 303: Perform dimensionality reduction processing on the composite feature vector to determine the feature set corresponding to all the principal component features of the composite feature vector.

[0096] Specifically, the cumulative contribution degree of the principal component features included in the feature set is greater than a preset threshold. In some specific embodiments, redundant feature components between acoustic features can be removed through covariance matrix decomposition, and the feature dimension can be compressed on the premise of key original information volume, which can significantly reduce the subsequent calculation load.

[0097] Step 304: Perform feature serialization processing on the feature set in the time dimension to obtain a target feature sequence.

[0098] Step 305: Determine the average range-Doppler spectrum of the radar echo signal.

[0099] Specifically, the average range-Doppler spectrum is used to characterize the differences in the movement amplitudes exhibited by the target parts of the user's body.

[0100] Step 306: Based on the average range-Doppler spectrum, extract the micro-Doppler features of multiple target parts of the user's body respectively.

[0101] Step 307: Construct target feature vectors corresponding to multiple micro-Doppler features.

[0102] Step 308: Input the target feature sequence and the target feature vector into a preset fusion processing model respectively to output a snoring detection result.

[0103] Specifically, the fusion processing model includes an LSTM network layer, a Dropout layer, and a fully connected layer; correspondingly, the above fusion processing process includes: inputting the target feature sequence into the LSTM network layer to extract the time-dependent relationship of the target feature sequence and output the encoded high-order temporal features; concatenating the target output feature of the last time step corresponding to the high-order temporal features with the target feature vector to obtain a fusion feature; inputting the fusion feature into the Dropout layer and the fully connected layer in sequence to output a snoring detection result.

[0104] In this embodiment, the effective information in the microphone sound signal is accurately extracted through the calculation and application of Mel-frequency cepstral coefficients and linear prediction coefficients, the millimeter-wave radar is used to obtain synchronous sleep information by virtue of its ability to perceive subtle movements, and the two parts of information are fused and processed, which can significantly improve the ability to distinguish between snoring and non-snoring under actual home conditions.

[0105] It should be understood that the magnitudes of the sequence numbers of the steps in this embodiment do not mean the sequence of step execution. The execution sequence of each step should be determined according to its function and internal logic, and should not constitute a unique limitation on the implementation process of the embodiments of this application.

[0106] Figure 4A snoring detection device provided for the third embodiment of this application. This snoring detection device can be applied to the aforementioned snoring detection method. As Figure 4 shown, this snoring detection device mainly includes:

[0107] A first calculation module 401, configured to calculate the Mel frequency cepstral coefficients and linear prediction coefficients of the microphone sound signal respectively;

[0108] A splicing and dimensionality reduction module 402, configured to perform feature splicing and dimensionality reduction processing on the Mel frequency cepstral coefficients and linear prediction coefficients to obtain a target feature sequence;

[0109] A second calculation module 403, configured to calculate a target feature vector corresponding to the micro-Doppler feature of the target part of the user's body based on the radar echo signal; wherein, the microphone sound signal and the radar echo signal are signals synchronously collected in the user's sleep environment; the micro-Doppler feature is used to reflect the vital activity state of the target part;

[0110] A fusion processing module 404, configured to input the target feature sequence and the target feature vector into a preset fusion processing model respectively to output a snoring detection result.

[0111] In some implementation manners of this embodiment, the first calculation module is specifically configured to: perform FFT transformation on the microphone sound signal, and calculate the spectral line energy of each frame of spectral data obtained after the FFT transformation; calculate the Mel frequency energy based on each spectral line energy; perform discrete cosine transformation on the Mel frequency energy to obtain the Mel frequency cepstral coefficients.

[0112] In some implementation manners of this embodiment, the splicing and dimensionality reduction is specifically configured to: calculate a composite feature vector of the Mel frequency cepstral coefficients and the linear prediction coefficients; perform dimensionality reduction processing on the composite feature vector to determine a feature set corresponding to all the principal component features of the composite feature vector; wherein, the cumulative contribution degree of the principal component features included in the feature set is greater than a preset threshold; perform feature serialization processing on the feature set in the time dimension to obtain a target feature sequence.

[0113] In some implementation manners of this embodiment, the second calculation module is specifically configured to: determine the average range-Doppler spectrum of the radar echo signal; wherein, the average range-Doppler spectrum is used to characterize the difference in the movement amplitude shown by the target part of the user's body; based on the average range-Doppler spectrum, extract the micro-Doppler features of multiple target parts of the user's body respectively; construct a target feature vector corresponding to the multiple micro-Doppler features.

[0114] Further, in some embodiments of the present embodiment, the radar echo signal is a discrete echo signal associated with a slow time stack, a fast time stack, and an antenna stack, and the number of antenna stacks is multiple; correspondingly, when the second calculation module executes the above function of determining the average range-Doppler spectrum of the radar echo signal, it specifically is used for: performing an FFT transform on the radar echo signal in the fast time dimension to obtain a range-slow time dimension-antenna dimension signal; performing range dimension clutter suppression on the range-slow time dimension-antenna dimension signal to obtain a target multi-dimensional signal; performing a Doppler-FFT conversion on the target multi-dimensional signal in the slow time dimension to obtain range-Doppler spectra corresponding to multiple antenna stacks; performing an accumulation calculation on the range-Doppler spectra corresponding to multiple antenna stacks to obtain an average range-Doppler spectrum.

[0115] Further, in some embodiments of the present embodiment, when the second calculation module executes the above function of respectively extracting the micro-Doppler features of multiple target parts of the user's body based on the average range-Doppler spectrum, it specifically is used for: extracting the micro-Doppler features of the user's neck based on the average range-Doppler spectrum, and the calculation formula is as follows:

[0116]

[0117]

[0118]

[0119] Wherein, represents the first micro-Doppler feature of the neck, represents the second micro-Doppler feature of the neck, represents the first non-zero range cell, represents the second non-zero range cell, represents the range cell variable, and , represents the Doppler accumulation feature of the human body along the range change direction, represents the Doppler transform frequency, represents the average range-Doppler spectrum;

[0120] Based on the average range-Doppler spectrum, extract the micro-Doppler features of the user's chest and abdomen, and the calculation formula is as follows:

[0121]

[0122]

[0123] Wherein, represents the third micro-Doppler feature of the chest and abdomen, represents the fourth micro-Doppler feature of the chest and abdomen, represents the sequence The first peak value represents the sequence The second peak value;

[0124] Based on the average range-Doppler spectrum, the micro-Doppler features of the user's head are extracted, and the calculation formula is as follows:

[0125]

[0126]

[0127]

[0128]

[0129]

[0130]

[0131] where represents the fifth micro-Doppler feature of the head, represents the sixth micro-Doppler feature of the head, represents The non-zero logical matrix of represents The number of non-zero elements of represents the decision threshold, represents the 0-norm calculation, represents the difference between the maximum and minimum values of the non-zero logic along the range image;

[0132] It also includes: extracting the micro-Doppler features corresponding to the user's body movement index, and the calculation formula is as follows:

[0133]

[0134] where represents the seventh micro-Doppler feature corresponding to the body movement index, represents the eighth micro-Doppler feature corresponding to the body movement index, represents the sampling time point corresponding to the slow-time stack, represents The body movement magnitude at the moment of represents the upper limit value of the body movement index, represents the energy value corresponding to the highest energy point in the current range-Doppler spectrum, represents the minimum value among the highest energy points of all previous range-Doppler spectra.

[0135] In some embodiments of this embodiment, the fusion processing model includes an LSTM network layer, a Dropout layer, and a fully connected layer; correspondingly, the fusion processing module is specifically configured to: input the target feature sequence into the LSTM network layer to extract the temporal dependence relationship of the target feature sequence and output the encoded high-order temporal features; splice the target output feature of the last time step corresponding to the high-order temporal features with the target feature vector to obtain the fusion features; input the fusion features into the Dropout layer and the fully connected layer in sequence to output the snoring detection result.

[0136] According to the snoring detection device provided in this embodiment, the Mel frequency cepstral coefficients and linear prediction coefficients of the microphone sound signal are calculated respectively; feature splicing and dimensionality reduction processing are performed on the Mel frequency cepstral coefficients and linear prediction coefficients to obtain the target feature sequence; based on the radar echo signal, the target feature vector corresponding to the micro-Doppler feature of the target body part of the user is calculated; wherein, the microphone sound signal and the radar echo signal are signals synchronously collected in the user's sleep environment; the micro-Doppler feature is used to reflect the vital activity state of the target part; the target feature sequence and the target feature vector are respectively input into a preset fusion processing model to output the snoring detection result. Through the implementation of this embodiment, feature splicing and dimensionality reduction processing are performed on the Mel frequency cepstral coefficients and linear prediction coefficients of the microphone sound signal, which can improve the effectiveness of data analysis while effectively reducing the calculation amount; the data collected by the radar sensor and the microphone are combined for processing to realize the fusion of multi-source features, enabling the network to comprehensively utilize multi-modal information from different sensors, improving the accuracy of snoring detection, and being able to non-contact, all-weather, and real-time monitor the human sleep condition, avoiding the discomfort brought by contact devices, and being not affected by factors such as environmental light and clothing occlusion, which is suitable for long-term and continuous monitoring at home.

[0137] Figure 5 An electronic device provided in the fourth embodiment of this application. This electronic device can be used to implement the snoring detection method in the foregoing embodiments, and mainly includes:

[0138] A memory 501, a processor 502, and a computer program 503 stored on the memory 501 and executable on the processor 502. The memory 501 and the processor 502 are communicatively connected. When the processor 502 executes the computer program 503, the methods in the foregoing Embodiment 1 or 2 are implemented. Wherein, the number of processors can be one or more.

[0139] The memory 501 may be a high-speed random access memory (RAM), or a non-volatile memory, such as a disk memory. The memory 501 is used to store executable program codes, and the processor 502 is coupled to the memory 501.

[0140] Furthermore, an embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium may be disposed in the above-mentioned electronic device, and the computer-readable storage medium may be the memory in the foregoing Figure 5 illustrated embodiment.

[0141] A computer program is stored on the computer-readable storage medium. When the program is executed by a processor, it implements the snoring detection method in the foregoing embodiment. Further, the computer-readable storage medium may also be various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk, or an optical disc that can store program codes.

[0142] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other may be through some interfaces. The indirect couplings or communication connections of devices or modules may be in electrical, mechanical or other forms.

[0143] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical modules, that is, they may be located in one place, or may be distributed to multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, in each embodiment of the present application, the functional modules may be integrated in a processing module, or each module may exist physically alone, or two or more modules may be integrated in one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0145] When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned readable storage medium includes: various media such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0146] It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0147] In the above embodiments, the descriptions of the various embodiments have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0148] The above is the description of the snoring detection method, device, equipment, and readable storage medium provided by this application. For those skilled in the art, according to the idea of the embodiments of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A snoring detection method, characterized in that, Including: Calculating the Mel-frequency cepstral coefficients and linear prediction coefficients of the microphone sound signal respectively; Performing feature splicing and dimensionality reduction processing on the Mel-frequency cepstral coefficients and the linear prediction coefficients to obtain a target feature sequence; Performing FFT transformation on the radar echo signal in the fast time dimension to obtain a range-slow time dimension-antenna dimension signal; wherein, the microphone sound signal and the radar echo signal are signals synchronously collected in the user's sleep environment; the radar echo signal is a discrete echo signal associated with a slow time stack, a fast time stack, and an antenna stack, and the number of the antenna stacks is multiple; Performing range dimension clutter suppression on the range-slow time dimension-antenna dimension signal to obtain a target multi-dimensional signal; Performing Doppler-FFT conversion on the target multi-dimensional signal in the slow time dimension to obtain range-Doppler spectra corresponding to multiple antenna stacks; Performing cumulative calculation on the range-Doppler spectra corresponding to multiple antenna stacks to obtain an average range-Doppler spectrum; wherein, the average range-Doppler spectrum is used to characterize the difference in the movement amplitude shown by the target part of the user's body; Based on the average range-Doppler spectrum, respectively extracting the micro-Doppler features of multiple target parts of the user's body; wherein, the micro-Doppler features are used to reflect the vital activity state of the target part; the extraction process of the micro-Doppler features specifically includes: Based on the average range-Doppler spectrum, extracting the micro-Doppler features of the user's neck, and the calculation formula is as follows: Among them, represents the first micro-Doppler feature of the neck, represents the second micro-Doppler feature of the neck, represents the first non-zero distance cell, represents the second non-zero distance cell, represents the distance cell variable, and , represents the Doppler cumulative feature of the human body along the distance change direction, represents the Doppler transform frequency, represents the average distance-Doppler spectrum; Based on the average range-Doppler spectrum, extracting the micro-Doppler features of the user's chest and abdomen, and the calculation formula is as follows: Among them, represents the third micro-Doppler feature of the chest and abdomen, represents the fourth micro-Doppler feature of the chest and abdomen, represents the sequence of the first peak, represents the sequence of the second peak; Based on the average range-Doppler spectrum, extracting the micro-Doppler features of the user's head, and the calculation formula is as follows: Among them, represents the fifth micro-Doppler feature of the head, represents the sixth micro-Doppler feature of the head, represents the non-zero logic matrix of represents the size of the number of non-zeros of represents the decision threshold, represents the 0-norm calculation, represents the difference between the maximum and minimum values of the non-zero logic along the range profile; Also including: extracting the micro-Doppler features corresponding to the user's body movement index, and the calculation formula is as follows: Among them, represents the seventh micro-Doppler feature corresponding to the body movement index, represents the eighth micro-Doppler feature corresponding to the body movement index, represents the sampling time point corresponding to the slow time stack, represents the magnitude of body movement at the moment, represents the upper limit value of the body movement index, represents the energy value corresponding to the highest energy point in the current range-Doppler spectrum, represents the minimum value among the highest energy points of all previous range-Doppler spectra; Constructing target feature vectors corresponding to multiple micro-Doppler features; Respectively inputting the target feature sequence and the target feature vector into a preset fusion processing model to output a snoring detection result.

2. The snoring detection method according to claim 1, wherein, Calculating the Mel-frequency cepstral coefficients of the microphone sound signal includes: Performing FFT transformation on the microphone sound signal and calculating the spectral line energy of each frame of spectral data obtained after the FFT transformation; Calculating the Mel-frequency energy based on each spectral line energy; Performing discrete cosine transformation on the Mel-frequency energy to obtain Mel-frequency cepstral coefficients.

3. The snoring detection method according to claim 1, wherein The performing feature splicing and dimensionality reduction processing on the Mel-frequency cepstral coefficients and the linear prediction coefficients to obtain a target feature sequence includes: Calculating a composite feature vector of the Mel-frequency cepstral coefficients and the linear prediction coefficients; Performing dimensionality reduction processing on the composite feature vector to determine a feature set corresponding to all principal component features of the composite feature vector; wherein, the cumulative contribution degree of the principal component features included in the feature set is greater than a preset threshold; Performing feature serialization processing on the feature set in the time dimension to obtain a target feature sequence.

4. The snoring detection method according to claim 1, characterized in that The fusion processing model includes an LSTM network layer, a Dropout layer, and a fully connected layer; Separately inputting the target feature sequence and the target feature vector into a preset fusion processing model to output a snoring detection result, including: Inputting the target feature sequence into the LSTM network layer to extract the temporal dependence of the target feature sequence and output the encoded high-order temporal features; Concatenating the target output feature at the last time step corresponding to the high-order temporal features with the target feature vector to obtain a fusion feature; Inputting the fusion feature into the Dropout layer and the fully connected layer in sequence to output a snoring detection result.

5. A snoring detection device, characterized in that, Including: A first calculation module for separately calculating the Mel frequency cepstral coefficients and linear prediction coefficients of the microphone sound signal; A concatenation and dimensionality reduction module for performing feature concatenation and dimensionality reduction processing on the Mel frequency cepstral coefficients and the linear prediction coefficients to obtain a target feature sequence; A second calculation module for performing FFT transformation on the radar echo signal in the fast time dimension to obtain a range-slow time dimension-antenna dimension signal; wherein, the microphone sound signal and the radar echo signal are signals synchronously collected in the user's sleep environment; the radar echo signal is a discrete echo signal associated with a slow time stack, a fast time stack, and an antenna stack, and the number of antenna stacks is multiple; Performing range dimension clutter suppression on the range-slow time dimension-antenna dimension signal to obtain a target multi-dimensional signal; Performing Doppler-FFT conversion on the target multi-dimensional signal in the slow time dimension to obtain range-Doppler spectra corresponding to multiple antenna stacks; Performing cumulative calculation on the range-Doppler spectra corresponding to multiple antenna stacks to obtain an average range-Doppler spectrum; wherein, the average range-Doppler spectrum is used to characterize the difference in the movement amplitude shown by the target body parts of the user; Based on the average range-Doppler spectrum, separately extracting the micro-Doppler features of multiple target body parts of the user; wherein, the micro-Doppler features are used to reflect the vital activity state of the target parts; the specific process of extracting the micro-Doppler features includes: Based on the average range-Doppler spectrum, extracting the micro-Doppler features of the user's neck, and the calculation formula is as follows: Among them, represents the first micro-Doppler feature of the neck, represents the second micro-Doppler feature of the neck, represents the first non-zero distance cell, represents the second non-zero distance cell, represents the distance cell variable, and , represents the Doppler cumulative feature of the human body along the distance change direction, represents the Doppler transform frequency, represents the said average range-Doppler spectrum; Based on the average range-Doppler spectrum, extracting the micro-Doppler features of the user's chest and abdomen, and the calculation formula is as follows: Among them, represents the third micro-Doppler feature of the chest and abdomen, represents the fourth micro-Doppler feature of the chest and abdomen, represents the sequence of the first peak, represents the sequence of the second peak; Based on the average range-Doppler spectrum, extracting the micro-Doppler features of the user's head, and the calculation formula is as follows: Among them, represents the fifth micro-Doppler feature of the head, represents the sixth micro-Doppler feature of the head, represents the non-zero logical matrix of represents the size of the number of non-zeros of represents the decision threshold, represents the 0-norm calculation, represents the difference between the maximum and minimum values of the non-zero logic along the range profile; Also including: extracting the micro-Doppler features corresponding to the user's body movement index, and the calculation formula is as follows: Among them, represents the seventh micro-Doppler feature corresponding to the body movement index, represents the eighth micro-Doppler feature corresponding to the body movement index, represents the sampling time point corresponding to the slow time stack, represents the magnitude of body movement at the moment, represents the upper limit value of the body movement index, represents the energy value corresponding to the highest energy point in the current range-Doppler spectrum, represents the minimum value among the highest energy points of all previous range-Doppler spectra; Constructing target feature vectors corresponding to multiple micro-Doppler features; A fusion processing module for separately inputting the target feature sequence and the target feature vector into a preset fusion processing model to output a snoring detection result.

6. An electronic device, characterized in that, Including a memory and a processor, wherein: The processor is used to execute the computer program stored on the memory; When the processor executes the computer program, it implements the steps in the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps in the method according to any one of claims 1 to 4.

Images