A method and system for multi-level fusion detection of sleep sounds for sleep staging

Abstract

The present application proposes a method and system for sleep staging by detecting sleep sounds at multiple levels of fusion, including: collecting audio data of the user during sleep; filtering the collected audio data, and performing multi-level decomposition on the filtered audio data; extracting multiple audio features using a multi-level fusion feature classification method; based on the extracted multiple audio features, using the iterative and hybrid feature selector RFINCA to select the optimal audio feature subset; sending the optimal audio feature subset to a fused classifier for classification; based on the classification results, matching with a sleep staging feature database to determine the user's sleep stage and corresponding abnormal conditions. The present application proposes a method and system for sleep staging by detecting sleep sounds at multiple levels of fusion, which solves the problem of high cost and low efficiency of sleep detection in the prior art, can reduce the cost of using sleep detection, and improve the efficiency of sleep detection.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a method and system for multi-level fusion detection of sleep sounds for sleep staging. Background Art

[0002] As people pay more attention to sleep quality, they find that breathing problems, emotions, irregular breathing, nutritional digestion problems, and dreams can adversely affect sleep quality and lead to many health problems. Sleep apnea, cardiovascular problems, insulin resistance, and psychological problems are the most common sleep-related problems. It can also have a negative impact on sleeping partners, social environment, and work life. Sleep disorders or diseases affect daily life and may disrupt reason, leading to a decline in people's quality of life.

[0003] In the prior art, detection of sleep problems is determined through tests conducted by a sleep diagnosis center. This current detection method requires the user to enter a laboratory environment or a medical clinic to fall asleep, and to connect electrodes and sensors to various parts of the body to measure numerous parameters, such as electroencephalogram, nasal and oral airflow, electromyogram, electrocardiogram, blood oxygen saturation and eye movement. This is not only expensive and requires numerous equipment and medical personnel, but also requires a lot of energy and is inefficient. Summary of the invention

[0004] In view of the above, the present application proposes a method and system for sleep staging by multi-level fusion detection of sleep sounds, which solves the problem of high cost and low efficiency of sleep detection in the prior art, can reduce the cost of sleep detection and improve the efficiency of sleep detection.

[0005] The present application provides a method for multi-level fusion detection of sleep sounds for sleep staging, the method comprising:

[0006] Collect audio data during the user's sleep;

[0007] Filtering the collected audio data, and performing multi-level decomposition on the filtered audio data;

[0008] Use a multi-level fusion feature classification method to extract multiple audio features;

[0009] Based on the extracted multiple audio features, using the iterative and hybrid feature selector RFINCA to select the optimal audio feature subset;

[0010] Sending the optimal audio feature subset to the fused classifier for classification;

[0011] Based on the classification result, matching is performed with a sleep stage feature database to determine the user's sleep stage and the corresponding abnormal situation.

[0012] In some embodiments, an iterative and hybrid feature selector RFINCA is used to select the best audio features, including:

[0013] Using the ReliefF algorithm to filter the multiple audio features to obtain a first feature subset;

[0014] The first feature subset is optimized using neighbor component analysis (NCA) to obtain a second feature subset;

[0015] The above screening and optimization process is iterated until the optimal audio feature subset is obtained.

[0016] In some embodiments, the plurality of audio features are screened using a ReliefF algorithm to obtain a first feature subset, including:

[0017] Randomly select a sample Ri from the multiple audio features, and find k nearest neighbor samples of Ri in the feature space;

[0018] For each feature f, calculate the distance difference between Ri and its nearest neighbor of the same type on f, diff(f,Ri,H), and the distance difference between Ri and its nearest neighbor of a different type on f, diff(f,Ri,M);

[0019] Update the weight of each feature f:

[0020]

[0021] in,

[0022] Ri is a randomly selected training sample, i is the i-th natural number, and j is the j-th natural number;

[0023] k is the number of nearest neighbor samples found in the feature space;

[0024] f represents a specific feature;

[0025] diff(f,Ri,H) is the distance difference between Ri and its nearest neighbor of the same type (same label) on feature f;

[0026] diff(f,Ri,M) is the distance difference between Ri and its heterogeneous (differently labeled) nearest neighbor on feature f;

[0027] W[f] is the weight of feature f;

[0028] m is the total number of samples;

[0029] Hj is the jth nearest neighbor sample of the same type as Ri;

[0030] Mj(C) is the jth nearest neighbor sample that is different from Ri, where C represents a category different from Ri;

[0031] P(C) is the prior probability of category C;

[0032] class(Ri) is the class of Ri;

[0033] Traverse all training samples;

[0034] According to the feature weight W, they are sorted from high to low, and the Ns features with the highest weight are selected to form the first feature subset S.

[0035] In some embodiments, the first feature subset is optimized using neighbor component analysis (NCA) to obtain a second feature subset, including:

[0036] The weighted distance between samples i and j is calculated based on the first feature subset S:

[0037]

[0038] The similarity between samples i and j is calculated based on the weighted distance:

[0039]

[0040] Calculate the expected probability for each sample

[0041] Calculate the objective function f(w) = ∑ i p i ;

[0042] Finding the optimal solution of the feature weight w to obtain the optimized second feature subset;

[0043] in,

[0044] S is the feature subset after being screened by the ReliefF algorithm;

[0045] wf is the weight of feature f in S;

[0046] xif is the value of sample i on feature f;

[0047] xjf is the value of sample j on feature f;

[0048] dijS is the weighted Euclidean distance between samples i and j under feature subset S;

[0049] pij is the probability that sample j is selected as the nearest neighbor by sample i under feature subset S;

[0050] pi is the expected probability that sample i is selected as the nearest neighbor by its similar samples;

[0051] f(w) is the objective function, which represents the sum of the probabilities of all samples being selected as nearest neighbors by their similar samples. The goal is to maximize this probability by adjusting the feature weight w.

[0052] In some embodiments, the filtered audio data is decomposed into multiple levels, including:

[0053] Perform the first-level discrete wavelet transform DWT decomposition on the original sleep audio T1 to obtain the first low-frequency component low1 and the first high-frequency component high1;

[0054] The low-frequency component low1 is subjected to the second-level DWT decomposition to obtain the second low-frequency component low2 and the second high-frequency component high2;

[0055] The second low-frequency component low2 is subjected to the next level of DWT decomposition to complete the iterative operation of the multi-level decomposition.

[0056] In some embodiments, performing a first level discrete wavelet transform DWT decomposition on the original sleep audio T1 to obtain a first low frequency component low1 and a first high frequency component high1 includes:

[0057] Construct a DWT filter and decomposition layer number, and input T1 into the DWT filter;

[0058] Using the DWT filter to perform low-pass and high-pass filtering on T1, respectively obtaining a low-frequency component low1 and a high-frequency component high1;

[0059] Downsample low1 to get the scale coefficient, and downsample high1 to get the wavelet coefficient.

[0060] In some embodiments, a multi-level fusion feature classification method is used to extract multiple audio features, including:

[0061] A low-pass filter is used to extract texture features of multiple low-frequency components;

[0062] Based on the extracted texture features, multiple audio features are extracted using local binary model LBP and local ternary pattern LTP.

[0063] In some embodiments, a low-pass filter is used to extract texture features of multiple low-frequency components, including:

[0064] The low-frequency component low1 obtained by the first-level DWT decomposition is used to extract the texture feature tpf1 using a low-pass filter;

[0065] The low-frequency components lowi (i=2, 3, ..., N) obtained by the 2nd to Nth level DWT decomposition are extracted using a low-pass filter to extract the texture features tpfi;

[0066] The local binary model LBP and the local ternary pattern LTP are used to extract multiple audio features, including:

[0067] Extracting local binary pattern feature lbpf1 using LBP operator for the texture feature tpf1 extracted at the first stage;

[0068] Extracting local ternary pattern feature ltpf1 using LTP operator for the texture feature tpf1 extracted at the first stage;

[0069] Extract LBP and LTP features from the texture features tpfi (i=2,3,...,8) extracted from the 2nd to Nth levels to obtain lbpfi and ltpfi;

[0070] The LBP features lbpf1, lbpf2, ..., lbpfN and the LTP features ltpf1, ltpf2, ..., ltpfN extracted at N levels are cascaded to form multi-level LBP feature vectors and multi-level LTP feature vectors, and the multi-level LBP feature vectors and the multi-level LTP feature vectors together constitute the multiple audio features.

[0071] The present application also provides a system for multi-level fusion detection of sleep sounds for sleep staging, including:

[0072] A collection module, used to collect audio data during the user's sleep;

[0073] A decomposition module, used for filtering the collected audio data and performing multi-level decomposition on the filtered audio data;

[0074] An extraction module, used for extracting multiple audio features using a multi-level fusion feature classification method;

[0075] A selection module, configured to select an optimal audio feature subset based on the extracted multiple audio features using an iterative and hybrid feature selector RFINCA;

[0076] A classification module, used for sending the optimal audio feature subset to the fused classifier for classification;

[0077] The judgment module is used to match the classification result with the sleep stage feature database to judge the sleep stage of the user and the corresponding abnormal situation.

[0078] An embodiment of the present application also provides a system for multi-level fusion detection of sleep sounds for sleep staging, including a memory and a processor, wherein the memory stores computer executable instructions, and the processor implements the above method when running the computer executable instructions on the memory.

[0079] In this application, only the microphone of a mobile smart device is used to collect audio during sleep, and the sleep staging can be evaluated in a lightweight manner through database data matching; multi-level DWT filtering and fusion of multiple fusion methods are used to extract the characteristics of non-stationary speech activities, and the final data matching is obtained through multiple classifier fusion classification methods, so as to achieve the effect of detecting sleep staging at night, which is lighter, more efficient and less costly; at the same time, it can assist in the detection of sleep problems, making it convenient for users to pay timely attention to the status of sleep health and discover potential sleep problems early. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is a flowchart of a method for performing sleep staging by multi-level fusion detection of sleep sounds provided in an embodiment of the present application.

[0081] Figure 2 This is a flowchart of a method for performing sleep staging by multi-level fusion detection of sleep sounds provided in another embodiment of the present application.

[0082] Figure 3 This is a flowchart of extracting multiple audio features using a multi-level fusion feature classification method provided by an embodiment of the present application.

[0083] Figure 4 It is a flowchart of using the iterative and hybrid feature selector RFINCA to select the optimal audio feature provided by an embodiment of the present application.

[0084] Figure 5 It is a functional module diagram of a system for performing sleep staging by detecting sleep sounds at multiple levels provided in one embodiment of the present application.

[0085] Figure 6 It is a functional module diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0086] In order to more clearly understand the purpose, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. In the following description, many specific details are set forth to facilitate a full understanding of the present application, and the embodiments described are only a part of the embodiments of the present application, rather than all of the embodiments.

[0087] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0088] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art to which the present application belongs. The terms used herein in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0089] The embodiments of the present application aim to solve the problems of existing sleep problem tests being expensive, inefficient, and requiring a certain environment, and to provide users with a sleep quality monitoring method that is low-cost, highly efficient, and does not require complex equipment or medical support in a home environment.

[0090] Specifically, the embodiment of the present application detects sleep sounds through a non-invasive method to stage sleep and then analyze possible sleep problems. The embodiment of the present application mainly provides sleep monitoring for the user's home sleeping place, without the need for time-consuming and costly sleep monitoring in an external laboratory or medical environment. By collecting and analyzing audio data through smart devices (such as smart speakers, smart phones, smart bracelets or dedicated monitoring equipment), users can obtain sleep monitoring results close to laboratory levels at home.

[0091] By analyzing audio data and classifying sleep stages, users can promptly discover and pay attention to sleep problems that may affect their health. Compared with traditional sleep monitoring equipment, this solution does not require professional equipment and operators, greatly reducing costs and usage barriers. In addition, the solution can quickly provide feedback to users and improve detection efficiency. By integrating it into smart terminal devices, users can complete sleep monitoring in their own homes, further enhancing the convenience of sleep quality management.

[0092] In addition, the embodiments of the present application can also be applied to remote medical monitoring, long-term sleep data collection, mental health assessment, etc. In these scenarios, the data obtained through audio monitoring can provide diagnostic references for remote doctors, and can also be used for personal long-term health records, or combined with mental state assessment to provide users with more comprehensive health management services.

[0093] The following is an explanation of the techniques used in the embodiments of the present application:

[0094] Discrete Wavelet Transform (DWT): It is a time and frequency analysis tool that can decompose the signal into different frequency components and is used to extract the characteristics of the signal.

[0095] Local Binary Pattern (LBP): is a texture analysis method that generates features by comparing the relationship between a pixel and its surrounding pixels for pattern recognition tasks.

[0096] Local Ternary Pattern (LTP): It is an extension of LBP, which adds a tolerance for grayscale changes and improves the stability of feature extraction.

[0097] Iterative and Hybrid Feature Selector (RFINCA): is a feature selection mechanism that combines the ReliefF algorithm and Neighbor Component Analysis (NCA) to filter the most informative feature subset from a large number of features.

[0098] Example 1: A flowchart of a method for multi-level fusion detection of sleep sounds for sleep staging provided in an embodiment of the present application. Specifically, the method includes the following:

[0099] 1. Data collection:

[0100] Use the smart device's microphone to collect audio data during sleep activities.

[0101] 2. Data filtering:

[0102] The collected audio data is filtered to remove invalid audio such as notification audio from the mobile phone and audio generated by turning over.

[0103] 3. Feature extraction:

[0104] A multi-level fusion feature classification method is used to extract the features of audio files level by level.

[0105] A total of 825 features were extracted in 8 levels, and a total of 6600 features were extracted.

[0106] 4. The feature extraction process includes:

[0107] Texture feature extraction: A total of 768 features are generated by extracting non-stationary speech activity using the local binary model (4) and the local ternary model (5).

[0108] Statistical feature generation: Process texture features to generate statistical features.

[0109] 5. Feature Selection:

[0110] The RFINCA method (fusion of ReliefF algorithm and nearest component analysis NCA) was used for feature selection.

[0111] First, the ReliefF algorithm is used to eliminate redundant features.

[0112] NCA is applied to the selected positively weighted features and iterative feature selection is performed to improve accuracy.

[0113] Select the 120 most meaningful features from 6600 features.

[0114] 6. Classification:

[0115] Distribute the selected features to the classifier.

[0116] The ensemble classifiers used are: DT classifier, MA moving average classification, LD linear classifier, SVM second-order and third-order polynomial and nearest neighbor KNN classification algorithm.

[0117] Perform regression, outlier analysis, ranking, and classification.

[0118] 7. Data matching:

[0119] Match the classification results with the established sleep stage data type library.

[0120] Select the data results with the confidence value reaching the standard for receipt.

[0121] 8. Sleep staging and abnormal sound identification:

[0122] The corresponding sleep stage characteristics in the database are integrated to generate a single data analysis report.

[0123] Sleep is divided into the falling asleep period, diving period, deep sleep period, and eye movement period.

[0124] Compare abnormal sounds that accompany sleep, such as frequent tossing and turning, talking in sleep, coughing, snoring, etc., and use artificial intelligence algorithm to compare the model to ultimately determine the status.

[0125] 9. Result feedback:

[0126] The generated data analysis report is returned to the user's mobile smart device in a visual form.

[0127] Figure 1 This is a flowchart of multi-level fusion detection of sleep sounds for sleep staging provided in Example 1 of the present application, combined with Figure 1 , illustratively, the above steps may specifically be:

[0128] The recorded sleep audio is first processed by DWT filter for noise reduction and decomposition.

[0129] The filtered audio data is divided into two types of features:

[0130] Texture features extracted by low-pass filter;

[0131] Statistical features generated based on texture features;

[0132] The extracted texture features and statistical features are passed to the NCA module for feature value calculation.

[0133] Among them, the feature values ​​calculated by the NCA module are passed to the classifier module, including:

[0134] MA (moving average) classifier;

[0135] KNN (K nearest neighbor) classifier;

[0136] DT (Decision Tree) classifier;

[0137] SVM (Support Vector Machine) classifier, using second-order and third-order polynomial kernel functions;

[0138] The results of each classifier are integrated to generate a conclusion report for the artificial intelligence analysis service.

[0139] Finally, the conclusion report is interpreted by the state module, which divides sleep into:

[0140] Sleeping period

[0141] Diving period

[0142] Deep sleep period

[0143] Eye movement period

[0144] And identify abnormal sounds during sleep, such as frequent tossing and turning, talking in sleep, coughing and snoring.

[0145] The detailed analysis results generated by the status module are finally presented to the user's mobile smart device in a visual form.

[0146] Figure 2 This is a flowchart of a method for performing sleep staging by multi-level fusion detection of sleep sounds provided in an embodiment of the present application.

[0147] like Figure 2 As shown, the embodiment of the present application specifically includes the following method:

[0148] S10, collecting audio data of the user during sleep;

[0149] The embodiments of the present application can use different smart devices to collect audio data through their microphones, such as smart phones, tablet computers or other devices with wireless communication functions, for collecting audio data during the user's sleep. Such devices are usually equipped with audio collection hardware (for example, built-in microphones), storage modules, processors and communication modules. The electronic device transmits data with the background server through communication networks such as Wi-Fi, Bluetooth, NFC, etc., and the server is responsible for storing and further processing the collected audio data.

[0150] Exemplarily, the smart device includes the following modules:

[0151] Hardware composition:

[0152] Processor: As the main computing unit of electronic equipment, it is responsible for executing audio data acquisition, preprocessing, wavelet transform decomposition and feature extraction algorithms.

[0153] External Storage Interface: Connects to external storage devices to store original audio data and intermediate data during processing.

[0154] Internal Memory: It is used to store programs and temporary data required by the processor to perform operations.

[0155] Communication Module: includes functional modules such as Wi-Fi and Bluetooth, responsible for data transmission between the device and the server or other devices.

[0156] Sensor Module: Integrates ambient light sensors, motion sensors, etc. to assist in adjusting audio acquisition settings to adapt to environmental changes.

[0157] Display Screen: Displays the operation interface and analysis results.

[0158] Audio module: includes a microphone for collecting sound and a speaker for providing feedback or prompting messages.

[0159] The software includes:

[0160] Operating system (OS): Provides basic software support for electronic devices and manages hardware resources, such as Android or iOS.

[0161] Application Layer: User interaction interface, collecting user input, and displaying data analysis results.

[0162] Audio processing application: software specifically used to process audio data, implementing basic processing functions such as audio recording, storage, filtering, wavelet conversion, etc.

[0163] Feature extraction and analysis module: software module that uses discrete wavelet transform, low-pass filter, LBP and LTP operators to extract relevant features from audio data.

[0164] Feature selection module: Use the ReliefF algorithm and NCA to select features and optimize feature subsets.

[0165] Classifier and matching module: used to match the extracted audio features with the existing sleep stage features and determine the user's sleep stage.

[0166] S20, filtering the collected audio data, and performing multi-level decomposition on the filtered audio data;

[0167] In S20, common sleep apnea, snoring, and sleep stage detection during sleep activities: awake, REM, light sleep, deep sleep, coughing, and sleep talking are classified, and the recording is collected using the microphone of the smart device to collect audio data; the obtained audio data is filtered once to remove invalid audio filters such as notification audio from the mobile phone and audio generated by turning over. Then, the audio is decomposed in multiple levels using discrete wavelet transform (DWT), preferably, 8 levels of decomposition are performed.

[0168] In S20, the filtered audio data is decomposed into multiple levels, including the following steps:

[0169] S201. Original sleep audio T 1 Perform the first level discrete wavelet transform DWT decomposition; get the first low frequency component low 1 and the first high frequency component high 1 ;

[0170] Among them, the original sleep audio T 1 Perform the first level discrete wavelet transform DWT decomposition; get the first low frequency component low 1 and the first high frequency component high 1 , including the following steps:

[0171] Construct DWT filter and decomposition layer, and convert T 1 Input into the DWT filter;

[0172] Using the DWT filter, T 1 Perform low-pass and high-pass filtering to obtain the low-frequency components low 1 and high frequency components 1 ;

[0173] To low 1 Downsample to get the scale factor, and 1 Downsampling is performed to obtain wavelet coefficients.

[0174] S202. The low frequency component is low 1 Perform the second-level DWT decomposition to obtain the second low-frequency component low 2 and the second high frequency component high 2 ;

[0175] Use low 1 As the new input, perform the second level DWT decomposition and repeat the above steps to get low2 and high 2 Specifically, 1 Input into DWT as a new decomposition object; for low 1 Perform low-pass and high-pass filtering to obtain the low-frequency components low 2 and high frequency components 2 ; for low 2 Downsample to get the scale factor, for high 2 Downsampling is performed to obtain wavelet coefficients.

[0176] S203. The second low frequency component low 2 The next level of DWT decomposition is performed to complete the iterative operation of multi-level decomposition.

[0177] Use low 2 As input, perform the third-level DWT decomposition and repeat step S201 to obtain low 3 and high 3 .

[0178] And so on, the low-frequency components obtained at each level are used as the input of the next level until N (preferably, N = 8) levels of DWT decomposition are performed:

[0179] The i-th level decomposition formula is: [low i ,high i ]=DWT({lowi-1}),i=2,3,...,N

[0180] Each level has low frequency components i Downsampling is performed to obtain the scale coefficient, and the high frequency component is high i Downsampling is performed to obtain wavelet coefficients.

[0181] In this way, after completing the 8-level DWT decomposition, 8 groups of low-frequency components low1, low2, ..., low8, and 8 groups of high-frequency components high1, high2, ..., high8 can be obtained.

[0182] In this way, the low-frequency and high-frequency components of each level can be further feature extracted and analyzed as needed.

[0183] By performing 8-level DWT decomposition on the sleep audio signal, the signal can be divided into different sub-bands from high to low frequency, and time-frequency representations of different scales can be obtained, which is helpful for extracting multi-scale features of the audio. The low-frequency component contains the main features of the signal, and the high-frequency component contains the detailed features of the signal. Components at different levels can be selected for feature extraction and analysis as needed for subsequent sleep stage discrimination tasks.

[0184] In summary, the main steps of using DWT to perform 8-level decomposition of audio can more comprehensively characterize the time-frequency characteristics of audio signals through multi-layer wavelet decomposition.

[0185] S30, extracting multiple audio features using a multi-level fusion feature classification method;

[0186] like Figure 3 As shown, in S30, a multi-level fusion feature classification method is used to extract multiple audio features, including S301-S302:

[0187] S301. Using a low-pass filter to extract texture features of multiple low-frequency components;

[0188] In some embodiments, a suitable low-pass filter is selected, such as Haar wavelet, Daubechies wavelet, etc.

[0189] Among them, a low-pass filter is used to extract the texture features of multiple low-frequency components, specifically including:

[0190] The low frequency component low obtained by the first-level DWT decomposition 1 Use low-pass filter to extract texture features tpf 1 ; That is, use the selected low-pass filter to 1 Perform convolution operation to obtain texture feature tpf 1 ;

[0191] The low frequency components obtained by decomposing the 2nd to Nth level DWT are low i (i=2,3,...,N) Use low-pass filter to extract texture features tpf i ;

[0192] That is, use the same low-pass filter to i Perform convolution operation to obtain the texture feature tpf of level i i , repeat the above steps until all 8 levels of texture features tpf are obtained 1 ,tpf 2 ,...,tpf 8 .

[0193] S302. Based on the extracted texture features, a local binary model LBP and a local ternary pattern LTP are used to extract multiple audio features.

[0194] In S302, a local binary model LBP and a local ternary pattern LTP are used to extract multiple audio features, specifically including:

[0195] The texture feature tpf extracted at the first level 1 Use LBP operator to extract local binary pattern features lbpf1 ;

[0196] Specifically, select tpf 1 For each pixel in the image, compare it with the surrounding 8 pixels; if the central pixel is greater than or equal to the adjacent pixel, the corresponding position is 1, otherwise it is 0, forming an 8-bit binary number; convert the binary number to a decimal number as the LBP feature value of the pixel; calculate the tpf 1 The LBP feature values ​​of all pixels are obtained to obtain the LBP feature histogram lbpf 1 .

[0197] The texture feature tpf extracted at the first level 1 Use LTP operator to extract local ternary pattern feature ltpf 1 ;

[0198] Specifically, select tpf 1 For each pixel in the image, compare it with the surrounding 8 pixels; if the adjacent pixel is greater than the central pixel threshold, the corresponding position is 1; if it is less than the threshold, it is -1; otherwise it is 0, forming an 8-bit ternary number; divide the ternary number into two binary numbers according to the positive and negative signs, and convert them into decimal numbers as the upper and lower LTP feature values ​​of the pixel; count the upper and lower LTP feature values ​​of all pixels to obtain the LTP feature histogram ltpf 1 .

[0199] Repeat the above steps to extract texture features tpf from the 2nd to the Nth level. i (i=2,3,...,8) extract LBP and LTP features and get lbpf i and ltpf i ;

[0200] The LBP features lbpf1,lbpf extracted from N levels 2 ,...,lbpf N and LTP feature ltpf 1 ,ltpf 2 ,...,ltpf N Cascading to form a multi-level LBP feature vector and a multi-level LTP feature vector, the multi-level LBP feature vector and the multi-level LTP feature vector together constitute the multiple audio features.

[0201] For example, the LBP features lbpf extracted from 8 levels are 1 ,lbpf 2 ,...,lbpf 8 and LTP feature ltpf 1 ,ltpf 2 ,...,ltpf8 Cascading to form a 576-dimensional multi-level LBP feature vector and a 768-dimensional multi-level LTP feature vector.

[0202] Preferably, in the embodiment of the present application, 825 features are extracted at each level, and a maximum of 6600 features are extracted.

[0203] Through the above steps, the texture features of the audio signal are first extracted in 8 frequency bands using a DWT low-pass filter, and then based on each level of texture features, the LBP and LTP operators are used to further extract local rotation-invariant statistical features. Multi-level LBP and LTP features can characterize the texture and statistical properties of audio at different scales and frequencies, providing richer discriminant information for subsequent feature selection and classification.

[0204] Optionally, in one of the embodiments, LBP can also be replaced by a local phase quantization (LPQ) algorithm. The LPQ algorithm performs texture analysis based on the phase information of the image in the frequency domain and has better resistance to blur and rotation changes.

[0205] When the LPQ algorithm processes a sound signal, the processor first needs to calculate the short-time Fourier transform (STFT) of the signal in a selected window, and then analyze the phase information of each window to extract features. These features correspond to the frequency domain characteristics of the sound data and can help identify and classify subtle changes in sleep quality.

[0206] The specific steps include:

[0207] Adjust the audio data preprocessing process, including the selection of window function, to meet the needs of the LPQ algorithm.

[0208] The window function suitable for LPQ is defined, usually a rectangular window or a Hanning window is used to reduce boundary effects. The window length is determined to adapt the time-frequency resolution required by the short-time Fourier transform (STFT).

[0209] Short-time Fourier transform (STFT) is performed on the preprocessed audio data to extract frequency domain features.

[0210] The phase information of the frequency components within each window is captured, which is the basis of the LPQ algorithm.

[0211] The results of STFT are used to calculate the phase angle of each frequency point of the audio signal in different windows. The extraction of phase information is the key part of the LPQ algorithm to distinguish features.

[0212] Update the feature extraction module, replace the LBP algorithm with the LPQ algorithm, and perform corresponding software updates and optimizations.

[0213] Implement the LPQ algorithm in the software module to extract the phase information of the signal as a feature. Determine and encode the mathematical model and algorithm steps for LPQ feature extraction. Commonly used LPQ mathematical models and algorithms can be selected and will not be repeated here. Integrate the LPQ algorithm into the feature extraction module and ensure compatibility with the preprocessing module. Update the user interface to ensure that users can continue to use the system for sleep monitoring seamlessly.

[0214] Retrain the classification model to adapt to the new feature set and ensure that the classification accuracy is maintained or improved.

[0215] Re-evaluate the performance of the entire sleep monitoring system to determine the impact of the LPQ algorithm and make necessary adjustments.

[0216] Specifically, the collected audio dataset is used to extract LPQ features, and data cleaning and normalization are performed to meet the input requirements of the classification model.

[0217] Retrain the classifier using the machine learning algorithm of your choice such as support vector machine (SVM) or a deep learning model such as convolutional neural network (CNN).

[0218] Combine cross-validation during training to evaluate model performance. Evaluate the new model's classification accuracy for each sleep stage and its ability to detect abnormal events. Adjust model parameters based on the evaluation results or further optimize feature selection strategies.

[0219] S40, based on the extracted multiple audio features, using the iterative and hybrid feature selector RFINCA to select the optimal audio feature subset;

[0220] In S40, an iterative and hybrid feature selector (RFINCA) is used to select the most valuable features: RFINCA combines the ReliefF and Neighbor Component Analysis (NCA) algorithms; ReliefF is used to eliminate redundant features, and NCA is used to iteratively select positive weighted features. Therefore, the RFINCA method combines the two feature selection algorithms, ReliefF and Neighbor Component Analysis (NCA), to select the best feature subset in an iterative and hybrid manner.

[0221] Specifically, Figure 4 As shown, in S40, the iterative and hybrid feature selector RFINCA is used to select the optimal audio features, including S401-S403:

[0222] S401. Using the ReliefF algorithm to filter the multiple audio features to obtain a first feature subset;

[0223] The ReliefF algorithm assigns a weight to each feature, and the higher the weight, the more important the feature. Some irrelevant features can be quickly removed through the ReliefF initial screening. Specifically, in S401, the ReliefF algorithm is used to screen the multiple audio features to obtain a first feature subset, including the following steps A1-A4:

[0224] A1: randomly select a sample Ri from the multiple audio features, and find k nearest neighbor samples of Ri in the feature space;

[0225] A2: For each feature f, calculate the distance difference between Ri and its nearest neighbor of the same type on f, diff(f,Ri,H), and the distance difference between Ri and its nearest neighbor of a different type on f, diff(f,Ri,M);

[0226] Update the weight of each feature f:

[0227]

[0228] in,

[0229] Ri is a randomly selected training sample, i is the i-th natural number, and j is the j-th natural number;

[0230] k is the number of nearest neighbor samples found in the feature space;

[0231] f represents a specific feature;

[0232] diff(f,Ri,H) is the distance difference between Ri and its nearest neighbor of the same type (same label) on feature f;

[0233] diff(f,Ri,M) is the distance difference between Ri and its heterogeneous (differently labeled) nearest neighbor on feature f;

[0234] W[f] is the weight of feature f;

[0235] m is the total number of samples;

[0236] Hj is the jth nearest neighbor sample of the same type as Ri;

[0237] Mj(C) is the jth nearest neighbor sample that is different from Ri, where C represents a category different from Ri;

[0238] P(C) is the prior probability of category C;

[0239] class(Ri) is the class of Ri;

[0240] A3: Traverse all training samples;

[0241] Repeat the A1-A2 method and traverse all training samples.

[0242] A4: Sort the features from high to low according to their weight W, and select the Ns features with the highest weight to form a first feature subset S. S402. Optimize the first feature subset using nearest component analysis NCA to obtain a second feature subset;

[0243] NCA optimizes feature weights to minimize the distance between samples of the same type and maximize the distance between samples of different types, thereby obtaining a feature subset with the best classification performance. In S402, the first feature subset is optimized using nearest component analysis (NCA) to obtain a second feature subset, including the following steps B1-B5:

[0244] B1. Calculate the weighted distance between samples i and j based on the first feature subset S:

[0245]

[0246] B2. Calculate the similarity between samples i and j based on the weighted distance:

[0247]

[0248] B3. Calculate the expected probability of each sample

[0249] B4. Calculate the objective function f(w) = ∑ i p i ;

[0250] B5. Find the optimal solution of the feature weight w to obtain the optimized second feature subset;

[0251] in,

[0252] S is the feature subset after being screened by the ReliefF algorithm;

[0253] wf is the weight of feature f in S;

[0254] x if is the value of sample i on feature f;

[0255] x jf is the value of sample j on feature f;

[0256] d ij S is the weighted Euclidean distance between samples i and j under feature subset S;

[0257] p ij is the probability that sample j is selected as the nearest neighbor by sample i under feature subset S;

[0258] pi is the expected probability that sample i is selected as the nearest neighbor by its similar samples;

[0259] f(w) is the objective function, which represents the sum of the probabilities of all samples being selected as nearest neighbors by their similar samples. The goal is to maximize this probability by adjusting the feature weight w.

[0260] By iteratively using ReliefF and NCA for rough screening and selection, RFINCA can efficiently select the optimal feature combination. ReliefF focuses on the importance evaluation of individual features, while NCA focuses on the overall optimization of feature subsets. The two complement each other and can obtain a feature subset that is both representative and discriminative.

[0261] S403. Iterate the above screening and optimization process until the optimal audio feature subset is obtained.

[0262] Specifically, steps S401-402 are repeated until the feature subset no longer changes or a preset number of iterations is reached; and the final feature subset is output as a selection result.

[0263] In this way, by iteratively using ReliefF and NCA for rough screening and selection, RFINCA can efficiently select the optimal feature combination. ReliefF focuses on the importance evaluation of individual features, while NCA focuses on the overall optimization of feature subsets. The two complement each other and can obtain a feature subset that is both representative and discriminative.

[0264] S50, sending the optimal audio feature subset to the fused classifier for classification;

[0265] Specifically, S50 distributes the selected features to the fused classifiers for classification, including decision tree (DT), moving average (MA), linear discriminant (LD), support vector machine (SVM) and K nearest neighbor (KNN) classifiers. The fusion methods include but are not limited to weighted fusion, parallel fusion, series fusion and other fusion methods.

[0266] The algorithms in this step include decision tree (DT), moving average (MA), linear classifier (LD), support vector machine (SVM) and K nearest neighbor (KNN). The purpose of this step is to analyze the optimal audio feature subset of the input through these classifiers and perform regression, outlier analysis, sorting and classification operations.

[0267] The specific description of how each classifier processes audio features is as follows:

[0268] Decision Tree (DT): It establishes a series of rules based on the input features and classifies the data according to these rules. Its main advantage is that the model is easy to understand and the computational cost is not high.

[0269] Moving Average (MA): Mainly used for classification based on characteristic value trends, it is often used in stock market analysis, but can also be applied to pattern recognition.

[0270] Linear classifier (LD): Distinguish different categories by constructing a classification surface of one or more linear combinations, such as logistic regression or linear discriminant analysis.

[0271] Support Vector Machine (SVM): SVM finds the best segmentation hyperplane in the feature space to achieve sample classification, which is suitable for binary classification and multi-classification problems.

[0272] K Nearest Neighbor (KNN): Classification is based on the categories of the closest K neighbors. It is an instance-based learning method that does not require an explicit training process.

[0273] In the present application embodiment:

[0274] For decision trees (DTs), information gain or Gini impurity will be used as the branching criteria, and the depth and number of branches will be determined based on the characteristics of the dataset. For example, if the data represents loud snoring with high intermittency, it may be classified as sleep apnea.

[0275] When using a moving average (MA), an appropriate time window length is selected to calculate the average, which is moved across all features to smooth the trend line and perform classification. If the feature value shows a smoothed line of audio intensity that continues to rise over a period of time, it may indicate that the user has entered a deep sleep state.

[0276] For the linear classifier (LD), the optimal weight vector is determined during the training phase so that the boundaries between different classes are as clear as possible. For example, a separating linear boundary is drawn based on the frequency and amplitude characteristics of the user's sleep sounds to distinguish normal sleep sounds from snoring sounds.

[0277] In SVM, different kernel functions such as linear kernel, polynomial kernel or radial basis function (RBF) may be selected to adapt to the data characteristics, and cross-validation is used to optimize the hyperparameters.

[0278] For the KNN algorithm, cross-validation is usually used to determine the value of K to ensure that it can maintain good classification performance without being too complex and resulting in excessive computational costs. Specifically, if the three nearest neighbors of a sleep audio clip in the feature space are all deep sleep, then the clip will also be classified as deep sleep.

[0279] In the embodiment of the present application, DT classifier and MA moving average classification are integrated (for example, weighted fusion, parallel or series connection, etc.), and regression, outlier analysis, sorting and classification are performed through LD linear classifier, SVM second-order and third-order polynomial and nearest neighbor KNN classification algorithm; finally, the data is matched with the established database, and the sleep state characteristic values ​​are used to divide the sleep into stages, namely, the falling asleep stage, the diving stage, the deep sleep stage and the eye movement stage; at the same time, the abnormal sounds accompanying sleep, such as frequent tossing and turning, talking in sleep, coughing, snoring, etc. are compared, and the model is compared through the artificial intelligence algorithm to finally determine the state, and the information is generated into a data visualization report, which is fed back to the user's mobile smart device.

[0280] S60: Based on the classification result, matching is performed with a sleep stage feature database to determine the sleep stage of the user and the corresponding abnormal situation.

[0281] In the embodiment of the present application, the matching results can be used to generate a visual sleep analysis report; and the report can be fed back to the user's smart mobile device.

[0282] Take the elderly as an example. Due to physical reasons, the elderly may be more prone to sleep problems, which may include sleep breathing disorders, sleep cycle disorders, etc. Therefore, monitoring the sleep quality of the elderly is crucial to maintaining their health. Traditional sleep monitoring requires going to professional institutions and often requires multiple sensors to record physiological activities, which is not only inconvenient and uncomfortable, but also costly. This requires a simple, comfortable, and low-cost sleep monitoring method.

[0283] Equipment deployment: Place smart devices, such as smartphones or dedicated sleep monitoring devices, in the elderly’s sleeping environment to ensure that the device can stably collect sleep sound data without disturbing the elderly.

[0284] Audio data collection and processing: When the elderly are sleeping, the device automatically starts recording audio. The collected raw audio data will first be filtered to remove irrelevant information such as environmental noise, and then multi-level decomposition will be performed through discrete wavelet transform (DWT) to obtain different frequency components.

[0285] Feature extraction: With the help of local binary patterns (LBP) and local ternary patterns (LTP), texture features and other statistical features are extracted from the low-frequency and high-frequency components of each level.

[0286] Feature selection: Using the iterative and hybrid feature selector RFINCA, the most representative feature subsets are selected from the large number of extracted features. These features will help to accurately evaluate the sleep data.

[0287] Classification and analysis: By fusing classifiers, the sleep sounds of the elderly are classified according to the selected feature subsets, such as judging the depth of sleep, whether snoring or apnea occurs, etc., and matching them with the sleep stage feature database.

[0288] Feedback: The results of the analysis can be fed back directly to the user or their caregiver through the application interface of the smart device. For example, if possible sleep apnea is detected, the application can issue an alert and recommend further medical diagnosis and treatment.

[0289] For example, Mr. Zhang wants to check his sleep condition, so he opens the sleep detection app on his phone before going to bed, turns on the microphone to record and goes to sleep. The phone records Mr. Zhang's sleep audio data for the whole night through the microphone.

[0290] The next day, the audio data was uploaded to a cloud server for analysis:

[0291] First, the audio is filtered for noise reduction and decomposed into 8 levels using DWT;

[0292] Then, LBP and LTP were used to extract multi-level texture statistical features, and a total of 6600 features were extracted;

[0293] Then RFINCA was used to select the 120 most valuable features;

[0294] Afterwards, the selected features are input into classifiers such as DT and SVM for classification and matching;

[0295] The matching results showed that Mr. Zhang experienced the sleep stages of falling asleep, light sleep, deep sleep, and rapid eye movement that night, and snored. It was initially determined that Mr. Zhang might have sleep apnea.

[0296] Finally, the analysis results are generated into a report and pushed to Mr. Zhang's mobile phone, reminding him to pay attention to his sleep condition and go to the hospital for further examination if necessary.

[0297] In the embodiment of the present application, only the microphone of the mobile smart device is used to collect audio during sleep, and the sleep stage can be evaluated in a lightweight manner through database data matching; multi-level DWT filtering and fusion of multiple fusion methods are used to extract the characteristics of non-stationary speech activities, and the final data matching is obtained through multiple classifier fusion classification methods, so as to achieve the effect of detecting sleep stages at night, which is lighter, more efficient and less costly; at the same time, it can assist in the detection of sleep problems, facilitate users to pay timely attention to the status of sleep health, and discover potential sleep problems early.

[0298] See also Figure 5 , Figure 5This is a functional module diagram of the sleep staging system for multi-level fusion detection of sleep sounds in this application.

[0299] The device 100 comprises:

[0300] The collection module 101 is used to collect audio data of the user during sleep;

[0301] A decomposition module 102, configured to filter the collected audio data and perform multi-level decomposition on the filtered audio data;

[0302] An extraction module 103, used to extract multiple audio features using a multi-level fusion feature classification method;

[0303] A selection module 104, configured to select an optimal audio feature subset based on the extracted multiple audio features using an iterative and hybrid feature selector RFINCA;

[0304] A classification module 105, configured to send the optimal audio feature subset to a fused classifier for classification;

[0305] The judgment module 106 is used to match the classification result with the sleep stage feature database to judge the sleep stage of the user and the corresponding abnormal situation.

[0306] In some embodiments, the selection module 103 is used to select the optimal audio features using the iterative and hybrid feature selector RFINCA, including:

[0307] Using the ReliefF algorithm to filter the multiple audio features to obtain a first feature subset;

[0308] The first feature subset is optimized using neighbor component analysis (NCA) to obtain a second feature subset;

[0309] The above screening and optimization process is iterated until the optimal audio feature subset is obtained.

[0310] In some embodiments, the plurality of audio features are screened using a ReliefF algorithm to obtain a first feature subset, including:

[0311] Randomly select a sample Ri from the multiple audio features, and find k nearest neighbor samples of Ri in the feature space;

[0312] For each feature f, calculate the distance difference between Ri and its nearest neighbor of the same type on f, diff(f,Ri,H), and the distance difference between Ri and its nearest neighbor of a different type on f, diff(f,Ri,M);

[0313] Update the weight of each feature f:

[0314]

[0315] in,

[0316] Ri is a randomly selected training sample, i is the i-th natural number, and j is the j-th natural number;

[0317] k is the number of nearest neighbor samples found in the feature space;

[0318] f represents a specific feature;

[0319] diff(f,Ri,H) is the distance difference between Ri and its nearest neighbor of the same type (same label) on feature f;

[0320] diff(f,Ri,M) is the distance difference between Ri and its heterogeneous (differently labeled) nearest neighbor on feature f;

[0321] W[f] is the weight of feature f;

[0322] m is the total number of samples;

[0323] Hj is the jth nearest neighbor sample of the same type as Ri;

[0324] Mj(C) is the jth nearest neighbor sample that is different from Ri, where C represents a category different from Ri;

[0325] P(C) is the prior probability of category C;

[0326] class(Ri) is the class of Ri;

[0327] Traverse all training samples;

[0328] According to the feature weight W, they are sorted from high to low, and the Ns features with the highest weight are selected to form the first feature subset S.

[0329] In some embodiments, the first feature subset is optimized using neighbor component analysis (NCA) to obtain a second feature subset, including:

[0330] The weighted distance between samples i and j is calculated based on the first feature subset S:

[0331]

[0332] The similarity between samples i and j is calculated based on the weighted distance:

[0333]

[0334] Calculate the expected probability for each sample

[0335] Calculate the objective function f(w) = ∑ i p i ;

[0336] Finding the optimal solution of the feature weight w to obtain the optimized second feature subset;

[0337] in,

[0338] S is the feature subset after being screened by the ReliefF algorithm;

[0339] wf is the weight of feature f in S;

[0340] xif is the value of sample i on feature f;

[0341] xjf is the value of sample j on feature f;

[0342] dijS is the weighted Euclidean distance between samples i and j under feature subset S;

[0343] pij is the probability that sample j is selected as the nearest neighbor by sample i under feature subset S;

[0344] pi is the expected probability that sample i is selected as the nearest neighbor by its similar samples;

[0345] f(w) is the objective function, which represents the sum of the probabilities of all samples being selected as nearest neighbors by their similar samples. The goal is to maximize this probability by adjusting the feature weight w.

[0346] In some embodiments, the decomposition module 102 is used to perform multi-level decomposition on the filtered audio data, including:

[0347] Perform the first-level discrete wavelet transform DWT decomposition on the original sleep audio T1 to obtain the first low-frequency component low1 and the first high-frequency component high1;

[0348] The low-frequency component low1 is subjected to the second-level DWT decomposition to obtain the second low-frequency component low2 and the second high-frequency component high2;

[0349] The second low-frequency component low2 is subjected to the next level of DWT decomposition to complete the iterative operation of the multi-level decomposition.

[0350] In some embodiments, performing a first level discrete wavelet transform DWT decomposition on the original sleep audio T1 to obtain a first low frequency component low1 and a first high frequency component high1 includes:

[0351] Construct a DWT filter and decomposition layer number, and input T1 into the DWT filter;

[0352] Using the DWT filter to perform low-pass and high-pass filtering on T1, respectively obtaining a low-frequency component low1 and a high-frequency component high1;

[0353] Downsample low1 to get the scale coefficient, and downsample high1 to get the wavelet coefficient.

[0354] In some embodiments, the extraction module 103 is used to extract multiple audio features using a multi-level fusion feature classification method, including:

[0355] A low-pass filter is used to extract texture features of multiple low-frequency components;

[0356] Based on the extracted texture features, multiple audio features are extracted using local binary model LBP and local ternary pattern LTP.

[0357] In some embodiments, a low-pass filter is used to extract texture features of multiple low-frequency components, including:

[0358] The low-frequency component low1 obtained by the first-level DWT decomposition is subjected to a low-pass filter to extract the texture feature tpf1;

[0359] The low-frequency components lowi (i=2,3,...,N) obtained by the 2nd to Nth level DWT decomposition are extracted using a low-pass filter to extract the texture features tpfi;

[0360] The local binary model LBP and the local ternary pattern LTP are used to extract multiple audio features, including:

[0361] Extracting local binary pattern feature lbpf1 using LBP operator for the texture feature tpf1 extracted at the first stage;

[0362] Extracting local ternary pattern feature ltpf1 using LTP operator for the texture feature tpf1 extracted at the first stage;

[0363] Extract LBP and LTP features from the texture features tpfi (i=2,3,...,8) extracted from the 2nd to Nth levels to obtain lbpfi and ltpfi;

[0364] The LBP features lbpf1, lbpf2, ..., lbpfN and the LTP features ltpf1, ltpf2, ..., ltpfN extracted at N levels are cascaded to form multi-level LBP feature vectors and multi-level LTP feature vectors, and the multi-level LBP feature vectors and the multi-level LTP feature vectors together constitute the multiple audio features.

[0365] In the embodiment of the present application, only the microphone of the mobile smart device is used to collect audio during sleep, and the sleep stage can be evaluated in a lightweight manner through database data matching; multi-level DWT filtering and fusion of multiple fusion methods are used to extract the characteristics of non-stationary speech activities, and the final data matching is obtained through multiple classifier fusion classification methods, so as to achieve the effect of detecting sleep stages at night, which is lighter, more efficient and less costly; at the same time, it can assist in the detection of sleep problems, facilitate users to pay timely attention to the status of sleep health, and discover potential sleep problems early.

[0366] See also Figure 6 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 6 includes a memory 62 and a processor 63. The electronic device can be, but is not limited to, a smart ring, and can also be a wearable electronic device such as a smart bracelet and smart glasses. The memory 62 is used to store computer-readable instructions, and the processor 63 executes the computer-readable instructions stored in the memory to implement the method for performing sleep staging by detecting sleep sounds through multi-level fusion as described in any of the above embodiments.

[0367] In an optional embodiment, the electronic device 6 further includes a bus, a computer program stored in the memory 62 and executable on the processor 63, such as a distributed storage program.

[0368] Figure 6 Only the electronic device 6 having the memory 62 and the processor 63 is shown, and those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on the electronic device 6 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0369] Combination Figure 2 The memory 62 in the electronic device 2 stores a plurality of computer-readable instructions, and the processor 63 can execute the plurality of instructions to implement a method for performing sleep staging by detecting sleep sounds at a multi-level fusion.

[0370] Specifically, the specific implementation method of the processor 63 for the above instructions can refer to Figure 3 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0371] Those skilled in the art will appreciate that the schematic diagram is merely an example of the electronic device 6 and does not constitute a limitation on the electronic device 6. The electronic device 6 may be a bus structure or a star structure. The electronic device 6 may also include more or less other hardware or software than shown in the diagram, or a different arrangement of components. For example, the electronic device 6 may also include input and output devices, network access devices, etc.

[0372] It should be noted that the electronic device 6 is only an example, and other existing or future electronic products that are suitable for the present application should also be included in the protection scope of the present application and included here by reference.

[0373] The memory 62 includes at least one type of readable storage medium, which may be non-volatile or volatile. The readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. The memory 62 may be an internal storage unit of the electronic device 6 in some embodiments, such as a mobile hard disk of the electronic device 6. The memory 62 may also be an external storage device of the electronic device 6 in other embodiments, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc., equipped on the electronic device 6. The memory 62 may be used not only to store application software and various types of data installed in the electronic device 6, such as the code of a distributed storage program, etc., but also to temporarily store data that has been output or is to be output.

[0374] In some embodiments, the processor 63 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 63 is the control core (Control Unit) of the electronic device 6, and uses various interfaces and lines to connect various components of the entire electronic device 6, and executes various functions and processes data of the electronic device 6 by running or executing programs or modules stored in the memory 62 (for example, executing distributed storage programs, etc.), and calling data stored in the memory 62.

[0375] The processor 63 executes the operating system of the electronic device 6 and various installed applications. The processor 63 executes the applications to implement the steps in the above-mentioned embodiments of the method for multi-level fusion detection of sleep sounds for sleep staging, for example Figure 1 Steps shown.

[0376] Exemplarily, the computer program may be divided into one or more units / modules, which are stored in the memory 62 and executed by the processor 63 to complete the present application. The one or more units / modules may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 6. For example, the computer program may be divided into a collection module 101, a data processing module 102, and a monitoring and analysis module 103.

[0377] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, and includes several instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute the part of the method for multi-level fusion detection of sleep sounds for sleep staging described in various embodiments of the present application.

[0378] If the integrated unit / module of the electronic device 6 is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware devices through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of each of the above-mentioned method embodiments can be implemented.

[0379] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory and other memory, etc.

[0380] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0381] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0382] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. Figure 6 Only one arrow is used in the figure, but it does not mean that there is only one bus or one type of bus. The bus is configured to realize the connection and communication between the memory 62 and at least one processor 63, etc.

[0383] An embodiment of the present application further provides a computer-readable storage medium (not shown), in which computer-readable storage medium is stored computer-readable instructions, which are executed by a processor in an electronic device to implement the method for sleep staging by multi-level fusion detection of sleep sounds as described in any of the above embodiments.

[0384] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0385] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0386] In addition, each functional module in each embodiment of the present application can be integrated into a processing unit, each module can exist physically separately, or two or more modules can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0387] In addition, it is obvious that the word "comprising" does not exclude other modules or steps, and the singular does not exclude the plural. Multiple modules or devices stated in the specification can also be implemented by one module or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0388] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. A method for sleep staging by multi-level fusion detection of sleep sounds, characterized in that: include: Collect audio data during the user's sleep; Filtering the collected audio data, and performing multi-level decomposition on the filtered audio data; Use a multi-level fusion feature classification method to extract multiple audio features; Based on the extracted multiple audio features, using the iterative and hybrid feature selector RFINCA to select the optimal audio feature subset; Sending the optimal audio feature subset to the fused classifier for classification; Based on the classification result, matching is performed with a sleep stage feature database to determine the sleep stage of the user and the corresponding abnormal condition; The filtered audio data is decomposed into multiple levels, including: Original Sleep Audio T 1 Perform the first level discrete wavelet transform DWT decomposition; get the first low frequency component low 1 and the first high frequency component high 1 ; The low frequency component 1 Perform the second-level DWT decomposition to obtain the second low-frequency component low 2 and the second high frequency component high 2 ; The second low frequency component is low 2 Perform the next level of DWT decomposition to complete the iterative operation of multi-level decomposition; Use a multi-level fusion feature classification method to extract multiple audio features, including: A low-pass filter is used to extract texture features of multiple low-frequency components; Based on the extracted texture features, a local binary model LBP and a local ternary pattern LTP are used to extract multiple audio features; Among them, a low-pass filter is used to extract the texture features of multiple low-frequency components, including: The low frequency component obtained by the first-level DWT decomposition is low 1 Use low-pass filter to extract texture features tpf 1 ; The low frequency components obtained by decomposing the 2nd to Nth level DWT are low i , i = 2, 3, ..., N, use low-pass filter to extract texture features tpf i ; The local binary model LBP and the local ternary pattern LTP are used to extract multiple audio features, including: The texture feature tpf extracted at the first level 1 Use LBP operator to extract local binary pattern features lbpf 1 ; The texture feature tpf extracted at the first level 1 Use LTP operator to extract local ternary pattern feature ltpf 1 ; Texture features tpf extracted from the 2nd to Nth levels i ,i=2,3,...,N,extract LBP and LTP features, and get lbpf i and ltpf i ; The LBP features lbpf extracted from N levels 1 ,lbpf 2 ,...,lbpf N and LTP feature ltpf 1 ,ltpf 2 ,...,ltpf N Cascading to form a multi-level LBP feature vector and a multi-level LTP feature vector, the multi-level LBP feature vector and the multi-level LTP feature vector together constitute the multiple audio features.

2. The method according to claim 1, characterized in that The iterative and hybrid feature selector RFINCA is used to select the best audio features, including: Using the ReliefF algorithm to filter the multiple audio features to obtain a first feature subset; The first feature subset is optimized using neighbor component analysis (NCA) to obtain a second feature subset; The above screening and optimization process is iterated until the optimal audio feature subset is obtained.

3. The method according to claim 2, characterized in that The plurality of audio features are screened using the ReliefF algorithm to obtain a first feature subset, including: Randomly select a sample R from the multiple audio features i , find R i k nearest neighbor samples in the feature space; For each feature f, calculate R i The distance difference between its nearest neighbor on f and its similar neighbor diff(f,R i ,H), and the distance difference between it and its heterogeneous nearest neighbor on f (diff(f,R i ,M); Update the weight of each feature f: in, Ri is a randomly selected training sample, i is the i-th natural number, and j is the j-th natural number; k is the number of nearest neighbor samples found in the feature space; f represents a specific feature; diff(f,R i ,H) is R i The distance difference between its nearest neighbor of the same type on feature f, where the same type refers to the same label; diff(f,R i ,M) is R i The distance difference between its nearest neighbor with different labels on feature f; W[f] is the weight of feature f; m is the total number of samples; H j For R i The jth nearest neighbor sample of the same type; M j (C) is with R i The jth nearest neighbor sample of the heterogeneous class, where C represents a i Different categories; P(C) is the prior probability of category C; class(R i ) is R i Category of; Traverse all training samples; Sort the feature weights W from high to low and select the N with the highest weight. s The features form the first feature subset S.

4. The method according to claim 3, characterized in that: The first feature subset is optimized using neighbor component analysis (NCA) to obtain a second feature subset, including: The weighted distance between samples i and j is calculated based on the first feature subset S: The similarity between samples i and j is calculated based on the weighted distance: Calculate the expected probability for each sample Calculate the objective function f(w) = ∑ i p i ; Finding the optimal solution of the feature weight w to obtain the optimized second feature subset; in, S is the feature subset after being screened by the ReliefF algorithm; w f is the weight of feature f in S; x if is the value of sample i on feature f; x jf is the value of sample j on feature f; d ij S is the weighted Euclidean distance between samples i and j under feature subset S; p ij is the probability that sample j is selected as the nearest neighbor by sample i under feature subset S; p i is the expected probability that sample i is selected as the nearest neighbor by its similar samples; f(w) is the objective function, which represents the sum of the probabilities of all samples being selected as nearest neighbors by their similar samples. The goal is to maximize this probability by adjusting the feature weight w.

5. The method according to claim 1, characterized in that Original Sleep Audio T 1 Perform the first level discrete wavelet transform DWT decomposition; Get the first low-frequency component low 1 and the first high frequency component high 1 ,include: Construct DWT filter and decomposition layer, and convert T 1 Input into the DWT filter; Using the DWT filter, T 1 Perform low-pass and high-pass filtering to obtain the low-frequency components low 1 and high frequency components 1 ; To low 1 Downsample to get the scale factor, and 1 Downsampling is performed to obtain wavelet coefficients.

6. A system for multi-level fusion detection of sleep sounds for sleep staging, characterized in that: include: A collection module, used to collect audio data during the user's sleep; A decomposition module, used for filtering the collected audio data and performing multi-level decomposition on the filtered audio data; An extraction module, used for extracting multiple audio features using a multi-level fusion feature classification method; A selection module, configured to select an optimal audio feature subset based on the extracted multiple audio features using an iterative and hybrid feature selector RFINCA; A classification module, used for sending the optimal audio feature subset to the fused classifier for classification; A judgment module, used for matching the classification result with a sleep stage feature database to judge the sleep stage of the user and the corresponding abnormal situation; The filtered audio data is decomposed into multiple levels, including: Original Sleep Audio T 1 Perform the first level discrete wavelet transform DWT decomposition; get the first low frequency component low 1 and the first high frequency component high 1 ; The low frequency component 1 Perform the second-level DWT decomposition to obtain the second low-frequency component low 2 and the second high frequency component high 2 ; The second low frequency component is low 2 Perform the next level of DWT decomposition to complete the iterative operation of multi-level decomposition; Use a multi-level fusion feature classification method to extract multiple audio features, including: A low-pass filter is used to extract texture features of multiple low-frequency components; Based on the extracted texture features, a local binary model LBP and a local ternary pattern LTP are used to extract multiple audio features; Among them, a low-pass filter is used to extract the texture features of multiple low-frequency components, including: The low frequency component obtained by the first-level DWT decomposition is low 1 Use low-pass filter to extract texture features tpf 1 ; The low frequency components obtained by decomposing the 2nd to Nth level DWT are low i , i = 2, 3, ..., N, use low-pass filter to extract texture features tpf i ; The local binary model LBP and the local ternary pattern LTP are used to extract multiple audio features, including: The texture feature tpf extracted at the first level 1 Use LBP operator to extract local binary pattern features lbpf 1 ; The texture feature tpf extracted at the first level 1 Use LTP operator to extract local ternary pattern feature ltpf 1 ; Texture features tpf extracted from the 2nd to Nth levels i ,i=2,3,...,N,extract LBP and LTP features, and get lbpf i and ltpf i ; The LBP features lbpf extracted from N levels 1 ,lbpf 2 ,...,lbpf N and LTP feature ltpf 1 ,ltpf 2 ,...,ltpf N Cascading to form a multi-level LBP feature vector and a multi-level LTP feature vector, the multi-level LBP feature vector and the multi-level LTP feature vector together constitute the multiple audio features.

7. A system for multi-level fusion detection of sleep sounds for sleep staging, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer executable instructions, and the processor implements the method according to any one of claims 1 to 5 when executing the computer executable instructions on the memory.

Images