Multi-sensor fusion sleep monitoring method and system, equipment and storage medium

Abstract

Through the multi-sensor fusion design of the microphone, the millimeter wave radar sensor and the piezoelectric film sensor, compared with an existing single or double-sensor sleep monitoring technology, the monitoring precision, the function dimension, the environmental adaptability and the like are remarkably improved. According to the method, a weighted voting decision fusion mechanism and an improved random forest model are adopted, and after a microphone acoustic monitoring dimension is newly added, the apnea detection sensitivity is improved; acoustic characteristics such as snore, breath sound rhythm, cough and dream can be accurately captured; health risk assessment dimensions are richer, preliminary screening of diseases such as sleep apnea syndrome and periodic limb dyskinesia can be realized, snore indexes and breathing stability are newly increased, and a multi-dimensional sleep quality report is provided for a user; in the whole process, an electrode plate or a wrist strap does not need to be worn, the three sensors all adopt non-intrusive design, no foreign body sensation exists after the piezoelectric film is embedded into the mattress, the problem of wearing discomfort of traditional monitoring is solved, and the user compliance is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sleep monitoring, in particular to a multi-sensor fusion sleep monitoring method and system, device and storage medium. BACKGROUND

[0002] With the increasing attention to health, sleep quality monitoring becomes increasingly important. At present, single sleep monitoring technology has many limitations. For example, although the traditional contact sensor (such as electrode sheet) can accurately obtain physiological signals, it is inconvenient to wear and affects the naturalness of sleep; some non-contact sensors, such as mobile phone sleep monitoring scheme relying only on acceleration sensor, are convenient but have single monitoring index and cannot fully reflect the sleep state. Sleep apnea, abnormal heart rate and other sleep problems often need to be judged comprehensively based on multi-dimensional data. For example, the Chinese patent with application number CN202411008671.2 discloses a sleep monitoring method and system based on millimeter wave radar and pressure sensor, which uses video analysis technology, but has many defects:

[0003] From the perspective of monitoring information dimension, this system only relies on millimeter wave radar and pressure sensor, and cannot fully collect acoustic information during sleep. For example, breathing sound, snoring sound, and other sound information such as coughing sound and talking in sleep during sleep are difficult to accurately capture and analyze. The frequency and rhythm of breathing sound are of great significance to judge the breathing state during sleep and whether there is breathing abnormality, such as rapid breathing and apnea; the intensity, frequency, duration and interval time of snoring sound can effectively reflect whether the sleeper has sleep apnea syndrome and other sleep diseases, and through analysis of these characteristics, the severity of the disease can also be preliminarily evaluated; the number of coughing sound and talking in sleep can also reflect the physical condition or mental state of the sleeper to some extent.

[0004] Therefore, there is an urgent need for a high-precision, non-interfering and comprehensive sleep quality monitoring scheme. SUMMARY

[0005] In order to achieve the above-mentioned purposes and other advantages of the present application, the first object of the present application is to provide a multi-sensor fusion sleep monitoring method, comprising the following steps:

[0006] Pretreating the sound signals collected by the microphone, the millimeter wave radar sensor signals and the piezoelectric film sensor signals;

[0007] Extracting features from the pretreated sound signals, millimeter wave radar sensor signals and piezoelectric film sensor signals;

[0008] Fusing the extracted sound signal features, millimeter wave radar sensor signal features, and piezoelectric film sensor signal features;

[0009] Based on the fused data, using a sleep staging model to stage sleep;

[0010] According to the sound signal features, millimeter wave radar sensor signal features, and piezoelectric film sensor signal features, the respiratory quality is evaluated;

[0011] Through the heart rate data obtained by the millimeter wave radar sensor signal and the piezoelectric film sensor signal, the change trend of heart rate, heart rate variability, the change rule of heart rate in different sleep stages, and whether there is heart rate abnormality are monitored.

[0012] Further, it also includes the steps of:

[0013] Detecting abnormal events through the fused data.

[0014] Further, the pre-processing steps of the sound signal collected by the microphone include:

[0015] An adaptive LMS filtering algorithm is used to remove environmental noise interference;

[0016] An initial noise model is established by collecting pre-sleep environmental noise, and the noise reference signal is updated in real time;

[0017] The sound signal is frame-processed for subsequent feature extraction;

[0018] The pre-processing steps of the millimeter wave radar sensor signal include:

[0019] A Kalman filtering algorithm is used to make optimal estimation using the estimated value at the previous time and the measured value at the current time, and to remove noise caused by environmental interference;

[0020] The millimeter wave radar sensor signal is calibrated to ensure the accuracy of the measurement of parameters such as respiratory rate and heart rate;

[0021] The pre-processing steps of the piezoelectric film sensor signal include:

[0022] A high-pass filtering algorithm is used to remove low-frequency baseline drift signals;

[0023] A low-noise amplifier is used to improve signal strength for subsequent analysis.

[0024] Further, the feature extraction steps from the pre-processed sound signal include:

[0025] The time-frequency characteristics of the sound signal are obtained by a short-time Fourier transform method, including the intensity, frequency, duration and interval time of snoring sound, the frequency and rhythm of breathing sound, the occurrence number of coughing sound and mumbling in sleep;

[0026] The feature extraction step from the pre-processed millimeter wave radar sensor signal includes:

[0027] The accurate respiration and heart rate values are calculated by analyzing the phase and frequency changes of the millimeter wave radar sensor reflected wave, using a phase demodulation algorithm;

[0028] The body movement characteristics are obtained by analyzing the change rate of the reflected wave amplitude and phase;

[0029] The feature extraction step from the pre-processed piezoelectric film sensor signal includes:

[0030] The frequency and amplitude of heart tremor, the pressure change characteristics caused by respiration, and the pressure distribution and change trend of body movement are obtained by analyzing the time and frequency domain of the piezoelectric film sensor output electrical signal.

[0031] Further, the step of fusing the extracted sound signal features, millimeter wave radar sensor signal features and piezoelectric film sensor signal features includes:

[0032] A decision layer fusion method is used to input the extracted sound signal features, millimeter wave radar sensor signal features and piezoelectric film sensor signal features into a sleep staging model for fusion; wherein the sleep staging model classifies and evaluates the sleep state by learning the combination mode of multi-sensor features in different sleep states.

[0033] Further, the sleep staging includes wakefulness, light sleep, deep sleep and rapid eye movement stage;

[0034] The step of evaluating the respiration quality according to the sound signal features, millimeter wave radar sensor signal features and piezoelectric film sensor signal features includes:

[0035] The respiration quality is evaluated according to the respiration sound characteristics monitored by the microphone and the respiration-related physiological parameters monitored by the millimeter wave radar sensor and the piezoelectric film sensor;

[0036] Respiratory pause events, low ventilation events and abnormal respiratory rhythm conditions are detected, and a respiratory pause hypopnea index is calculated to evaluate the severity of sleep disordered breathing.

[0037] Further, the abnormal events include periodic limb movement disorder and night terror.

[0038] The second object of the present application is to provide a multi-sensor fusion sleep monitoring system, which applies the above method, comprising a microphone, a millimeter wave radar sensor, a piezoelectric film sensor, and a data processing module.

[0039] The microphone is used to continuously collect various sound signals during sleep.

[0040] The millimeter wave radar sensor is used to emit high-frequency electromagnetic waves to monitor chest and abdominal micro-movements and body movement signals by receiving reflected waves.

[0041] The piezoelectric film sensor is used to sense subtle pressure changes and vibration signals caused by heartbeats, breathing, and body movements.

[0042] The data processing module is used to preprocess and extract features from the signals collected by the microphone, the millimeter wave radar sensor, and the piezoelectric film sensor, and fuse the extracted features. A decision layer fusion method is used to input the fused data into a sleep staging model to stage sleep. The quality of breathing is evaluated based on sound signal features, millimeter wave radar sensor signal features, and piezoelectric film sensor signal features. The heart rate data obtained from the millimeter wave radar sensor signal and the piezoelectric film sensor signal are used to monitor the trend of heart rate changes and heart rate variability, analyze the variation of heart rate in different sleep stages, and determine whether there is an abnormal heart rate.

[0043] The third object of the present application is to provide a computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above method.

[0044] The fourth object of the present application is to provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the above method.

[0045] Compared with the prior art, the present application has the following advantages:

[0046] The present application uses a multi-sensor fusion design of microphone, millimeter wave radar sensor, and piezoelectric film sensor, which significantly improves the monitoring accuracy, functional dimensions, and environmental adaptability compared with existing single or dual sensor sleep monitoring technology. The specific advantages are as follows:

[0047] The monitoring accuracy and data reliability are significantly improved: the weighted voting decision fusion mechanism and the improved random forest model are used, and the acoustic monitoring dimension of the microphone is added, which improves the sensitivity of apnea detection.

[0048] Function dimension and health assessment ability expansion: can accurately capture snoring sound (intensity, frequency, interval), breathing sound rhythm, cough and dream speech, etc. Acoustic characteristics, filling the gap of existing technology acoustic information monitoring. Health risk assessment dimension is more abundant, which can realize the preliminary screening of sleep apnea syndrome, periodic limb movement disorder and other diseases, add snoring index and breathing stability, and provide multi-dimensional sleep quality report for users.

[0049] User experience and implementation cost optimization: non-contact monitoring improves comfort, no need to wear electrode sheet or wristband throughout the process, three sensors are non-invasive design, piezoelectric film is embedded in the mattress without foreign body sensation, solving the wearing discomfort problem of traditional monitoring, improving user compliance.

[0050] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, and can be implemented according to the content of the specification, the following is the preferred embodiment of the present application and the detailed description of the drawings. The specific embodiment of the present application is given in detail by the following examples and their drawings. BRIEF DESCRIPTION OF DRAWINGS

[0051] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The schematic embodiments of the present application and their description are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0052] Figure 1 It is a multi-sensor fusion sleep monitoring system diagram;

[0053] Figure 2 It is a sensor application schematic diagram;

[0054] Figure 3 It is a multi-sensor fusion sleep monitoring method flow chart;

[0055] Figure 4 It is a sound signal preprocessing flow chart;

[0056] Figure 5 It is a millimeter wave radar sensor signal preprocessing flow chart;

[0057] Figure 6 It is a piezoelectric film sensor signal preprocessing flow chart;

[0058] Figure 7 It is a feature extraction flow chart from preprocessed signal;

[0059] Figure 8 It is a breathing quality assessment flow chart;

[0060] Figure 9 It is a computer device schematic diagram;

[0061] Figure 10 A schematic diagram of a computer-readable storage medium. DETAILED DESCRIPTION

[0062] The application will be further described below in conjunction with the drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments. It should be noted that, under the premise of no conflict, the embodiments described below or the technical features between the embodiments can be combined to form new embodiments.

[0063] All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the application without creative labor fall within the scope of protection of the application.

[0064] The figure numbers in the present application are only used to distinguish the steps in the scheme, and are not used to limit the execution order of the steps. The specific execution order is subject to the description in the specification.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs. The terms used in the specification of the application are only for the purpose of describing the specific embodiments and are not intended to limit the application.

[0066] The application provides a multi-sensor fusion sleep monitoring system and method based on microphone sound collection, millimeter wave radar physiological micro-motion detection, and piezoelectric film sensor limb vibration sensing. The application integrates a microphone for sound collection, which can comprehensively collect these acoustic information, greatly enriching the data dimension of sleep monitoring. In terms of signal analysis capability, multiple signals complement and verify each other, combined with advanced data fusion algorithms and signal processing technology, which can effectively improve the monitoring accuracy and anti-interference ability. The specific scheme is as follows:

[0067] Embodiment 1

[0068] A multi-sensor fusion sleep monitoring system, as shown in Figure 1 , includes a microphone, a millimeter wave radar sensor, a piezoelectric film sensor, and a data processing module. Wherein,

[0069] The microphone is used to continuously collect various sound signals during sleep.

[0070] Preferably, an electret microphone with a frequency response range of 20Hz-20kHz is used. Compared with ordinary consumer-grade microphones, the microphone with this parameter can clearly capture low-frequency breathing sounds of 20-80Hz and high-frequency snoring sound details above 1kHz.

[0071] Optionally, as shown in Figure 2As shown, the microphone is built into the bedside device (such as a smart speaker, a sleep monitoring dedicated device), and the bedside device with the built-in microphone is placed at a distance of 30-50 cm from the bedside (such as the bedside table plane), the microphone pickup hole is directed towards the human head area, and the height difference with the bed surface is 0.5-1 m, which ensures that the microphone can clearly collect sound signals, while avoiding sound attenuation caused by bedding obstruction. The device should be noise reduction processed to reduce the influence of environmental noise on sound collection.

[0072] The millimeter wave radar sensor is used to emit high-frequency electromagnetic waves to monitor the chest and abdominal micro-movement and body movement signals of the human body through the reflected waves;

[0073] Optionally, a 60GHz frequency band FMCW radar is used. The millimeter wave radar sensor is installed on the bedside or the side of the bed at a suitable position, emits 60GHz high-frequency electromagnetic waves, and monitors the chest and abdominal micro-movement, body movement, and other information of the human body through the reflected waves. The millimeter wave radar accurately acquires the respiratory frequency (through the fluctuation of the chest and abdomen), heart rate (micro-vibration of the chest), and body movement amplitude and frequency data.

[0074] For example, the millimeter wave radar sensor is installed at a height of 1-1.5 meters above the bedside, and the emission direction is aligned with the human chest position. The installation angle should be adjusted according to the actual use scene to ensure that the chest and abdominal micro-movement and body movement of the human body can be effectively monitored. The radar needs to be initialized and calibrated, and appropriate monitoring parameters such as monitoring range and sensitivity should be set.

[0075] The piezoelectric film sensor is used to sense the subtle pressure changes and vibration signals caused by heartbeat, breathing, and body movement;

[0076] Optionally, the piezoelectric film sensor is embedded in the mattress or pillow. When the human body contacts the mattress or pillow, the piezoelectric film sensor can sense the subtle pressure changes and vibration signals caused by heartbeat, breathing, and body movement. The sensor has high sensitivity and can detect extremely weak vibrations.

[0077] For example, the piezoelectric film sensor is uniformly embedded in the mattress surface layer below 10-15 cm, ensuring that the sensor can accurately sense the pressure changes and vibrations of the human body. The sensor connection line should be shielded to reduce electromagnetic interference.

[0078] The data processing module is used for pre-processing and feature extraction of signals collected by the microphone, the millimeter wave radar sensor and the piezoelectric film sensor, fusing the extracted features, inputting the fused data into a sleep staging model by using a decision layer fusion method to stage sleep, evaluating breathing quality according to sound signal features, millimeter wave radar sensor signal features and piezoelectric film sensor signal features, monitoring the change trend of heart rate and heart rate variability through heart rate data obtained by the millimeter wave radar sensor signal and the piezoelectric film sensor signal, analyzing the change rule of heart rate in different sleep stages, and whether there is heart rate abnormality.

[0079] Specifically, during the sleep of a user, the microphone continuously collects sound signals at a certain sampling frequency (such as 44.1 kHz) and transmits the signals to the data processing module. The data processing module first performs real-time noise reduction processing on the sound signals, and then frames the signals according to a certain time interval (such as 1 second) to extract sound features of each frame.

[0080] The millimeter wave radar sensor transmits and receives millimeter wave signals at a set frequency (such as 10 times per second) to monitor the physiological micro-movement and body movement of the human body in real time. The data collected by the millimeter wave radar sensor is transmitted to the data processing module by wireless or wired means, and pre-processing such as denoising and calibration is performed in the data processing module, and then features such as breathing rate, heart rate and body movement are extracted.

[0081] The piezoelectric film sensor transmits the generated electrical signals to the data processing module after sensing the pressure change and vibration of the human body. The data processing module performs pre-processing such as baseline drift removal and amplification on the signals, and then extracts features related to heartbeat tremor, breathing pressure change and body movement.

[0082] The data processing module fuses the features extracted by the three sensors, inputs the features into a trained machine learning model by using a decision layer fusion method, the model classifies and evaluates the sleep state, outputs the sleep stage, evaluates the breathing quality, performs heart rate monitoring and analysis, and detects abnormal events.

[0083] Detailed descriptions of the monitoring method corresponding to the multi-sensor fusion sleep monitoring system can be referred to the corresponding descriptions in the following method embodiments, which will not be repeated here.

[0084] Before actual application, 30 volunteers with different ages, genders and sleep conditions are selected for system testing. The multi-sensor fusion sleep monitoring system of the present application is synchronously monitored with the clinical gold standard polysomnography (PSG), and the monitoring results of the two are compared.

[0085] The Kappa coefficient of sleep staging, the mean absolute error (MAE), the root mean square error (RMSE) of heart rate and respiratory rate, and the sensitivity and specificity of abnormal event detection are calculated to evaluate the accuracy of the system. According to the test results, the parameter settings, data processing algorithms and machine learning models of the system are optimized and adjusted to improve the monitoring performance of the system.

[0086] During long-term use, user feedback is continuously collected to further optimize and improve the system, ensuring its stability and reliability. For example, based on user feedback on environmental interference, the anti-interference algorithm of the sensor is optimized; based on the differences in sleep habits of different users, the training data of the machine learning model is adjusted to improve the adaptability of the model.

[0087] The present application provides a multi-sensor fusion sleep monitoring scheme, which overcomes the shortcomings of single sensor through the cooperative work of microphone, millimeter wave radar sensor and piezoelectric film sensor, realizes comprehensive and accurate monitoring of sleep state, including accurate sleep staging, respiratory quality evaluation, heart rate monitoring and abnormal event detection, etc., provides detailed and reliable sleep health report for users, and provides strong support for early screening and diagnosis of sleep disorders.

[0088] Embodiment 2

[0089] A multi-sensor fusion sleep monitoring method based on the multi-sensor fusion sleep monitoring system provided in Embodiment 1, for detailed description of the multi-sensor fusion sleep monitoring system, please refer to the corresponding description in the above system embodiment, here will not be repeated. As shown in Figure 3 The method comprises the following steps:

[0090] S100, pre-processing the sound signals collected by the microphone, the millimeter wave radar sensor signals and the piezoelectric film sensor signals;

[0091] Further, as shown in Figure 4 The pre-processing step of the sound signals collected by the microphone comprises:

[0092] The sound signals collected by the microphone are denoised, S110, an adaptive LMS filtering algorithm is used to remove environmental noise interference;

[0093] S120, an initial noise model is established by collecting pre-sleep environmental noise, and the noise reference signal is updated in real time;

[0094] For example, an initial noise model is established by collecting 3 minutes of pre-sleep environmental noise, and the noise reference signal is updated in real time. For example, by analyzing the characteristics of environmental noise, the filter parameters are adjusted in real time to retain the sound signals related to sleep.

[0095] S130, frame the sound signal for subsequent feature extraction;

[0096] Further, as shown in Figure 5 The preprocessing step of the millimeter wave radar sensor signal includes:

[0097] S140, using Kalman filtering algorithm, using the estimated value of the previous time and the measurement value of the current time for optimal estimation, removing the noise caused by environmental interference (such as metal object reflection);

[0098] S150, calibrate the millimeter wave radar sensor signal to ensure the accuracy of the measurement of parameters such as respiratory rate and heart rate;

[0099] Further, as shown in Figure 6 The preprocessing step of the piezoelectric film sensor signal includes:

[0100] Remove the baseline drift, S160, use high-pass filtering algorithm to remove the low-frequency baseline drift signal;

[0101] Because the signal output by the piezoelectric film sensor is weak, it needs to be amplified, S170, use low-noise amplifier to improve the signal strength for subsequent analysis.

[0102] S200, extract features from the preprocessed sound signal, millimeter wave radar sensor signal and piezoelectric film sensor signal;

[0103] Further, as shown in Figure 7 The feature extraction step from the preprocessed sound signal includes:

[0104] Sound signal feature extraction: extract multiple features from the preprocessed sound signal, such as snoring intensity, frequency, duration and interval time, respiratory frequency, rhythm, cough and dream speech occurrence frequency, etc. S210, obtain the time-frequency characteristics of the sound signal by short-time Fourier transform method, for subsequent sleep event recognition and sleep quality evaluation.

[0105] Further, the feature extraction step from the preprocessed millimeter wave radar sensor signal includes:

[0106] Millimeter wave radar sensor signal feature extraction: extract features such as respiratory rate, heart rate, body movement amplitude, body movement frequency, etc. For respiratory and heart rate signals, S220, analyze the phase and frequency changes of the millimeter wave radar sensor reflected wave, and use phase demodulation algorithm to calculate the accurate respiratory and heart rate values;

[0107] S230, obtain body movement features by analyzing the change rate of reflected wave amplitude and phase;

[0108] Further, the feature extraction step from the pre-processed piezoelectric film sensor signal includes:

[0109] Piezoelectric film sensor signal feature extraction: Extract the frequency and amplitude of heartbeat tremor, pressure change characteristics caused by respiration, and pressure distribution and change trend of body movement, etc. S240, by analyzing the time domain and frequency domain of the electric signal output by the piezoelectric film sensor, these key features are obtained.

[0110] S300, fuse the extracted sound signal features, millimeter wave radar sensor signal features, and piezoelectric film sensor signal features;

[0111] Further, the step of fusing the extracted sound signal features, millimeter wave radar sensor signal features, and piezoelectric film sensor signal features includes:

[0112] Using a decision layer fusion method, the extracted sound signal features, millimeter wave radar sensor signal features, and piezoelectric film sensor signal features are input into a sleep staging model for fusion; wherein the sleep staging model is a machine learning model (such as random forest, support vector machine, etc.), which learns the combination patterns of multi-sensor features in different sleep states to classify and evaluate sleep states. For example, when the millimeter wave radar sensor detects apnea, the microphone captures the interruption of snoring, and the piezoelectric film sensor monitors the stop of chest and abdominal movement, the model can more accurately judge it as a sleep apnea event.

[0113] S400, based on the fused data, the sleep is divided into four stages of wakefulness, light sleep, deep sleep, and rapid eye movement (REM) sleep using the sleep staging model trained by the machine learning model. The model learns the differences in multi-sensor features in different stages, such as stable respiration and heart rate, less body movement in deep sleep, slight fluctuations in heart rate and respiration, and weak body movement related to eye movement in REM, to accurately determine the sleep stage.

[0114] S500, evaluate the quality of respiration according to the sound signal features, millimeter wave radar sensor signal features, and piezoelectric film sensor signal features;

[0115] Further, as shown in Figure 8 the step of evaluating the quality of respiration according to the sound signal features, millimeter wave radar sensor signal features, and piezoelectric film sensor signal features includes:

[0116] S510, evaluate the quality of respiration according to the respiration sound features monitored by the microphone and the respiration-related physiological parameters monitored by the millimeter wave radar sensor and the piezoelectric film sensor;

[0117] S520, detecting apnea events (respiratory stop time exceeding 10 seconds), hypopnea events (respiratory airflow significantly weakened), and respiratory rhythm abnormalities, etc., calculating the apnea hypopnea index (AHI) for evaluating the severity of sleep-disordered breathing.

[0118] S600, through the heart rate data obtained by the millimeter wave radar sensor signal and the piezoelectric film sensor signal, the change trend of heart rate, heart rate variability, etc. are monitored, the change rule of heart rate in different sleep stages is analyzed, and whether there is heart rate abnormality (such as tachycardia, bradycardia) and the like are analyzed, which provides a basis for cardiovascular health evaluation.

[0119] Further, the method further comprises the steps of:

[0120] In addition to apnea and heart rate abnormality detection, other abnormal events can also be detected through the fused data, such as periodic limb movement disorder (regular limb tremor detected by the piezoelectric film sensor, combined with the body movement data of the millimeter wave radar), night terror (sudden screams captured by the microphone, and violent reactions of the body detected by the millimeter wave radar and the piezoelectric film sensor) and the like.

[0121] The multi-sensor fusion sleep monitoring system and method provided by the application has the following beneficial effects:

[0122] Signal dimension complementarity: the microphone, the millimeter wave radar sensor and the piezoelectric film sensor respectively collect data from three dimensions of sound events, physiological micro-movement and limb vibration, form a multi-modal data closed loop, and comprehensively cover the core indicators of sleep monitoring, such as sleep staging, respiratory quality, heart rate stability and abnormal event detection, which is more comprehensive and accurate than single sensor monitoring.

[0123] Strong anti-interference capability: through multi-signal cross verification, environmental noise interference and individual sleep habit differences can be effectively resisted. For example, the sound signal collected by the microphone can be verified with the physiological signals of the millimeter wave radar and the piezoelectric film sensor, avoiding misjudgment caused by external noise. At the same time, different sensors can provide reliable data for users with different sleep habits, and the adaptation range is wider.

[0124] High precision of sleep staging and abnormal event recognition: multi-sensor fusion data provides more abundant information for sleep staging and abnormal event recognition, significantly improving the recognition accuracy. In terms of sleep staging, it can more accurately distinguish between deep sleep, light sleep and REM sleep stages; in terms of abnormal event detection, the recognition accuracy of sleep apnea, periodic limb movement disorder and other diseases is close to the level of professional equipment, which helps to early detect sleep disorder problems.

[0125] Non-contact and good comfort: all three sensors support non-wearable and non-contact monitoring, without direct contact with the human body, greatly improving the user's use comfort and compliance, especially suitable for people sensitive to sleep environment, and can monitor the sleep state for a long time.

[0126] Data redundancy and fault tolerance: multi-sensor fusion has a data redundancy design, when a sensor fails or is disturbed, the data of other sensors can still maintain the basic sleep monitoring function, ensuring the continuity and reliability of monitoring, and avoiding monitoring interruption due to a single sensor problem.

[0127] Embodiment 3

[0128] A computer device 700, as shown in Figure 9 includes a memory 710, a processor 720, and a computer program 730 stored on the memory and executable on the processor, and the processor implements the steps of a multi-sensor fusion sleep monitoring method when executing the computer program. For detailed description of the method, please refer to the corresponding description in the above method embodiments, which will not be repeated here.

[0129] Embodiment 4

[0130] A computer-readable storage medium, as shown in Figure 10 has a computer program stored thereon, and the computer program implements the steps of a multi-sensor fusion sleep monitoring method when executed by a processor. For detailed description of the method, please refer to the corresponding description in the above method embodiments, which will not be repeated here.

[0131] The number of devices and the scale of processing described herein are used to simplify the description of the present application. Applications, modifications and variations of the present application that are obvious to those skilled in the art are obvious.

[0132] Although the embodiments of the present application have been disclosed as above, they are not limited to the applications and embodiments listed in the specification, and can be fully applied to various fields suitable for the present application, and additional modifications can be easily realized by those skilled in the art, therefore, the present application is not limited to specific details and the figures shown and described herein, without departing from the general concept defined by the claims and the equivalent scope.

[0133] The device, computer device, non-volatile computer storage medium and method provided by the embodiments of the present application are corresponding, therefore, the device, computer device, non-volatile computer storage medium also has similar beneficial technical effects as the corresponding method, since the beneficial technical effects of the method have been described in detail above, therefore, the beneficial technical effects of the corresponding device, computer device, non-volatile computer storage medium will not be repeated here.

[0134] Those skilled in the art will also appreciate that, in addition to being implemented in purely computer readable program code means, the controller can be implemented using logical programming, to cause the controller to perform the same functions, in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like. Such a controller can therefore be considered as a hardware component, and the means for performing the various functions comprised therein can be considered as structures within the hardware component. Alternatively, or even additionally, the means for performing the various functions can be considered as both software elements implementing the method and structures within the hardware component.

[0135] The system, apparatus or unit illustrated in the above embodiments can be implemented by a computer chip or entity, or by a product having certain functions. For the convenience of description, the above apparatus is described in various units according to functions. Of course, the functions of the units can be implemented in one or more software and / or hardware when implementing one or more embodiments of the present specification.

[0136] Those skilled in the art will appreciate that the embodiments of the present specification can be provided as a method, a system, or a computer program product. Therefore, the embodiments of the present specification can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) having computer-usable program code embodied therein.

[0137] The present specification is described with reference to flowcharts and / or block diagrams of methods, apparatus (system) and computer program products according to embodiments of the present specification. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The means for performing the functions specified in one or more flows and / or blocks.

[0138] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The means for performing the functions specified in one or more flows and / or blocks.

[0139] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable devices provide a process for implementing the functions specified in the flowcharts Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0140] It should also be noted that the terms "comprising", "comprises", "including", "includes" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article or apparatus. An element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article or apparatus that comprises the recited element.

[0141] The specification can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0142] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between each of the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0143] The above only describes the embodiments of the specification and does not limit one or more embodiments of the specification. One or more embodiments of the specification can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of the specification shall be included in the scope of claims of one or more embodiments of the specification.

Claims

1. A multi-sensor fusion sleep monitoring method, characterized in that, Includes the following steps: Preprocessing is performed on the sound signals collected by the microphone, the millimeter-wave radar sensor signals, and the piezoelectric film sensor signals. Features are extracted from preprocessed audio signals, millimeter-wave radar sensor signals, and piezoelectric thin film sensor signals. The extracted sound signal features, millimeter-wave radar sensor signal features, and piezoelectric thin film sensor signal features are fused together; Based on the fused data, sleep stages are determined using a sleep staging model. The respiratory quality is assessed based on the characteristics of sound signals, millimeter-wave radar sensor signals, and piezoelectric thin film sensor signals. Heart rate data acquired through millimeter-wave radar sensor signals and piezoelectric thin-film sensor signals are used to monitor heart rate trends and variability, analyze the patterns of heart rate changes in different sleep stages, and identify any heart rate abnormalities.

2. The multi-sensor fusion sleep monitoring method as described in claim 1, characterized in that, It also includes the following steps: Anomalies are detected by integrating the data.

3. The multi-sensor fusion sleep monitoring method as described in claim 1, characterized in that, The preprocessing steps for the audio signals captured by the microphone include: An adaptive LMS filtering algorithm is used to remove environmental noise interference; An initial noise model is established by collecting ambient noise before sleep, and the noise reference signal is updated in real time. The audio signal is segmented into frames for subsequent feature extraction. The preprocessing steps for millimeter-wave radar sensor signals include: The Kalman filter algorithm is used to make the optimal estimate by using the estimated value at the previous time step and the measured value at the current time step, and to remove noise caused by environmental interference. The millimeter-wave radar sensor signal is calibrated to ensure the accuracy of measurements of parameters such as respiratory rate and heart rate; The preprocessing steps for piezoelectric thin film sensor signals include: A high-pass filtering algorithm is used to remove low-frequency baseline drift signals; A low-noise amplifier is used to improve signal strength, which facilitates subsequent analysis.

4. The multi-sensor fusion sleep monitoring method as described in claim 3, characterized in that, The steps for extracting features from the preprocessed audio signal include: The time-frequency characteristics of the sound signal are obtained by using the short-time Fourier transform method, including the intensity, frequency, duration and interval of snoring, the frequency and rhythm of breathing sounds, and the frequency of coughing and sleep talking. The steps for extracting features from preprocessed millimeter-wave radar sensor signals include: By analyzing the phase and frequency changes of the reflected waves from the millimeter-wave radar sensor, an accurate respiratory and heart rate value is calculated using a phase demodulation algorithm. Body motion characteristics are obtained by analyzing the rate of change of the amplitude and phase of the reflected wave; The steps for extracting features from the preprocessed piezoelectric thin film sensor signal include: By performing time-domain and frequency-domain analysis on the electrical signals output by the piezoelectric thin-film sensor, the frequency and amplitude of heartbeat tremors, the characteristics of pressure changes caused by respiration, and the pressure distribution and trends of body movement can be obtained.

5. The multi-sensor fusion sleep monitoring method as described in claim 4, characterized in that, The step of fusing the extracted sound signal features, millimeter-wave radar sensor signal features, and piezoelectric thin film sensor signal features includes: A decision-level fusion method is adopted to input the extracted sound signal features, millimeter-wave radar sensor signal features, and piezoelectric thin film sensor signal features into a sleep staging model for fusion; wherein, the sleep staging model classifies and evaluates sleep states by learning the combination patterns of multi-sensor features under different sleep states.

6. The multi-sensor fusion sleep monitoring method as described in claim 4, characterized in that, The sleep stages include wakefulness, light sleep, deep sleep, and REM sleep; The steps for assessing respiratory quality based on sound signal characteristics, millimeter-wave radar sensor signal characteristics, and piezoelectric thin-film sensor signal characteristics include: The respiratory quality is assessed based on the respiratory sound characteristics monitored by the microphone and the respiratory-related physiological parameters monitored by the millimeter-wave radar sensor and the piezoelectric film sensor. It detects apnea events, hypopnea events, and abnormal breathing rhythms, and calculates the apnea-hypopnea index to assess the severity of sleep-disordered breathing.

7. The multi-sensor fusion sleep monitoring method as described in claim 2, characterized in that: The abnormal events include periodic limb movement disorders and night terrors.

8. A multi-sensor fusion sleep monitoring system, using the method described in any one of claims 1 to 7, characterized in that: Includes a microphone, millimeter-wave radar sensor, piezoelectric thin-film sensor, and data processing module; among which, The microphone is used to continuously collect various sound signals during sleep; The millimeter-wave radar sensor is used to emit high-frequency electromagnetic waves and monitor the micro-movements and body movements of the human body by receiving reflected waves. The piezoelectric thin film sensor is used to sense subtle pressure changes and vibration signals caused by heartbeat, breathing and body movement; The data processing module is used to preprocess and extract features from the signals collected by the microphone, the millimeter-wave radar sensor, and the piezoelectric thin-film sensor, and fuse the extracted features. Using a decision-level fusion method, the fused data is input into a sleep staging model to segment sleep. Breathing quality is assessed based on the characteristics of the sound signal, the millimeter-wave radar sensor signal, and the piezoelectric thin-film sensor signal. Heart rate data obtained from the millimeter-wave radar sensor signal and the piezoelectric thin-film sensor signal are used to monitor the trend and variability of heart rate changes, analyze the variation patterns of heart rate in different sleep stages, and determine whether there are any heart rate abnormalities.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images