Sleep monitoring method, system and storage medium based on millimeter wave radar

Abstract

The present application discloses a sleep monitoring method, system and storage medium based on millimeter wave radar, which relates to the field of millimeter wave monitoring technology. It uses artificial intelligence-based signal processing and analysis technology to extract time-frequency features and semantically combine millimeter wave detection signals and millimeter wave detection response signals, so as to automatically determine whether central sleep apnea occurs based on the fusion representation features of the millimeter wave detection signal and the millimeter wave detection response signal in multimodality. In the present application, the millimeter wave radar can monitor without contacting the human body, avoiding the inconvenience and interference of wearing sensors in traditional methods, and improving the user's comfort and compliance. At the same time, the millimeter wave radar has high temporal and spatial resolution, can accurately detect tiny body movements and breathing changes, and uses artificial intelligence technology for comprehensive and detailed analysis, which helps to more accurately identify and analyze sleep apnea problems.

Description

Technical Field

[0001] The present application relates to the field of millimeter wave monitoring technology, and more specifically, to a sleep monitoring method, system and storage medium based on millimeter wave radar. Background Art

[0002] Sleep is one of the most basic life activities of human beings, and good sleep is essential to maintaining life. By monitoring and analyzing sleep data, sleep disorders can be discovered and intervened in time, sleep quality can be improved, and related diseases can be prevented.

[0003] However, traditional sleep monitoring methods such as polysomnography (PSG) and portable sleep monitoring devices usually require wearing or placing sensors on the body, which may interfere with sleep and affect the naturalness of the monitoring results. In addition, traditional sleep monitoring devices such as polysomnography (PSG) rely on multiple sensors to monitor brain waves, eye movements, electromyography, breathing, heart rate, etc. These sensors may be interfered with or limited, resulting in inaccurate monitoring results.

[0004] Therefore, an optimized sleep monitoring method is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide a sleep monitoring method, system and storage medium based on millimeter wave radar, which uses artificial intelligence-based signal processing and analysis technology to extract time-frequency features and semantic union of millimeter wave detection signals and millimeter wave detection response signals, so as to automatically judge whether central sleep apnea occurs according to the fusion representation features of the millimeter wave detection signal and the millimeter wave detection response signal in multimodality. In this way, the millimeter wave radar can monitor without contacting the human body, avoiding the inconvenience and interference of wearing sensors in traditional methods, thereby improving the user's comfort and compliance. At the same time, the millimeter wave radar has high temporal and spatial resolution, can accurately detect tiny body movements and breathing changes, and use artificial intelligence technology for comprehensive and detailed analysis, which helps to more accurately identify and analyze sleep apnea problems.

[0006] According to one aspect of the present application, a sleep monitoring method based on millimeter wave radar is provided, which includes:

[0007] Transmitting a millimeter wave detection signal through a millimeter wave radar;

[0008] The millimeter wave radar receives the reflected millimeter wave detection signal to obtain a millimeter wave detection response signal;

[0009] Performing signal time-frequency extraction on the millimeter wave detection signal and the millimeter wave detection response signal to obtain a millimeter wave detection signal time-frequency image and a millimeter wave detection response signal time-frequency image;

[0010] Performing signal time-frequency feature analysis on the millimeter wave detection signal time-frequency image and the millimeter wave detection response signal time-frequency image to obtain a millimeter wave detection signal time-frequency semantic feature vector and a millimeter wave detection response signal time-frequency semantic feature vector;

[0011] Inputting the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection response signal time-frequency semantic feature vector into a feature interaction integration module to obtain a millimeter wave detection signal-detection response signal joint semantic feature vector;

[0012] Calculating the position difference between the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection response signal time-frequency semantic feature vector to obtain a millimeter wave detection signal-detection response signal differential semantic feature vector;

[0013] Cascading the millimeter wave detection signal-detection response signal joint semantic feature vector, the millimeter wave detection signal-detection response signal differential semantic feature vector, the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection response signal time-frequency semantic feature vector to obtain a millimeter wave detection multimodal fusion representation vector as a millimeter wave detection multimodal fusion representation feature;

[0014] Based on the millimeter wave detection multimodal fusion representation characteristics, a monitoring result is obtained, and the monitoring result is used to indicate whether central sleep apnea occurs.

[0015] According to another aspect of the present application, a millimeter wave radar-based sleep monitoring system is provided, which includes:

[0016] A millimeter wave detection signal transmitting module, used for transmitting a millimeter wave detection signal through a millimeter wave radar;

[0017] A millimeter wave detection response signal receiving module, used for receiving the millimeter wave detection signal reflected back by the millimeter wave radar to obtain a millimeter wave detection response signal;

[0018] A signal time-frequency extraction module, used to perform signal time-frequency extraction on the millimeter wave detection signal and the millimeter wave detection response signal to obtain a millimeter wave detection signal time-frequency image and a millimeter wave detection response signal time-frequency image;

[0019] A signal time-frequency feature analysis module, used to perform signal time-frequency feature analysis on the millimeter wave detection signal time-frequency image and the millimeter wave detection response signal time-frequency image to obtain a millimeter wave detection signal time-frequency semantic feature vector and a millimeter wave detection response signal time-frequency semantic feature vector;

[0020] A vector dynamic interaction integration module, used for inputting the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection response signal time-frequency semantic feature vector into a feature interaction integration module to obtain a millimeter wave detection signal-detection response signal joint semantic feature vector;

[0021] A difference module, used for calculating the position difference between the time-frequency semantic feature vector of the millimeter wave detection signal and the time-frequency semantic feature vector of the millimeter wave detection response signal to obtain a millimeter wave detection signal-detection response signal differential semantic feature vector;

[0022] A millimeter wave detection multimodal fusion module, used for cascading the millimeter wave detection signal-detection response signal joint semantic feature vector, the millimeter wave detection signal-detection response signal differential semantic feature vector, the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection response signal time-frequency semantic feature vector to obtain a millimeter wave detection multimodal fusion representation vector as a millimeter wave detection multimodal fusion representation feature;

[0023] The monitoring result generating module is used to obtain the monitoring result based on the multi-modal fusion representation feature of the millimeter wave detection, and the monitoring result is used to indicate whether central sleep apnea occurs.

[0024] According to another aspect of the present application, a storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the sleep monitoring method based on millimeter wave radar as described above.

[0025] The present application has at least the following technical effects: Compared with the prior art, the present application provides a sleep monitoring method, system and storage medium based on millimeter wave radar, which uses artificial intelligence-based signal processing and analysis technology to extract time-frequency features and semantic union of millimeter wave detection signals and millimeter wave detection response signals, so as to automatically determine whether central sleep apnea occurs based on the fusion representation features of the millimeter wave detection signal and the millimeter wave detection response signal in multimodality. In this way, the millimeter wave radar can monitor without contacting the human body, avoiding the inconvenience and interference of wearing sensors in traditional methods, thereby improving the user's comfort and compliance. At the same time, the millimeter wave radar has high temporal and spatial resolution, can accurately detect tiny body movements and breathing changes, and use artificial intelligence technology for comprehensive and detailed analysis, which helps to more accurately identify and analyze sleep apnea problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0027] Figure 1 is a flowchart of a sleep monitoring method based on millimeter wave radar according to an embodiment of the present application;

[0028] Figure 2 4 is a block diagram of a sleep monitoring system based on millimeter wave radar according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0030] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0031] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0032] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0033] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0034] Traditional sleep monitoring methods such as polysomnography (PSG) and portable sleep monitoring devices usually require wearing or placing sensors on the body, which may disturb sleep and affect the naturalness of the monitoring results. In addition, traditional sleep monitoring devices such as polysomnography (PSG) rely on multiple sensors to monitor brain waves, eye movements, electromyography, breathing, heart rate, etc. These sensors may be interfered with or limited, resulting in inaccurate monitoring results. Therefore, an optimized sleep monitoring method is desired.

[0035] In the technical solution of the present application, a sleep monitoring method based on millimeter wave radar is proposed. Figure 1 FIG. 1 is a flow chart of a sleep monitoring method based on millimeter wave radar according to an embodiment of the present application. Figure 1 As shown, according to the embodiment of the present application, the sleep monitoring method based on millimeter wave radar includes the following steps: S1, transmitting a millimeter wave detection signal through a millimeter wave radar; S2, receiving the reflected millimeter wave detection signal through the millimeter wave radar to obtain a millimeter wave detection response signal; S3, performing signal time-frequency extraction on the millimeter wave detection signal and the millimeter wave detection response signal to obtain a millimeter wave detection signal time-frequency image and a millimeter wave detection response signal time-frequency image; S4, performing signal time-frequency feature analysis on the millimeter wave detection signal time-frequency image and the millimeter wave detection response signal time-frequency image to obtain a millimeter wave detection signal time-frequency semantic feature vector and a millimeter wave detection reaction signal time-frequency semantic feature vector; S5, inputting the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection reaction signal time-frequency semantic feature vector into feature interactive integration Module to obtain a millimeter wave detection signal-detection response signal joint semantic feature vector; S6, calculate the position difference between the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection response signal time-frequency semantic feature vector to obtain a millimeter wave detection signal-detection response signal differential semantic feature vector; S7, cascade the millimeter wave detection signal-detection response signal joint semantic feature vector, the millimeter wave detection signal-detection response signal differential semantic feature vector, the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection response signal time-frequency semantic feature vector to obtain a millimeter wave detection multimodal fusion representation vector as a millimeter wave detection multimodal fusion representation feature; S8, based on the millimeter wave detection multimodal fusion representation feature, obtain a monitoring result, and the monitoring result is used to indicate whether central sleep apnea occurs.

[0036] In particular, the S1 and S2 transmit millimeter wave detection signals through millimeter wave radars; and receive the reflected millimeter wave detection signals through the millimeter wave radars to obtain millimeter wave detection response signals. It should be understood that millimeter wave radar is a radar system that uses millimeter wave bands (usually in the 30-300 GHz frequency band, with a wavelength of 1-10 mm) for detection and ranging. It has the characteristics of small size, light weight, high spatial resolution, and strong penetration ability. In sleep monitoring, millimeter wave radars can monitor without contacting the human body. This non-invasive monitoring method is suitable for long-term continuous monitoring and will not interfere with the user's normal sleep. In addition, millimeter wave radars can detect tiny movements of the human body, such as breathing and heartbeats. By analyzing these tiny movement changes, millimeter wave radars can accurately monitor breathing frequency and heart rate, thereby evaluating sleep status.

[0037] In particular, S3 performs signal time-frequency extraction on the millimeter-wave detection signal and the millimeter-wave detection response signal to obtain a millimeter-wave detection signal time-frequency image and a millimeter-wave detection response signal time-frequency image. In particular, in a specific example of the present application, the millimeter-wave detection signal and the millimeter-wave detection response signal are subjected to wavelet transform to obtain the millimeter-wave detection signal time-frequency image and the millimeter-wave detection response signal time-frequency image. Considering that different local time-frequency in the millimeter-wave detection signal and the millimeter-wave detection response signal have different characteristic information, the wavelet transform can jointly analyze the signal in the time domain and the frequency domain, and can effectively extract the local features in the signal, so as to better describe the time-frequency characteristics of the signal. Based on this, in the technical solution of the present application, the millimeter-wave detection signal and the millimeter-wave detection response signal are subjected to wavelet transform to respectively extract the millimeter-wave detection signal time-frequency image and the millimeter-wave detection response signal time-frequency image containing local characteristic time-frequency information.

[0038] Specifically, wavelet transform is a signal processing technique used to analyze the frequency and time domain characteristics of a signal. Wavelet transform uses a set of functions called wavelet bases to analyze signals. These wavelet bases are localized and have finite extensions in the time and frequency domains, so they can better capture the instantaneous characteristics of the signal.

[0039] In particular, the S4 performs signal time-frequency feature analysis on the millimeter-wave detection signal time-frequency image and the millimeter-wave detection reaction signal time-frequency image to obtain a millimeter-wave detection signal time-frequency semantic feature vector and a millimeter-wave detection reaction signal time-frequency semantic feature vector. In particular, in a specific example of the present application, the millimeter-wave detection signal time-frequency image and the millimeter-wave detection reaction signal time-frequency image are input into a signal time-frequency feature extractor based on a hole convolutional neural network model to obtain the millimeter-wave detection signal time-frequency semantic feature vector and the millimeter-wave detection reaction signal time-frequency semantic feature vector. Considering that the millimeter-wave detection signal time-frequency image and the millimeter-wave detection reaction signal time-frequency image express time-frequency feature information of different local time scales, the hole convolutional neural network can increase the receptive field size of the convolution layer by introducing hole parameters, thereby realizing multi-scale feature learning and mining. Therefore, in order to better capture and mine the local temporal frequency information of different scales in the millimeter wave detection signal time-frequency image and the millimeter wave detection reaction signal time-frequency image, and provide richer input for subsequent sleep breathing detection, in the technical solution of the present application, the millimeter wave detection signal time-frequency image and the millimeter wave detection reaction signal time-frequency image are input into a signal time-frequency feature extractor based on a void convolutional neural network model to obtain a millimeter wave detection signal time-frequency semantic feature vector and a millimeter wave detection reaction signal time-frequency semantic feature vector.

[0040] Specifically, the dilated convolutional neural network is a convolutional neural network structure that increases the receptive field by inserting holes (or called dilation rates) between convolution kernels while maintaining the computational efficiency of the network. Dilated convolution can expand the field of view of the convolution kernel without increasing parameters and computational complexity, thereby better capturing long-distance contextual information.

[0041] In particular, in S5, the time-frequency semantic feature vector of the millimeter-wave detection signal and the time-frequency semantic feature vector of the millimeter-wave detection reaction signal are input into the feature interaction integration module to obtain the millimeter-wave detection signal-detection reaction signal joint semantic feature vector. Considering that the time-frequency semantic feature vector of the millimeter-wave detection signal and the time-frequency semantic feature vector of the millimeter-wave detection reaction signal respectively reflect the frequency feature information of the millimeter-wave detection signal and the millimeter-wave detection reaction signal on the local time scale. Therefore, in order to better integrate the time-frequency feature information between the two, so as to improve the representation ability of the model, in the technical solution of the present application, the time-frequency semantic feature vector of the millimeter-wave detection signal and the time-frequency semantic feature vector of the millimeter-wave detection reaction signal are input into the feature interaction integration module to obtain the millimeter-wave detection signal-detection reaction signal joint semantic feature vector. It is worth mentioning that the feature interaction integration module introduces a gating function, so that the weight and importance of the features can be adaptively adjusted according to different task requirements and data characteristics, and the model can better utilize the correlation and complementarity between the millimeter-wave detection signal and the detection reaction signal.

[0042] In an embodiment of the present application, the process of inputting the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection reaction signal time-frequency semantic feature vector into a feature interaction integration module to obtain a millimeter wave detection signal-detection reaction signal joint semantic feature vector includes: first, inputting the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection reaction signal time-frequency semantic feature vector into a feature combination module for cascade processing to obtain a millimeter wave detection signal-reaction signal time-frequency semantic joint feature vector; that is, by cascading the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection reaction signal time-frequency semantic feature vector to form a richer and more comprehensive time-frequency semantic joint feature representation. Next, the millimeter-wave detection signal-reaction signal time-frequency semantic joint feature vector is input into the gated response function to obtain the millimeter-wave detection signal-reaction signal time-frequency semantic information fusion response gating value; specifically, after calculating the matrix product of the millimeter-wave detection signal-reaction signal time-frequency semantic joint feature vector and the parameter matrix, the obtained feature vector and the bias vector are added by position to obtain the millimeter-wave detection signal-reaction signal time-frequency semantic joint bias feature vector; the millimeter-wave detection signal-reaction signal time-frequency semantic joint bias feature vector is input into the sigmoid activation function to obtain the millimeter-wave detection signal-reaction signal time-frequency semantic information fusion response gating value. Then, the positional product between the time-frequency semantic feature vector of the millimeter-wave detection signal and the time-frequency semantic information fusion response gating value of the millimeter-wave detection signal-reaction signal is calculated to obtain the time-frequency semantic gating adjustment vector of the millimeter-wave detection signal; the positional product between the time-frequency semantic feature vector of the millimeter-wave detection reaction signal and 1 minus the time-frequency semantic information fusion response gating value of the millimeter-wave detection signal-reaction signal is calculated to obtain the time-frequency semantic gating adjustment vector of the millimeter-wave detection reaction signal; that is, the time-frequency semantic joint feature is input into the gating function for adaptive adjustment of the weight, so as to automatically select according to the importance of the time-frequency semantic feature of the millimeter-wave detection signal and the time-frequency semantic feature of the millimeter-wave detection reaction signal, thereby improving the sensitivity of the model to key time-frequency semantic features, and suppressing the influence of unimportant or redundant features to obtain the time-frequency semantic information fusion response gating value. Finally, the millimeter-wave detection signal time-frequency semantic gating adjustment vector and the millimeter-wave detection response signal time-frequency semantic gating adjustment vector are added according to position to obtain the millimeter-wave detection signal-detection response signal joint semantic feature vector; that is, the time-frequency semantic information fusion response gating value is used to weightedly fuse the millimeter-wave detection signal time-frequency semantic feature vector and the millimeter-wave detection response signal time-frequency semantic feature vector to interactively learn and integrate the time-frequency semantic features of the millimeter-wave detection signal and the detection response signal, thereby obtaining a more information-rich and comprehensive millimeter-wave detection signal-detection response signal joint semantic feature vector.

[0043] In summary, in the above embodiment, the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection response signal time-frequency semantic feature vector are input into the feature interaction integration module to obtain the millimeter wave detection signal-detection response signal joint semantic feature vector, including: inputting the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection response signal time-frequency semantic feature vector into the feature interaction integration module, and processing them with the following dynamic interaction formula to obtain the millimeter wave detection signal-detection response signal joint semantic feature vector; wherein the dynamic integration formula is:

[0044]

[0045] in, and are respectively the time-frequency semantic feature vector of the millimeter wave detection signal and the time-frequency semantic feature vector of the millimeter wave detection response signal, It is cascade processing. is the parameter matrix, is the bias vector, is the sigmoid function, It is the joint semantic feature vector of the millimeter wave detection signal-detection response signal.

[0046] In particular, the S6 calculates the positional difference between the time-frequency semantic feature vector of the millimeter-wave detection signal and the time-frequency semantic feature vector of the millimeter-wave detection reaction signal to obtain a millimeter-wave detection signal-detection reaction signal differential semantic feature vector. In the technical solution of the present application, by calculating the positional difference between the time-frequency semantic feature vector of the millimeter-wave detection signal and the time-frequency semantic feature vector of the millimeter-wave detection reaction signal, the difference characteristics between the time-frequency semantic features of the millimeter-wave detection signal and the time-frequency semantic features of the millimeter-wave detection reaction signal are highlighted, so as to better capture the differences and associations between the signals to obtain a millimeter-wave detection signal-detection reaction signal differential semantic feature vector.

[0047] In particular, in S7, the millimeter wave detection signal-detection response signal joint semantic feature vector, the millimeter wave detection signal-detection response signal differential semantic feature vector, the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection response signal time-frequency semantic feature vector are cascaded to obtain a millimeter wave detection multimodal fusion representation vector as a millimeter wave detection multimodal fusion representation feature. It should be understood that in order to comprehensively utilize different aspects of information of the millimeter wave detection signal and the detection response signal, so as to more comprehensively describe the comprehensive characteristics of the millimeter wave detection signal, so as to facilitate the subsequent more accurate judgment of whether central sleep apnea occurs, in the technical solution of the present application, the millimeter wave detection signal-detection response signal joint semantic feature vector, the millimeter wave detection signal-detection response signal differential semantic feature vector, the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection response signal time-frequency semantic feature vector are cascaded to obtain a millimeter wave detection multimodal fusion representation vector with more distinguishing and expressive power.

[0048] In particular, the S8 obtains a monitoring result based on the millimeter wave detection multimodal fusion representation feature, and the monitoring result is used to indicate whether central sleep apnea occurs. In particular, in a specific example of the present application, the millimeter wave detection multimodal fusion representation vector is input into a sleep state monitor based on a classifier to obtain the monitoring result. That is, the millimeter wave detection multimodal fusion representation feature obtained by cascading the time-frequency semantic features of the millimeter wave detection signal and the time-frequency semantic features of the millimeter wave detection reaction signal in different aspects is used for classification processing to automatically determine whether central sleep apnea occurs. In this way, the millimeter wave radar can monitor without contacting the human body, avoiding the inconvenience and interference of wearing sensors in traditional methods, thereby improving the user's comfort and compliance. At the same time, the millimeter wave radar has high temporal and spatial resolution, can accurately detect tiny body movements and breathing changes, and use artificial intelligence technology for comprehensive and detailed analysis, which helps to more accurately identify and analyze sleep apnea problems.

[0049] In the technical solution of the present application, the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection reaction signal time-frequency semantic feature vector respectively represent the time-frequency semantic features of the millimeter wave detection signal and the millimeter wave detection reaction signal. The millimeter wave detection signal-detection reaction signal joint semantic feature vector represents the gate-driven feature interaction integrated representation of the millimeter wave detection signal and the millimeter wave detection reaction signal, and the millimeter wave detection signal-detection reaction signal differential semantic feature vector represents the signal semantic differential representation between the millimeter wave detection signal and the millimeter wave detection reaction signal. Cascading the millimeter wave detection signal-detection reaction signal joint semantic feature vector, the millimeter wave detection signal-detection reaction signal differential semantic feature vector, the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection reaction signal time-frequency semantic feature vector can improve the feature representation enrichment, but due to partial redundancy between features and structural alignment deviation between local features, the obtained millimeter wave detection multimodal fusion representation vector has a complex feature manifold representation, which makes it difficult to regress and identify the specific class relative to the classifier, affecting the accuracy of the monitoring results.

[0050] Based on this, in the technical solution of the present application, before the millimeter wave detection multimodal fusion representation vector is input into the classifier-based sleep state monitor, the millimeter wave detection multimodal fusion representation vector is feature modulated, and the feature modulation process includes:

[0051] Inputting the millimeter wave detection multimodal fusion representation vector into a pre-classifier based on a Softmax function to obtain a millimeter wave detection multimodal fusion fine-grained class probability label vector;

[0052] Calculate the product between the millimeter wave detection multimodal fusion fine-grained class probability label vector and its transposed vector to obtain a millimeter wave detection multimodal fusion fine-grained full label domain modulation matrix;

[0053] Dividing the eigenvalues ​​of each position in the millimeter-wave detection multimodal fusion fine-grained full-label domain modulation matrix by the square root of the scale value of the millimeter-wave detection multimodal fusion representation vector to obtain the millimeter-wave detection multimodal fusion fine-grained full-label domain scale modulation matrix;

[0054] Inputting the millimeter wave detection multimodal fusion fine-grained full-label domain scale modulation matrix into a quasi-probability domain attention sparse module based on a multi-level gating function to obtain a millimeter wave detection multimodal fusion fine-grained full-label domain scale sparse modulation matrix;

[0055] The millimeter-wave detection multimodal fusion representation vector is used as a query feature vector, and the millimeter-wave detection multimodal fusion fine-grained full-label domain scale sparse modulation matrix is ​​matrix-multiplied with the millimeter-wave detection multimodal fusion representation vector to obtain an optimized millimeter-wave detection multimodal fusion representation vector.

[0056] The optimization process of the millimeter wave detection multimodal fusion representation vector is expressed by the formula:

[0057]

[0058]

[0059]

[0060]

[0061]

[0062] in, represents the millimeter wave detection multimodal fusion representation vector, The first representation vector of the millimeter wave detection multimodal fusion representation vector is represented by The eigenvalues ​​at the positions, represents the scale value of the millimeter wave detection multimodal fusion representation vector, The first one represents the fine-grained class probability label vector of millimeter wave detection multimodal fusion The eigenvalues ​​at the positions, represents the millimeter wave detection multimodal fusion fine-grained class probability label vector, represents matrix multiplication, represents the transpose of a vector, represents the millimeter wave detection multimodal fusion fine-grained full-label domain scale modulation matrix, represents a predetermined threshold, The first one represents the fine-grained full-label domain scale modulation matrix of millimeter-wave detection multimodal fusion The eigenvalues ​​at the positions, represents the predetermined hyperparameters, represents the mask function, The first one represents the fine-grained full-label domain-scale sparse modulation matrix of millimeter-wave detection multimodal fusion The eigenvalues ​​at the positions, represents the fine-grained full-label domain-scale sparse modulation matrix of millimeter-wave detection multimodal fusion, Represents the optimized millimeter-wave detection multimodal fusion representation vector.

[0063] Thus, by adding a class probability domain-level optimizable perturbation to the original feature vector to strengthen the dependency entanglement between different variables in the original feature vector based on label domain modulation, the source domain feature vector is optimized based on class probability query scale modulation. In this way, the significant inherent modal information in the source domain features can be more effectively retained, and the interference components in the feature distribution can be effectively masked, thereby improving the adversarial robustness of the manifold expression of the feature vector to promote the class regression decomposition recognition of the complex feature structure of the millimeter wave detection multimodal fusion representation vector, and improve the accuracy of the monitoring results. In this way, the millimeter wave radar can monitor without contacting the human body, avoiding the inconvenience and interference of wearing sensors in traditional methods, thereby improving the user's comfort and compliance. At the same time, the millimeter wave radar has high temporal and spatial resolution, can accurately detect tiny body movements and breathing changes, and use artificial intelligence technology for comprehensive and detailed analysis, which helps to more accurately identify and analyze sleep apnea problems.

[0064] In summary, the sleep monitoring method based on millimeter-wave radar according to the embodiment of the present application is explained, which uses artificial intelligence-based signal processing and analysis technology to extract time-frequency features and semantically combine millimeter-wave detection signals and millimeter-wave detection response signals, so as to automatically determine whether central sleep apnea occurs based on the fusion representation features of the millimeter-wave detection signal and the millimeter-wave detection response signal in multimodality. In this way, the millimeter-wave radar can monitor without contacting the human body, avoiding the inconvenience and interference of wearing sensors in traditional methods, thereby improving the user's comfort and compliance. At the same time, the millimeter-wave radar has high temporal and spatial resolution, can accurately detect tiny body movements and breathing changes, and use artificial intelligence technology for comprehensive and detailed analysis, which helps to more accurately identify and analyze sleep apnea problems.

[0065] Furthermore, a sleep monitoring system based on millimeter wave radar is also provided. Figure 2 FIG. 1 is a block diagram of a sleep monitoring system based on millimeter wave radar according to an embodiment of the present application. Figure 2As shown, according to the sleep monitoring system 300 based on millimeter wave radar according to the embodiment of the present application, it includes: a millimeter wave detection signal transmitting module 310, which is used to transmit a millimeter wave detection signal through a millimeter wave radar; a millimeter wave detection reaction signal receiving module 320, which is used to receive the millimeter wave detection signal reflected back through the millimeter wave radar to obtain a millimeter wave detection reaction signal; a signal time-frequency extraction module 330, which is used to perform signal time-frequency extraction on the millimeter wave detection signal and the millimeter wave detection reaction signal to obtain a millimeter wave detection signal time-frequency image and a millimeter wave detection reaction signal time-frequency image; a signal time-frequency feature analysis module 340, which is used to perform signal time-frequency feature analysis on the millimeter wave detection signal time-frequency image and the millimeter wave detection reaction signal time-frequency image to obtain a millimeter wave detection signal time-frequency semantic feature vector and a millimeter wave detection reaction signal time-frequency semantic feature vector; a vector dynamic interaction integration module 350, which is used to integrate the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection reaction signal The time-frequency semantic feature vector of the millimeter wave detection signal is input into the feature interaction integration module to obtain the millimeter wave detection signal-detection reaction signal joint semantic feature vector; the difference module 360 ​​is used to calculate the position difference between the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection reaction signal time-frequency semantic feature vector to obtain the millimeter wave detection signal-detection reaction signal differential semantic feature vector; the millimeter wave detection multimodal fusion module 370 is used to cascade the millimeter wave detection signal-detection reaction signal joint semantic feature vector, the millimeter wave detection signal-detection reaction signal differential semantic feature vector, the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection reaction signal time-frequency semantic feature vector to obtain the millimeter wave detection multimodal fusion representation vector as the millimeter wave detection multimodal fusion representation feature; the monitoring result generation module 380 is used to obtain the monitoring result based on the millimeter wave detection multimodal fusion representation feature, and the monitoring result is used to indicate whether central sleep apnea occurs.

[0066] As described above, the millimeter wave radar-based sleep monitoring system 300 according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a millimeter wave radar-based sleep monitoring algorithm. In one possible implementation, the millimeter wave radar-based sleep monitoring system 300 according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the millimeter wave radar-based sleep monitoring system 300 can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the millimeter wave radar-based sleep monitoring system 300 can also be one of the many hardware modules of the wireless terminal.

[0067] Alternatively, in another example, the millimeter-wave radar-based sleep monitoring system 300 and the wireless terminal may also be separate devices, and the millimeter-wave radar-based sleep monitoring system 300 may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0068] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps in the functions of the millimeter-wave radar-based sleep monitoring method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0069] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0070] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A sleep monitoring system based on millimeter wave radar, characterized in that: include: A millimeter wave detection signal transmitting module, used for transmitting a millimeter wave detection signal through a millimeter wave radar; A millimeter wave detection response signal receiving module, used for receiving the millimeter wave detection signal reflected back by the millimeter wave radar to obtain a millimeter wave detection response signal; A signal time-frequency extraction module, used to perform signal time-frequency extraction on the millimeter wave detection signal and the millimeter wave detection response signal to obtain a millimeter wave detection signal time-frequency image and a millimeter wave detection response signal time-frequency image; A signal time-frequency feature analysis module, used to perform signal time-frequency feature analysis on the millimeter wave detection signal time-frequency image and the millimeter wave detection response signal time-frequency image to obtain a millimeter wave detection signal time-frequency semantic feature vector and a millimeter wave detection response signal time-frequency semantic feature vector; A vector dynamic interaction integration module, used for inputting the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection response signal time-frequency semantic feature vector into a feature interaction integration module to obtain a millimeter wave detection signal-detection response signal joint semantic feature vector; A difference module, used for calculating the position difference between the time-frequency semantic feature vector of the millimeter wave detection signal and the time-frequency semantic feature vector of the millimeter wave detection response signal to obtain a millimeter wave detection signal-detection response signal differential semantic feature vector; A millimeter wave detection multimodal fusion module, used for cascading the millimeter wave detection signal-detection response signal joint semantic feature vector, the millimeter wave detection signal-detection response signal differential semantic feature vector, the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection response signal time-frequency semantic feature vector to obtain a millimeter wave detection multimodal fusion representation vector as a millimeter wave detection multimodal fusion representation feature; A monitoring result generating module, used to obtain a monitoring result based on the millimeter wave detection multimodal fusion representation feature, wherein the monitoring result is used to indicate whether central sleep apnea occurs; The method of inputting the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection response signal time-frequency semantic feature vector into a feature interaction integration module to obtain a millimeter wave detection signal-detection response signal joint semantic feature vector comprises: inputting the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection response signal time-frequency semantic feature vector into a feature interaction integration module, and processing them with the following dynamic interaction formula to obtain the millimeter wave detection signal-detection response signal joint semantic feature vector; wherein the dynamic integration formula is: ； in, and are respectively the time-frequency semantic feature vector of the millimeter wave detection signal and the time-frequency semantic feature vector of the millimeter wave detection response signal, It is cascade processing. is the parameter matrix, is the bias vector, is the sigmoid function, It is the joint semantic feature vector of the millimeter wave detection signal-detection response signal.

2. The sleep monitoring system based on millimeter wave radar according to claim 1, characterized in that: The millimeter wave detection signal and the millimeter wave detection response signal are subjected to signal time-frequency extraction to obtain a millimeter wave detection signal time-frequency image and a millimeter wave detection response signal time-frequency image, including: performing wavelet transform on the millimeter wave detection signal and the millimeter wave detection response signal to obtain a millimeter wave detection signal time-frequency image and a millimeter wave detection response signal time-frequency image.

3. The sleep monitoring system based on millimeter wave radar according to claim 2, characterized in that: The signal time-frequency feature analysis is performed on the millimeter-wave detection signal time-frequency image and the millimeter-wave detection reaction signal time-frequency image to obtain the millimeter-wave detection signal time-frequency semantic feature vector and the millimeter-wave detection reaction signal time-frequency semantic feature vector, including: inputting the millimeter-wave detection signal time-frequency image and the millimeter-wave detection reaction signal time-frequency image into a signal time-frequency feature extractor based on a void convolutional neural network model to obtain the millimeter-wave detection signal time-frequency semantic feature vector and the millimeter-wave detection reaction signal time-frequency semantic feature vector.

4. The millimeter wave radar-based sleep monitoring system according to claim 3, characterized in that: Inputting the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection response signal time-frequency semantic feature vector into a feature interaction integration module to obtain a millimeter wave detection signal-detection response signal joint semantic feature vector, including: Inputting the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection reaction signal time-frequency semantic feature vector into a feature combination module for cascade processing to obtain a millimeter wave detection signal-reaction signal time-frequency semantic combination feature vector; Inputting the millimeter wave detection signal-reaction signal time-frequency semantic joint feature vector into the gated response function to obtain the millimeter wave detection signal-reaction signal time-frequency semantic information fusion response gated value; Calculating the positional product between the millimeter wave detection signal time-frequency semantic feature vector and the millimeter wave detection signal-reaction signal time-frequency semantic information fusion response gating value to obtain a millimeter wave detection signal time-frequency semantic gating adjustment vector; Calculate the positional product of the millimeter wave detection reaction signal time-frequency semantic feature vector and 1 minus the millimeter wave detection signal-reaction signal time-frequency semantic information fusion response gating value to obtain a millimeter wave detection reaction signal time-frequency semantic gating adjustment vector; The millimeter-wave detection signal time-frequency semantic gating adjustment vector and the millimeter-wave detection response signal time-frequency semantic gating adjustment vector are added according to position to obtain the millimeter-wave detection signal-detection response signal joint semantic feature vector.

5. The millimeter wave radar-based sleep monitoring system according to claim 4, characterized in that: Inputting the millimeter wave detection signal-reaction signal time-frequency semantic joint feature vector into the gated response function to obtain the millimeter wave detection signal-reaction signal time-frequency semantic information fusion response gated value, including: After calculating the matrix product of the millimeter wave detection signal-reaction signal time-frequency semantic joint feature vector and the parameter matrix, the obtained feature vector and the bias vector are added according to the position to obtain the millimeter wave detection signal-reaction signal time-frequency semantic joint bias feature vector; The millimeter wave detection signal-reaction signal time-frequency semantic joint bias feature vector is input into a sigmoid activation function to obtain the millimeter wave detection signal-reaction signal time-frequency semantic information fusion response gating value.

6. The millimeter wave radar-based sleep monitoring system according to claim 5, characterized in that: Based on the millimeter wave detection multimodal fusion representation feature, a monitoring result is obtained, including: inputting the millimeter wave detection multimodal fusion representation vector into a classifier-based sleep state monitor to obtain the monitoring result.

7. A storage medium, characterized in that , including a computer program, which, when executed on a computer device, enables a processing unit in the computer device to execute a sleep monitoring system based on millimeter-wave radar as described in any one of claims 1 to 6.

Images