Sleep monitoring model training method, sleep monitoring method, and device

Abstract

The application provides a sleep monitoring model training method, a sleep monitoring method and equipment. It is applied to the field of radar signal processing. The method comprises the following steps: obtaining a data set S1 and a data set S2; training a pure radar feature extractor M3 by using a pre-trained teacher network model M2 and the data set S1, the teacher network model M2 fuses radar data and pulse wave data in the data set S1 and outputs fusion feature data, and the pure radar feature extractor M3 extracts features from the radar data in the data set S1 and outputs radar feature data; training a dual-modal sleep monitoring model M4 and a pure radar modal sleep monitoring model M5 by using the data set S2, a first identification layer determines first sleep staging and / or respiratory event information according to the fusion feature data output by the teacher network model M2, and a second identification layer determines second sleep staging and / or respiratory event information according to the radar feature data output by the pure radar feature extractor M3. The application realizes a low-cost and efficient sleep monitoring task.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of radar signal processing, in particular to a sleep monitoring model training method, a sleep monitoring method and equipment. BACKGROUND

[0002] Sleep is a core physiological activity for maintaining human health, and is crucial for neural system repair, metabolic regulation and immune balance. Sleep disorders are common and serious worldwide, and not only affect daytime status (such as memory decline, sleepiness), but also increase long-term health risks such as cardiovascular disease and depression, driving the demand for accurate and convenient sleep monitoring to continue to grow.

[0003] The current mainstream sleep monitoring technology is polysomnography (PSG) - known as the "gold standard". By pasting electrodes and sensors such as electroencephalogram, electrooculogram and electromyogram on the body surface of the patient, physiological signals such as brain waves, respiration and heart rate are synchronously collected, providing a basis for sleep staging and disease diagnosis. However, the traditional polysomnography is limited in that the monitoring process requires multiple electrodes to be pasted on the body surface of the patient and connected to sensors. The fixation and connection of these elements are easily affected by factors such as wearing time and body position, resulting in poor contact between the electrodes and the skin or interruption of the sensor signal. Since the equipment needs to be maintained stable, the patient is limited in turning over and adjusting body position during sleep, and the sleep state deviates from the natural mode, resulting in distorted monitoring data and difficulty in accurately reflecting the true sleep disorder degree.

[0004] The application of new sensors such as millimeter wave radar sensors makes non-contact and low-invasive sleep monitoring possible. According to the high-frequency radio signals transmitted and received by the millimeter wave radar sensor, information such as human respiration, heartbeat and body movement can be obtained, which can be used to determine sleep staging and is suitable for sleep monitoring in home and medical environments, with the advantages of non-contact, no interference with sleep and high monitoring accuracy.

[0005] In order to determine sleep staging according to the human information detected by the millimeter wave radar sensor, a feasible solution is to use a neural network model to identify the data. The conventional model training idea in this application scenario is to synchronously collect PSG and radar data, and use the PSG monitoring results as labels to train the neural network model. However, the PSG data collection process is complex, costly and has a great disturbance to the subjects, making it difficult to quickly obtain a large number of samples.

[0006] Due to the difficulty in efficiently obtaining training samples with labels, the neural network model training efficiency is low, which further affects the performance of the model in sleep state monitoring. SUMMARY

[0007] Therefore, the first aspect of the present application provides a sleep monitoring model training method, comprising:

[0008] obtaining a dataset S1 and a dataset S2, wherein the dataset S1 comprises radar data and pulse wave data collected synchronously by a millimeter wave radar device and a pulse sensor, the dataset S2 comprises radar data, pulse wave data and PSG data collected synchronously by the millimeter wave radar device, the pulse sensor and a PSG device, and sleep stage labels and / or respiratory event labels annotated based on the PSG data, and the amount of data of the dataset S2 is less than that of the dataset S1;

[0009] training a pure radar feature extractor M3 using the pre-trained teacher network model M2 and the dataset S1, the training process comprising the teacher network model M2 fusing the radar data and the pulse wave data in the dataset S1 to output fusion feature data, the pure radar feature extractor M3 extracting features from the radar data in the dataset S1 to output radar feature data, and then calculating a loss according to the fusion feature data and the radar feature data;

[0010] training a dual-modal sleep monitoring model M4 and a pure radar modal sleep monitoring model M5 using the dataset S2, wherein the dual-modal sleep monitoring model M4 comprises the trained teacher network model M2 and a first recognition layer, the pure radar modal sleep monitoring model M5 comprises the trained pure radar feature extractor M3 and a second recognition layer, the training process comprising the first recognition layer determining first sleep stage and / or respiratory event information according to the fusion feature data output by the teacher network model M2, the second recognition layer determining second sleep stage and / or respiratory event information according to the radar feature data output by the pure radar feature extractor M3, and calculating a loss according to the first sleep stage and / or respiratory event information, the second sleep stage and / or respiratory event information, the sleep stage labels and / or respiratory event labels.

[0011] Optionally, the pre-trained teacher network model M2 is obtained as follows:

[0012] training an auto-encoder network model M1 using the dataset S1 by an unsupervised learning method, the auto-encoder network model M1 comprising an encoder, a fusion layer and a decoder, the training process comprising the encoder encoding the radar data and the pulse wave data in the dataset S1 respectively to convert them into radar feature vectors and pulse wave feature vectors of the same dimension, the fusion layer fusing the radar feature vectors and the pulse wave feature vectors to obtain fusion feature data, the decoder decoding the fusion feature data to obtain reconstructed radar data and reconstructed pulse wave data, calculating a loss using the radar data, the pulse wave data, the reconstructed radar data and the reconstructed pulse wave data, and the encoder and the fusion layer in the trained auto-encoder network model M1 constituting the pre-trained teacher network model M2.

[0013] Optionally, the loss is calculated using the radar data, the pulse wave data, the reconstructed radar data and the reconstructed pulse wave data in the following manner :

[0014] ;

[0015] wherein, represents the radar data, represents the pulse wave data, represents the reconstructed radar data obtained by simultaneously inputting and into the model, represents the reconstructed pulse wave data obtained by simultaneously inputting and into the model, represents the reconstructed pulse wave data obtained by only inputting into the model, 、 、 is a weighting coefficient, represents the mean square error loss of and , represents the mean square error loss of and , represents the mean square error loss of and .

[0016] Optionally, the encoder comprises a radar data encoder and a pulse wave data encoder, and the decoder comprises a radar data decoder and a pulse wave data decoder, the radar data encoder is configured to encode the radar data in the data set S1 to obtain the radar feature vector, the pulse wave data encoder is configured to encode the pulse wave data in the data set S1 to obtain the pulse wave feature vector, the radar data decoder is configured to decode the fusion feature data to obtain the reconstructed radar data, and the pulse wave data decoder is configured to decode the fusion feature data to obtain the reconstructed pulse wave data.

[0017] Optionally, the first identification layer comprises a first full connection layer and a first probability activation layer, the first full connection layer is configured to perform linear transformation and non-linear mapping on the fusion feature data output by the teacher network model M2 to obtain a first classification feature vector matching the number of sleep staging and / or respiratory event categories, and the first probability activation layer is configured to calculate the probability distribution of each sleep staging and / or respiratory event category based on the first classification feature vector to obtain the first sleep staging and / or respiratory event information.

[0018] Optionally, the second identification layer comprises a second full connection layer and a second probability activation layer, the second full connection layer is used for linear transformation and nonlinear mapping of the radar feature data output by the pure radar feature extractor M3, to obtain a second classification feature vector matched with the number of sleep staging and / or respiratory event categories, and the second probability activation layer is used for calculating the probability distribution of each sleep staging and / or respiratory event category based on the second classification feature vector, to obtain second sleep staging and / or respiratory event information.

[0019] Optionally, the present application provides a sleep monitoring model training method, further comprising: preprocessing the data set S1 and the data set S2 respectively, wherein the preprocessed radar data is a distance-time two-dimensional spectrum, containing three channels of low-frequency energy spectrum, high-frequency energy spectrum and phase Doppler spectrum, and the preprocessed pulse wave data is a frequency-time two-dimensional spectrum, containing one channel of energy spectrum.

[0020] The second aspect of the present application provides a sleep monitoring method based on millimeter wave radar, comprising:

[0021] Obtaining radar data of a target object;

[0022] The pure radar modality sleep monitoring model M5 trained by any one of the sleep monitoring model training methods is used for feature recognition of the radar data, to obtain target sleep staging and / or respiratory event information.

[0023] The third aspect of the present application provides a sleep monitoring model training device, comprising: a processor and a memory connected with the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor to make the processor execute the sleep monitoring model training method described above.

[0024] In a fourth aspect, the present application provides a millimeter wave radar-based sleep monitoring device, comprising a processor and a memory connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor to enable the processor to perform the millimeter wave radar-based sleep monitoring method described above. The sleep monitoring model training method provided by the present application first acquires a large amount of unlabeled sample data set S1 (radar + pulse wave) collected synchronously, and combines a small amount of labeled sample data set S2 (radar + pulse wave + sleep stage label + respiratory event label), which takes into account both data size and labeling accuracy. Second, the pre-trained teacher network model M2 is used to perform knowledge transfer training on the pure radar feature extractor M3, and the information contained in the pulse wave data is used to help the pure radar feature extractor M3 better learn and understand the radar data, thereby enhancing the feature representation capability of the pure radar feature extractor M3 on the radar data. Without PSG equipment or manual interpretation, the problem of insufficient sample size caused by low PSG labeling efficiency and high cost in traditional methods is solved. Then, a small amount of high-labeled data set S2 is introduced to fine-tune the dual-modal sleep monitoring model M4 and the pure radar modal sleep monitoring model M5, calculate the feature loss and classification loss, and optimize through multi-loss weighting, correct the feature deviation of the pure radar feature extractor M3 on the single-modal radar data, and enable the pure radar feature extractor M3 to accurately capture the key discriminant information (such as respiratory abnormalities, body movement intensity, and sleep stage correlation) of sleep staging and / or respiratory events through radar data only. This fine-tuning process only requires a small amount of labeled data, greatly reducing the labeling cost. Through the cooperative strategy of "unlabeled big data pre-training + small sample labeling fine-tuning", combined with the low-load advantage of the pure radar modal, the present application not only solves the problem of insufficient labeled data, but also avoids the high cost and high interference defects of traditional multi-modal devices, and realizes efficient and comfortable sleep monitoring.

[0025] The millimeter wave radar-based sleep monitoring method provided by the present application only collects data through non-contact millimeter wave radar, avoiding the physical discomfort of the target object caused by traditional contact devices, and being more suitable for long-term and comfortable sleep monitoring requirements. Moreover, during the model training process, cross-modal knowledge transfer (using pulse wave data for auxiliary learning) is used to enhance the feature representation capability of the pure radar modal sleep monitoring model on single-modal radar data, without relying on high-load pulse wave or PSG equipment. The present application can realize accurate sleep monitoring through non-contact millimeter wave radar, so that it can more accurately capture the key discriminant information of sleep staging and / or respiratory events, and finally realize efficient and reliable non-contact sleep monitoring. BRIEF DESCRIPTION OF DRAWINGS

[0026] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a flowchart of the sleep monitoring model training method in an embodiment of the present invention;

[0028] Figure 2 This is a structural diagram of the teacher network model M2 in this embodiment of the invention;

[0029] Figure 3 This is a structural diagram of the autoencoder network model M1 in this embodiment of the invention;

[0030] Figure 4 This is a structural diagram of the dual-modal sleep monitoring model M4 and the pure radar modal sleep monitoring model M5 in the embodiments of the present invention;

[0031] Figure 5 This is a flowchart of a sleep monitoring method based on millimeter-wave radar according to an embodiment of the present invention. Detailed Implementation

[0032] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] In the description of this invention, it should be noted that the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0034] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0035] like Figure 1 As shown, this embodiment of the invention provides a method for training a sleep monitoring model, which is executed by an electronic device such as a computer or server, and specifically includes:

[0036] S11, obtain a data set S1 and a data set S2, wherein the data set S1 includes radar data and pulse wave data synchronously collected by the millimeter wave radar device and the pulse sensor, the data set S2 includes radar data, pulse wave data and PSG data synchronously collected by the millimeter wave radar device, the pulse sensor and the PSG device, and sleep staging labels and / or respiratory event labels labeled based on the PSG data, and the data amount of the data set S2 is less than the data amount of the data set S1.

[0037] Synchronization acquisition of sleep monitoring data of a target for a corresponding time length (such as one night) by using a millimeter wave radar and a pulse sensor, one night of monitoring data of a person is referred to as a data sample, without manual labeling of the collected data sample, and a data set S1 composed of a collection of all the collected data samples.

[0038] Synchronization acquisition of sleep monitoring data of a target for a corresponding time length (such as one night) by using a millimeter wave radar, a pulse sensor and a polysomnogram (PSG), one night of monitoring data of a person is referred to as a data sample, and each collected data sample is manually labeled (a professional technician interprets PSG data and labels), and the labeling content is sleep staging (such as a wake period, non-rapid eye movement sleep periods N1-N3 period, rapid eye movement sleep period REM period) and / or respiratory events (such as obstructive sleep apnea, central sleep apnea, low ventilation events, etc.), and finally the labeled radar data, pulse wave data and corresponding sleep staging labels and / or respiratory event labels (staging / event categories) are integrated to form a data set S2 for model training; since the polysomnogram in the collected data set S2 and the labeling efficiency are low, the sample amount of the collected data set S1 is much larger than the sample amount of the data set S2.

[0039] S12, training of a pure radar feature extractor M3 by using a pre-trained teacher network model M2 and the data set S1, the training process including fusion of radar data and pulse wave data in the data set S1 by the teacher network model M2, output of fusion feature data, feature extraction of the radar data in the data set S1 by the pure radar feature extractor M3, output of radar feature data, and further calculation of a loss according to the fusion feature data and the radar feature data.

[0040] As Figure 2As shown, the teacher network model M2 is a pre-trained model, and a feature distillation method is used to train the pure radar feature extractor M3 using the data set S1, wherein the input of the teacher network model M2 is radar data and pulse wave data, and the input of the student network model M3 is only radar data. The output of the teacher network model M2 is feature data, and the output of the pure radar feature extractor M3 is also feature data. The loss function used when training the pure radar feature extractor M3 is mean square error loss, and the parameters of the teacher network model M2 are fixed during the training process, and the feature extraction capability of the teacher network model M2 is migrated to the pure radar feature extractor M3.

[0041] S13, training the dual-modal sleep monitoring model M4 and the pure radar modal sleep monitoring model M5 using the data set S2, wherein the dual-modal sleep monitoring model M4 includes the trained teacher network model M2 and the first identification layer, the pure radar modal sleep monitoring model M5 includes the trained pure radar feature extractor M3 and the second identification layer, and the training process includes determining the first sleep staging and / or respiratory event information according to the fusion feature data output by the teacher network model M2 by the first identification layer, determining the second sleep staging and / or respiratory event information according to the radar feature data output by the pure radar feature extractor M3 by the second identification layer, and calculating the loss according to the first sleep staging and / or respiratory event information, the second sleep staging and / or respiratory event information, the sleep staging label and / or the respiratory event label.

[0042] The dual-modal sleep monitoring model M4 and the pure radar modal sleep monitoring model M5 are fine-tuned using the data set S2, specifically, the dual-modal sleep monitoring model M4 is trained using the pulse wave data and the radar data in the data set S2, and the pure radar modal sleep monitoring model M5 is trained using only the radar data in the data set S2. During the training process, flexible parameter adjustment strategies can be adopted for the dual-modal sleep monitoring model M4 and the pure radar modal sleep monitoring model M5: the parameters of the encoder, the fusion layer, and the pure radar feature extractor M3 can be directly fixed (to avoid destroying the learned multi-modal / single-modal feature extraction capability), or the influence of training on the parameters can be weakened by reducing the learning rate or using other learning rate adjustment strategies (such as weight decay). For example, Figure 4As shown, the bimodal sleep monitoring model M4 includes the trained teacher network model M2 and a first recognition layer, and the pure radar modality sleep monitoring model M5 includes the trained pure radar feature extractor M3 and a second recognition layer. The first recognition layer determines first sleep staging and / or respiratory event information (predicted label 2) according to the fusion feature data output by the teacher network model M2, and the second recognition layer determines second sleep staging and / or respiratory event information (predicted label 1) according to the radar feature data output by the pure radar feature extractor M3. Then, the difference (such as cross-entropy loss) between the second sleep staging and / or respiratory event information (predicted label 1) output by the pure radar modality sleep monitoring model M5, the first sleep staging and / or respiratory event information (predicted label 2), and the true labeled information (sleep staging label and / or respiratory event label) is calculated, to obtain a first classification loss (classification loss 1) and a second classification loss (classification loss 2), which measure the discrimination accuracy of the two for sleep staging and / or respiratory event. The feature loss calculated from the fusion feature data output by the teacher network model M2 and the radar feature data output by the pure radar feature extractor M3, such as mean square error, is calculated to ensure that the single modality feature of the pure radar feature extractor can inherit the multi-modality discrimination information of the teacher network model. Then, the feature loss, the first classification loss, and the second classification loss are weighted and calculated to obtain a total loss. When the total loss converges to a first preset threshold, the training of the pure radar modality sleep monitoring model M5 is completed.

[0043] The embodiment first acquires a large sample data set S1 (radar + pulse wave) without labeling in synchronous collection, and combines a small amount of labeled sample data set S2 (radar + pulse wave + sleep staging label + respiratory event label), so as to balance data size and labeling accuracy; secondly, the pre-trained teacher network model M2 is used to perform knowledge transfer training on the pure radar feature extractor M3, the information contained in the pulse wave data is used to help the pure radar feature extractor M3 to better learn and understand the radar data, and the feature representation capability of the pure radar feature extractor M3 on the radar data is enhanced, without PSG device or manual interpretation, so as to solve the problem of insufficient sample quantity caused by low PSG labeling efficiency and high cost in the traditional method; then, a small amount of high labeling value data set S2 is introduced to fine-tune the dual-mode sleep monitoring model M4 and the pure radar mode sleep monitoring model M5, the feature loss and the classification loss are calculated, and the multi-loss weighted optimization is used to correct the feature deviation of the pure radar feature extractor M3 on the single-mode radar data, so that the pure radar feature extractor M3 can accurately capture the key discriminant information (such as respiratory abnormalities, body movement intensity and sleep staging correlation) of sleep staging and / or respiratory events through only radar data. The fine-tuning process only needs a small amount of labeled data, which greatly reduces the labeling cost. Through the cooperative strategy of "unlabeled big data pre-training + small sample labeling fine-tuning", combined with the low load advantage of the pure radar mode, the method not only solves the problem of insufficient labeled data, but also avoids the defects of high cost and high interference of the traditional multi-modal device, and realizes efficient and comfortable sleep monitoring.

[0044] In some optional embodiments of the embodiment, the pre-trained teacher network model M2 in step S12 is obtained as follows:

[0045] The unsupervised learning method is used to train the autoencoder network model M1 by using the data set S1. The autoencoder network model M1 includes an encoder, a fusion layer and a decoder. The training process includes that the encoder encodes the radar data and the pulse wave data in the data set S1 respectively, and converts them into radar feature vectors and pulse wave feature vectors with the same dimension. The fusion layer fuses the radar feature vectors and the pulse wave feature vectors to obtain fusion feature data. The decoder decodes the fusion feature data to obtain reconstructed radar data and reconstructed pulse wave data. Loss is calculated by using the radar data, the pulse wave data, the reconstructed radar data and the reconstructed pulse wave data. The encoder and the fusion layer in the trained autoencoder network model M1 constitute the pre-trained teacher network model M2.

[0046] As Figure 3As shown, the autoencoder network model M1 includes an encoder, a fusion layer, and a decoder, wherein the encoder is used to encode the radar data and pulse wave data in the data set S1 respectively to obtain radar feature vectors and pulse wave feature vectors, and send them to the fusion layer. Since the fusion layer needs to jointly process the radar feature vectors and pulse wave feature vectors (such as cross-attention fusion mechanism or feature alignment), only when the two have the same dimension can the fusion layer effectively integrate multi-modal information and avoid information loss or calculation errors due to dimension mismatch. The specific fusion layer can use a cross-attention mechanism to obtain the correlation weight between features of different modalities (radar and pulse wave), and then weight and sum the features to obtain fusion feature data. Then the decoder is used to reconstruct the fusion feature data to obtain reconstructed radar data and reconstructed pulse wave data, which have the same dimension as the radar data in the data set S1 and the pulse wave data in the data set S1 respectively.

[0047] The fused fusion feature data is sent to the decoder, which can have a structure symmetrical to the encoder or other structures. The output of the decoder keeps the same data dimension as the input of the encoder. Among them, the decoder has the same structure and dimension as the encoder, which is to realize the reverse recovery of information (such as convolution transpose and attention alignment) through the symmetrical structure, and to ensure that the compressed features and the original input are strictly aligned in data dimension, so as to guarantee the integrity and accuracy of the reconstructed data.

[0048] The model parameters of the autoencoder network model M1 are optimized by calculating the loss function of the radar data, the pulse wave data, the reconstructed radar data, and the reconstructed pulse wave data, until the loss converges to a second preset threshold, and the training of the autoencoder network model M1 is completed. The encoder and the fusion layer in the trained autoencoder network model M1 are split out to form a teacher network model M2 with multi-modal joint feature extraction capability, and the structure of the teacher network model M2 is as shown in Figure 2 Only the encoder and the fusion layer are needed to form the teacher network model M2, because the training target of the autoencoder network model M1 is to let the encoder learn to extract high-quality features from multi-modal data (radar + pulse wave), and verify the effectiveness of the features through the reconstruction task of the decoder. After training, the encoder has completed the learning of “multi-modal feature extraction capability”, and the fusion layer has mastered the logic of “integrating multi-modal information”; the core role of the decoder is to assist training (verify whether the features are complete by reconstructing the original data), and it does not directly participate in feature extraction. Therefore, the teacher network model only needs to retain the encoder (responsible for feature extraction) and the fusion layer (responsible for multi-modal information integration), so as to realize the joint feature extraction function of multi-modal data, without the need to additionally retain the decoder.

[0049] The present embodiment trains the autoencoder network model M1 using the unlabeled multi-modal dataset S1 through unsupervised learning, so that the encoder learns the key features (such as respiratory rate, body movement, heart rate, etc.) of the radar and pulse wave data and maps them into a same-dimension feature vector. The fusion layer integrates the multi-modal features through a cross-attention mechanism, and the decoder verifies the integrity of the feature extraction by inversely reconstructing the fused features into radar data and pulse wave data, and then ensures that the features retain the core information through loss constraint (minimizing the reconstruction loss). Finally, the encoder and the fusion layer form the teacher network model M2. Therefore, the teacher network model M2 is trained only through the unlabeled and easily accessible dataset S1 (without relying on high-cost PSG labeled data), which avoids the dependence on labeled data and lays a solid foundation for the feature learning of the pure radar feature extractor M3, significantly enhancing its comprehensive representation ability for sleep staging and / or respiratory events.

[0050] In some optional embodiments of the present embodiment, the loss is calculated using the radar data, the pulse wave data, the reconstructed radar data, and the reconstructed pulse wave data in the following manner :

[0051] ;

[0052] wherein, represents the radar data, represents the pulse wave data, represents the reconstructed radar data obtained by simultaneously inputting and into the model, represents the reconstructed pulse wave data obtained by simultaneously inputting and into the model, represents the reconstructed pulse wave data obtained by inputting into the model only, , is a weighting coefficient, represents the mean square error loss of and , represents the mean square error loss of and , represents the mean square error loss of and .

[0053] The loss function used in the present embodiment includes two types of multi-modal reconstruction error loss and single-modal reconstruction error loss, and the total loss function is obtained by weighted summation of the two types of loss. The multi-modal reconstruction error loss ( , ​) and the weighted joint optimization of the single-modal cross-modal reconstruction error loss ( ), which can respectively constrain the reconstruction accuracy of radar and pulse wave data to improve the reliability of multi-modal feature extraction, and can force the model to learn the correlation between multi-modal through the cross-modal reconstruction task, thereby enhancing the fusion representation ability of the autoencoder network model M1 for multi-modal information, and providing a more robust feature basis for the knowledge transfer of the subsequent pure radar feature extractor M3.

[0054] In some optional embodiments of the present embodiment, the encoder in the autoencoder network model M1, the teacher network model M2 and the dual-modal sleep monitoring model M4 includes a radar data encoder and a pulse wave data encoder, and the decoder includes a radar data decoder and a pulse wave data decoder. The radar data encoder is configured to encode the radar data in the data set S1 to obtain a radar feature vector. The pulse wave data encoder is configured to encode the pulse wave data in the data set S1 to obtain a pulse wave feature vector. The radar data decoder is configured to decode the fusion feature data to obtain reconstructed radar data. The pulse wave data decoder is configured to decode the fusion feature data to obtain reconstructed pulse wave data.

[0055] As shown in Figures 2-4 , the encoder of the autoencoder network model M1, the teacher network model M2 and the dual-modal sleep monitoring model M4 can be divided into a radar data encoder and a pulse wave data encoder. The radar data encoder and the pulse wave data encoder can adopt a network structure of a convolutional neural network, a Transformer, etc. The outputs of the radar data encoder and the pulse wave data encoder have the same dimension (i.e., the radar data and the pulse wave data are mapped into the same dimensional feature space by the respective encoders), obtaining a radar feature vector and a pulse wave feature vector, and being sent to a fusion layer. The decoder in the autoencoder network model M1 can be divided into a radar data decoder and a pulse wave data decoder. The fusion feature data output by the fusion layer is sent to the radar data decoder and the pulse wave data decoder for decoding. The radar data decoder can adopt a structure symmetrical to the radar data encoder, or can use other structures. The output of the radar data decoder maintains the same data dimension as the input of the radar data encoder. The pulse wave data decoder can adopt a structure symmetrical to the pulse wave data encoder, or can use other structures. The output of the pulse wave data decoder maintains the same data dimension as the input of the pulse wave data encoder.

[0056] The radar data encoder and the pulse wave data encoder in the embodiment can independently process different modal data (such as radar data and pulse wave data), can efficiently extract single-modal discriminative features (such as body movement details and heart rate variability) according to the characteristics of the radar data and the pulse wave data by using special networks such as convolution or Transformer, and avoid information confusion; the two types of encoders output feature vectors of the same dimension, ensuring that the fusion layer can effectively integrate multi-modal information by cross-attention and the like, and avoid loss caused by dimension mismatch; the decoder adopts a symmetric or flexible structure with the encoder and outputs the same dimension as the original input, thereby ensuring that the M1 self-encoder network model M1 has precise double-modal feature extraction capability, the radar data encoder and the pulse wave data encoder extract single-modal key information (such as respiratory period and heart rate variability) according to the characteristics of the respective modal, and the decoder verifies the feature integrity through the “encoding-decoding” closed-loop task, and if the decoded data is highly consistent with the original input, it means that the feature retains the core information, and finally provides accurate and reliable feature basis for subsequent sleep staging, respiratory event detection and the like.

[0057] In some optional embodiments of the embodiment, the first identification layer in step S13 includes a first full connection layer and a first probability activation layer, the first full connection layer is used for linear transformation and non-linear mapping of the fusion feature data output by the teacher network model M2 to obtain a first classification feature vector matching the number of sleep staging and / or respiratory event categories, and the first probability activation layer is used for calculating the probability distribution of each sleep staging and / or respiratory event category based on the first classification feature vector to obtain the first sleep staging and / or respiratory event information.

[0058] As shown in FIG. 4, the double-modal sleep monitoring model M4 includes an encoder, a fusion layer and a first identification layer. The encoder is the teacher network model M2, and the fusion layer is used for fusing the feature data output by the teacher network model M2 to obtain the fusion feature data. Figure 4 As shown in FIG. 4, the double-modal sleep monitoring model M4 includes an encoder, a fusion layer and a first identification layer. The encoder is the teacher network model M2, and the fusion layer is used for fusing the feature data output by the teacher network model M2 to obtain the fusion feature data.

[0059] In some optional implementations of the embodiment, the second recognition layer in step S13 includes a second full connection layer and a second probability activation layer, the second full connection layer is configured to perform linear transformation and nonlinear mapping on the radar feature data output by the pure radar feature extractor M3 to obtain a second classification feature vector matching the number of sleep staging and / or respiratory event categories, and the second probability activation layer is configured to calculate the probability distribution of each sleep staging and / or respiratory event category based on the second classification feature vector to obtain the second sleep staging and / or respiratory event information.

[0060] As shown in Figure 4 The pure radar modality sleep monitoring model M5 includes a second recognition layer in addition to the trained pure radar feature extractor M3, the second recognition layer includes a second full connection layer and a second probability activation layer (softmax layer), and other more complex structures can also be used, the second recognition layer adjusts the feature dimension of the radar feature data through the second full connection layer to match the number of sleep staging categories or the number of respiratory event categories, and then calculates the probability distribution of each sleep staging or respiratory event through the second probability activation layer (such as the Softmax layer) to finally output the second sleep staging and / or respiratory event information. The second recognition layer efficiently completes the end-to-end mapping from the single modality radar feature to the sleep staging and / or respiratory event probability through the second full connection layer and the second probability activation layer. Although the input of the pure radar modality sleep monitoring model M5 only contains radar data, its core capability (such as the extraction of key features such as respiratory cycles and body movements) is derived from the knowledge transfer of the teacher network model M2 to the general feature representation related to sleep staging and / or respiratory events, the pure radar feature extractor M3 extracts single modality discriminative features highly related to the fused features of the teacher network model M2 by transferring this capability, and only uses radar data; and the second recognition layer further maps these radar features to the probability of a specific sleep staging or respiratory event, and finally outputs the second sleep staging and / or respiratory event information. This design not only avoids relying on pulse wave or PSG data, but also retains the discriminative ability of the multi-modal model through transfer learning, significantly improving the sleep staging and / or respiratory event monitoring accuracy and practicability of the pure radar model under single modality data.

[0061] The sleep monitoring model training method provided in the embodiment further includes: preprocessing the data set S1 and the data set S2 respectively, wherein the preprocessed radar data is a distance-time two-dimensional spectrum, including three channels of low-frequency energy spectrum, high-frequency energy spectrum and phase Doppler spectrum. The preprocessed pulse wave data is a frequency-time two-dimensional spectrum, including one channel of energy spectrum.

[0062] When training the teacher network model M2, the pure radar feature extractor M3, the bimodal sleep monitoring model M4, and the pure radar modal sleep monitoring model M5, preprocessing of the radar and pulse wave data in datasets S1 and S2 is required. Preprocessing involves converting the raw radar and pulse wave data into time-frequency domain spectrograms that are easily processed by the models: radar data is converted into a range-time two-dimensional spectrogram, specifically including three channels: low-frequency energy spectrum (respiratory rate), high-frequency energy spectrum (body movement), and phase Doppler spectrum (respiratory dynamics), used to capture sleep-related information such as respiratory cycle and body movement frequency; pulse wave data is converted into a frequency-time two-dimensional spectrogram, specifically including one channel of energy spectrum, used to reflect heart rate variability. After preprocessing, both types of data are formatted uniformly (time-frequency domain two-dimensional spectrograms), and both enhance the distinguishability of sleep stages and / or respiratory event-related features, providing higher-quality input for the bimodal sleep monitoring model and improving training efficiency and feature discriminative ability.

[0063] This embodiment converts the original training data into a two-dimensional spectrogram with joint time-frequency representation through preprocessing, which can effectively filter noise interference and enhance the discriminability of key features related to sleep stages and / or respiratory events (such as respiratory cycle and heart rate variability). This provides higher quality input for feature learning of the bimodal sleep monitoring model and the pure radar modal sleep monitoring model, thereby improving the discriminability of subsequent features and the efficiency of model training.

[0064] In some optional embodiments of this example, the specific process of transferring the feature extraction capability of the teacher network model M2 to the pure radar feature extractor M3 by fixing the parameters of the teacher network model M2 during training is as follows:

[0065] Step 1: Obtain the model parameters of the teacher network model.

[0066] The trained teacher network model has the ability to extract high-quality features from radar and pulse wave multimodal data. Obtaining its parameters (such as the encoder and the weights of the fusion layer) is to transfer this "multimodal feature extraction capability" as initial knowledge to the pure radar feature extractor M3, avoiding the inefficiency of training the pure radar feature extractor M3 from scratch.

[0067] Step 2: Use the teacher network model to extract features from the radar data and pulse wave data in dataset S1 to obtain fused features.

[0068] like Figure 2 As shown, a teacher network model is used to extract features from radar data and pulse wave data to obtain fused features. These features are obtained through joint training with multimodal data, providing a clear learning objective for the training of the pure radar feature extractor. That is, the pure radar feature extractor needs to extract single-modal features that are highly correlated with this multimodal feature from the single-modal radar data.

[0069] Step three, training the pure radar feature extractor M3 based on the radar data in the data set S1 with the model parameters to obtain target features.

[0070] The parameters (such as the weights of the encoder) of the teacher network model are used as initial values to initialize the pure radar feature extractor M3, and then the pure radar feature extractor M3 is trained to extract target features using only radar data by using the feature distillation method. The initialization function is to inherit the general feature extraction logic learned by the teacher network model M2, avoiding the inefficiency of training the pure radar feature extractor M3 from random parameters; training only with radar data forces the model to focus on the discriminative features of single-modal data, rather than relying on auxiliary information such as pulse waves.

[0071] Step four, calculate the mean square error loss of the fusion features and the target features, and complete the training of the pure radar feature extractor M3 when the mean square error loss converges to the third preset threshold.

[0072] The mean square error loss of the fusion features (multimodal features) output by the teacher network model and the target features (single-modal features) output by the pure radar feature extractor is calculated, and the parameters of the pure radar feature extractor are optimized through back propagation until the loss converges to the third preset threshold. This process aligns the single-modal features extracted by the pure radar feature extractor with the discriminative features of the multimodal model as much as possible, thereby inheriting the key discriminative logic of the teacher network model for sleep staging or respiratory events.

[0073] The present embodiment provides high-quality initialization for the pure radar feature extractor M3 by obtaining the multimodal feature extraction parameters of the trained teacher network model M2; then extracts multimodal fusion features as the learning target through the teacher network model M2, and clearly defines the target features that the pure radar feature extractor M3 needs to approximate; then initializes the pure radar feature extractor M3 with the parameters of the teacher network model, and trains it to extract single-modal target features using only radar data; and through the mean square error loss constraint, the single-modal features of the pure radar feature extractor M3 are aligned with the multimodal benchmark features, so that the pure radar feature extractor M3 can learn discriminative features highly consistent with the multimodal model using only radar data. This process does not require the collection of pulse wave data, significantly reducing the cost and device load of synchronous collection of multimodal data, while ensuring the high discriminative ability of the pure radar feature extractor M3 for sleep staging and / or respiratory events, significantly improving the practicality and generalizability of the single-modal monitoring scheme.

[0074] As shown in Figure 5 The present embodiment provides a sleep monitoring method based on a millimeter wave radar, which is executed by a computer or a server, and specifically includes the following steps:

[0075] S21, radar data of a target object is acquired.

[0076] In actual application, only the target object is monitored by the millimeter wave radar device for non-contact sleep monitoring, and radar data (such as respiratory frequency, body movement and other time sequence signals) in the sleep process of the target object is directly collected. Compared with the traditional contact device (such as PSG, portable sensor), the millimeter wave radar does not need to be attached to the human body, and the discomfort caused by the electrode contact is avoided, and it is more suitable for long-term, non-disturbing sleep monitoring scene.

[0077] S22, the radar data is subjected to feature recognition by using the pure radar modality sleep monitoring model M5 trained by the sleep monitoring model training method, to obtain target sleep staging and / or respiratory event information.

[0078] This step calls the trained pure radar modality sleep monitoring model M5 (composed of a pure radar feature extractor and a second recognition layer) to process the radar data acquired in step S21: first, the pure radar feature extractor extracts high-level semantic features (such as respiratory cycle stability, signal fluctuation rule caused by body movement and other single modality discriminative information) strongly related to sleep staging and / or respiratory events from the radar data; then, the second recognition layer maps the high-level semantic features to preset sleep staging labels (such as wakefulness, deep sleep, light sleep, REM period, etc.) or respiratory event labels (such as obstructive sleep apnea, central sleep apnea, hypopnea event, etc.), and finally outputs target sleep staging and / or respiratory event information (i.e. the predicted label of the model for the sleep staging or respiratory event of the current target object).

[0079] The sleep monitoring method based on millimeter wave radar provided by the present application only collects data by non-contact millimeter wave radar, avoids the physical discomfort of the target object caused by the traditional contact device, and is more suitable for long-term, comfortable sleep monitoring needs; and in the model training process, cross-modal knowledge transfer (auxiliary learning by using pulse wave data) is used to enhance the feature representation ability of the pure radar modality sleep monitoring model for single modality radar data, without relying on high-load pulse wave or PSG device, and only through non-contact millimeter wave radar, precise sleep monitoring can be realized, so that it can more accurately capture the key discriminative information of sleep staging and / or respiratory events, and finally realize efficient and reliable non-contact sleep monitoring.

[0080] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In a software embodiment, various software modules in accordance with embodiments of the present application are stored in a memory such as a computer program product (e.g., a disk storage) and executed by a computer processor. As such, various program modules in accordance with embodiments of the present application can comprise program instructions stored on computer-readable media, such as magnetic or optical disks, or memory such as an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), or flash memory. Program instructions can also be downloaded to a computer's memory from the Internet and / or another network. It is to be understood that the present application can be implemented in various forms of hardware, software, or a combination thereof, and that the present disclosure encompasses one or more of the following embodiments.

[0081] The present application is described in reference to the drawings using flow diagrams and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present application. It will be understood that each flow and / or block in the flow diagrams and / or block diagrams, and combinations of flows and / or blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams. Figure One one or more flows and / or blocks Figure One means for carrying out the functions specified in the flow diagram flow(s) and / or block(s).

[0082] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow diagrams and / or block diagrams. Figure One one or more flows and / or blocks Figure One means for carrying out the functions specified in the flow diagram flow(s) and / or block(s).

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow diagrams and / or block diagrams. Figure One one or more flows and / or blocks Figure One means for carrying out the functions specified in the flow diagram flow(s) and / or block(s).

[0084] Obviously, the embodiments described above are only examples and are not intended to limit the present application. Other variations and modifications of the embodiments disclosed can be apparent to those with skill in the art, and the present application is intended to include all such steps and features which fall within the scope of the embodiments. Accordingly, the disclosure of the present application is intended to be illustrative, but not limiting, of the scope of the present application, which is set forth in the following claims.

Claims

1. A method for training a sleep monitoring model, characterized in that, The method comprises the following steps: obtaining a data set S1 and a data set S2, wherein the data set S1 comprises radar data and pulse wave data collected synchronously by a millimeter wave radar device and a pulse sensor, the data set S2 comprises radar data, pulse wave data and PSG data collected synchronously by a millimeter wave radar device, a pulse sensor and a PSG device, and sleep staging labels and / or respiratory event labels labeled based on the PSG data, and the data quantity of the data set S2 is less than that of the data set S1; training a pure radar feature extractor M3 by using a pre-trained teacher network model M2 and the data set S1, wherein the training process comprises that the teacher network model M2 fuses the radar data and the pulse wave data in the data set S1 to output fusion feature data, and the pure radar feature extractor M3 extracts features from the radar data in the data set S1 to output radar feature data, and then calculates a loss according to the fusion feature data and the radar feature data; training a dual-modal sleep monitoring model M4 and a pure radar modal sleep monitoring model M5 by using the data set S2, wherein the dual-modal sleep monitoring model M4 comprises the trained teacher network model M2 and a first recognition layer, the pure radar modal sleep monitoring model M5 comprises the trained pure radar feature extractor M3 and a second recognition layer, and the training process comprises that the first recognition layer determines first sleep staging and / or respiratory event information according to the fusion feature data output by the teacher network model M2, the second recognition layer determines second sleep staging and / or respiratory event information according to the radar feature data output by the pure radar feature extractor M3, and a loss is calculated according to the first sleep staging and / or respiratory event information, the second sleep staging and / or respiratory event information, the sleep staging labels and / or respiratory event labels.

2. The sleep monitoring model training method of claim 1, wherein, The pre-trained teacher network model M2 is obtained in the following manner: training an auto-encoder network model M1 by using an unsupervised learning method and the data set S1, wherein the auto-encoder network model M1 comprises an encoder, a fusion layer and a decoder, the training process comprises that the encoder encodes the radar data and the pulse wave data in the data set S1 respectively to convert them into radar feature vectors and pulse wave feature vectors of the same dimension, the fusion layer fuses the radar feature vectors and the pulse wave feature vectors to obtain fusion feature data, the decoder decodes the fusion feature data to obtain reconstructed radar data and reconstructed pulse wave data, a loss is calculated by using the radar data, the pulse wave data, the reconstructed radar data and the reconstructed pulse wave data, and the encoder and the fusion layer in the trained auto-encoder network model M1 constitute the pre-trained teacher network model M2.

3. The sleep monitoring model training method of claim 2, wherein, The radar data, the pulse wave data, the reconstructed radar data and the reconstructed pulse wave data are utilized to compute a loss in the following way : ； wherein, denotes radar data, denotes pulse wave data, denotes reconstructed radar data, and denotes reconstructed pulse wave data, denotes reconstructed radar data, and denotes reconstructed pulse wave data, denotes reconstructed pulse wave data, denotes reconstructed pulse wave data, , , is a weighting factor, denotes a mean squared error loss of and , denotes a mean squared error loss of and , denotes a mean squared error loss of and .

4. The sleep monitoring model training method of claim 2, wherein, The encoder comprises a radar data encoder and a pulse wave data encoder, and the decoder comprises a radar data decoder and a pulse wave data decoder, the radar data encoder is used for encoding radar data in the data set S1 to obtain the radar feature vector, the pulse wave data encoder is used for encoding pulse wave data in the data set S1 to obtain the pulse wave feature vector, the radar data decoder is used for decoding the fusion feature data to obtain the reconstructed radar data, and the pulse wave data decoder is used for decoding the fusion feature data to obtain the reconstructed pulse wave data.

5. The sleep monitoring model training method of claim 1, wherein, The first identification layer comprises a first full connection layer and a first probability activation layer, the first full connection layer is used for linear transformation and nonlinear mapping of the fusion feature data output by the teacher network model M2 to obtain a first classification feature vector matched with the number of sleep staging and / or respiratory event categories, and the first probability activation layer is used for calculating the probability distribution of each sleep staging and / or respiratory event category based on the first classification feature vector to obtain the first sleep staging and / or respiratory event information.

6. The sleep monitoring model training method of claim 1, wherein, The second identification layer comprises a second full connection layer and a second probability activation layer, the second full connection layer is used for linear transformation and nonlinear mapping of the radar feature data output by the pure radar feature extractor M3 to obtain a second classification feature vector matched with the number of sleep staging and / or respiratory event categories, and the second probability activation layer is used for calculating the probability distribution of each sleep staging and / or respiratory event category based on the second classification feature vector to obtain the second sleep staging and / or respiratory event information.

7. The method of claim 1, wherein, Further comprising: The data set S1 and the data set S2 are respectively preprocessed, wherein the preprocessed radar data is a distance-time two-dimensional spectrum, containing three channels of low-frequency energy spectrum, high-frequency energy spectrum and phase Doppler spectrum, and the preprocessed pulse wave data is a frequency-time two-dimensional spectrum, containing one channel of energy spectrum.

8. A millimeter wave radar-based sleep monitoring method, characterized by, Comprising: Obtaining radar data of a target object; The pure radar modality sleep monitoring model M5 trained by the method of any one of claims 1-7 is used for feature recognition on the radar data to obtain target sleep staging and / or respiratory event information. 9.A sleep monitoring model training device, comprising: Comprising: A processor and a memory connected with the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor to enable the processor to execute the sleep monitoring model training method according to any one of claims 1-7.

10. A millimeter wave radar based sleep monitoring device, characterized by, Comprising: A processor and a memory connected with the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor to enable the processor to execute the sleep monitoring method based on millimeter wave radar according to claim 8.

Images