Heart rate and respiration rate millimeter wave radar detection method based on CNN fusion features

Abstract

The present invention provides a heart rate and respiratory rate millimeter wave radar detection method based on CNN fusion features, comprising: obtaining a real-time one-dimensional radar signal of a target person, wherein the one-dimensional radar signal is obtained by monitoring the vital signs of the target person through a millimeter wave radar sensor; performing time-frequency analysis based on the one-dimensional radar signal to obtain a two-dimensional time-frequency signal; based on the one-dimensional radar signal and the two-dimensional time-frequency signal, performing human heart rate and respiratory rate detection on the target person to obtain the heart rate and respiratory rate of the target person, overcoming the problems of long-term monitoring inconvenience, discomfort, and limited application on specific populations in traditional contact detection methods, realizing comprehensive and accurate human heart rate and respiratory rate detection through non-invasive monitoring means, and being particularly beneficial to the elderly, patients with chronic diseases, and individuals who need long-term rehabilitation monitoring, providing support for home medical care and remote health monitoring.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a heart rate and respiratory rate millimeter wave radar detection method based on CNN fusion features. Background Art

[0002] In the field of medical health, heart rate and respiratory rate have always been regarded as important parameters for evaluating human physiological status and health information. They not only directly reflect the quality of human heart and lung functions, but also their stability has important guiding significance for disease prevention. Therefore, accurate and real-time detection of human heart rate and respiratory rate is crucial to maintaining human health.

[0003] At present, the detection of human heart rate and respiratory rate usually relies on contact sensing devices, such as electrocardiographs, breathing belts, etc. These methods require users to wear electrodes or other sensing devices, which not only restricts the user's daily activities, but also may cause discomfort to the user, especially in long-term monitoring. In addition, for certain special groups, such as newborns, burn patients, skin allergies, infectious disease patients, etc., the current contact detection methods may have the risk of secondary injury or infection, and are not applicable and have a narrow scope of application. Summary of the invention

[0004] The present invention provides a heart rate and respiratory rate millimeter wave radar detection method based on CNN fusion features, which is used to solve the defects of the contact detection method in the prior art that it restricts user activities, easily causes discomfort, and is limited in application to specific groups of people. It can realize fast and accurate human heart rate and respiratory rate detection and improve the accuracy and reliability of the detection results.

[0005] The present invention provides a heart rate and respiratory rate millimeter wave radar detection method based on CNN fusion features, comprising:

[0006] Acquire a real-time one-dimensional radar signal of a target person, wherein the one-dimensional radar signal is obtained by monitoring the vital signs of the target person through a millimeter-wave radar sensor;

[0007] Performing time-frequency analysis based on the one-dimensional radar signal to obtain a two-dimensional time-frequency signal;

[0008] Based on the one-dimensional radar signal and the two-dimensional time-frequency signal, the human heart rate and respiratory rate of the target person are detected to obtain the heart rate and respiratory rate of the target person.

[0009] According to a method for detecting heart rate and respiration rate by millimeter wave radar based on CNN fusion features provided by the present invention, based on the one-dimensional radar signal and the two-dimensional time-frequency signal, the heart rate and respiration rate of the target person are detected to obtain the heart rate and respiration rate of the target person, including:

[0010] Inputting the one-dimensional radar signal and the two-dimensional time-frequency signal into a vital sign detection model to obtain a heart rate and a respiratory rate of a target person output by the vital sign detection model;

[0011] The vital sign detection model is obtained by training the sample one-dimensional radar signal of the sample personnel and their heart rate label and respiratory rate label based on the convolutional neural network;

[0012] The vital sign detection model is used to detect the human heart rate and respiratory rate of the target person based on the time series characteristics of the one-dimensional radar signal and the frequency component characteristics of the two-dimensional time-frequency signal to obtain the heart rate and respiratory rate of the target person.

[0013] According to a method for detecting heart rate and respiratory rate by millimeter wave radar based on CNN fusion features provided by the present invention, the vital sign detection model includes a feature extraction network, a feature fusion network and a perception detection network;

[0014] The feature extraction network is used to extract features from the one-dimensional radar signal and the two-dimensional time-frequency signal to obtain time series features of the one-dimensional radar signal and frequency component features of the two-dimensional time-frequency signal;

[0015] The feature fusion network is used to perform feature fusion on the time series feature and the frequency component feature based on the correlation between the time series feature and the frequency component feature to obtain a fused feature;

[0016] The perception detection network is used to detect the human heart rate and respiratory rate of the target person based on the fusion features to obtain the heart rate and respiratory rate of the target person.

[0017] According to a method for detecting heart rate and respiratory rate by millimeter wave radar based on CNN fusion features provided by the present invention, the method performs time-frequency analysis based on the one-dimensional radar signal to obtain a two-dimensional time-frequency signal, including:

[0018] Performing clutter separation on the one-dimensional radar signal to obtain a target radar signal;

[0019] Performing time-frequency analysis on the target radar signal to obtain a two-dimensional time-frequency signal.

[0020] According to a method for detecting heart rate and respiratory rate by millimeter wave radar based on CNN fusion features provided by the present invention, the time-frequency analysis of the target radar signal is performed to obtain a two-dimensional time-frequency signal, including:

[0021] Performing wavelet transform on the target radar signal to obtain a wavelet spectrum diagram;

[0022] Phase extraction is performed based on the wavelet spectrum to obtain a two-dimensional time-frequency signal.

[0023] According to a method for detecting heart rate and respiratory rate by millimeter wave radar based on CNN fusion features provided by the present invention, the one-dimensional radar signal is subjected to clutter separation to obtain a target radar signal, including:

[0024] Acquire a background signal of a target area where the target person is located, wherein the background signal is obtained by monitoring the target area through a millimeter wave radar sensor when no one is in the target area;

[0025] Performing median calculation based on the background signal to obtain background clutter corresponding to the target area, and performing background subtraction on the one-dimensional radar signal based on the background clutter;

[0026] Artifacts are filtered out based on the radar signal after background subtraction to obtain the target radar signal.

[0027] The present invention also provides a heart rate and respiratory rate millimeter wave radar detection device based on CNN fusion features, comprising:

[0028] An acquisition unit, used for acquiring a real-time one-dimensional radar signal of a target person, wherein the one-dimensional radar signal is obtained by monitoring the vital signs of the target person through a millimeter-wave radar sensor;

[0029] An analysis unit, configured to perform time-frequency analysis based on the one-dimensional radar signal to obtain a two-dimensional time-frequency signal;

[0030] The detection unit is used to detect the human heart rate and respiratory rate of the target person based on the one-dimensional radar signal and the two-dimensional time-frequency signal to obtain the heart rate and respiratory rate of the target person.

[0031] The present invention also provides a heart rate and respiratory rate millimeter wave radar detection system based on CNN fusion features, comprising: a millimeter wave radar sensor and a processor;

[0032] The processor is used to obtain a real-time one-dimensional radar signal of the target person obtained by the millimeter-wave radar sensor for monitoring the vital signs of the target person, and perform time-frequency analysis based on the one-dimensional radar signal to obtain a two-dimensional time-frequency signal; based on the one-dimensional radar signal and the two-dimensional time-frequency signal, perform human heart rate and respiratory rate detection on the target person to obtain the heart rate and respiratory rate of the target person.

[0033] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a heart rate and respiratory rate millimeter wave radar detection method based on CNN fusion features as described in any one of the above is implemented.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for detecting heart rate and respiratory rate by millimeter-wave radar based on CNN fusion features.

[0035] The heart rate and respiratory rate millimeter wave radar detection method based on CNN fusion features provided by the present invention uses the real-time one-dimensional radar signal obtained by monitoring the vital signs of a target person through a millimeter wave radar sensor to perform time-frequency analysis to obtain a two-dimensional time-frequency signal; based on the one-dimensional radar signal and the two-dimensional time-frequency signal, the human heart rate and respiratory rate of the target person are detected, and the key information of the human vital signs contained in the multi-dimensional signal is used to realize the detection of the vital signs of the target person, and obtain accurate and reliable heart rate and respiratory rate, while overcoming the problems of long-term monitoring inconvenience, discomfort, and limited application on specific populations existing in traditional contact detection methods, and realizing comprehensive and accurate human heart rate and respiratory rate detection through non-invasive monitoring means, and it is particularly beneficial for the elderly, patients with chronic diseases and individuals who need long-term rehabilitation monitoring, and provides support for home medical care and remote health monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 It is a flow chart of a method for detecting heart rate and respiratory rate by millimeter wave radar based on CNN fusion features provided by the present invention;

[0038] Figure 2 It is a flow chart of a process of detecting a human heart rate and a respiratory rate based on a vital sign detection model provided by the present invention;

[0039] Figure 3 It is a structural schematic diagram of a heart rate and respiratory rate millimeter wave radar detection device based on CNN fusion features provided by the present invention;

[0040] Figure 4 It is a structural schematic diagram of a heart rate and respiratory rate millimeter wave radar detection system based on CNN fusion features provided by the present invention;

[0041] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Heart rate (HR) and respiratory rate (RR) are two important parameters that can reflect the physiological state and health information of the human body. They play an important role in the assessment of human health status. They can not only reflect the good or bad heart and lung functions of the human body, but their stability also has important guiding significance for disease prevention.

[0044] At present, human heart rate and respiratory rate detection mostly rely on contact sensing devices, such as electrocardiographs, breathing belts, etc. These methods require users to wear electrodes or other sensing devices, which not only restricts the user's daily activities, but also may cause discomfort to the user, especially in long-term monitoring, which is particularly inconvenient. In addition, for certain special populations, such as newborns, burn patients, skin allergies, infectious disease patients, etc., contact detection methods may cause secondary injuries or infection risks, so they are not suitable for special populations and have a narrow scope of application.

[0045] Based on this, non-contact detection methods have emerged, which solve the problem of user compliance in medical health monitoring, especially in long-term monitoring. Specifically, it avoids wearing electrodes or other sensing devices, reduces user discomfort, and improves monitoring convenience.

[0046] In this regard, the present invention provides a heart rate and respiratory rate millimeter wave radar detection method based on CNN fusion features, which aims to monitor the vital signs of a target person through a non-contact monitoring method, and detect the human heart rate and respiratory rate through the one-dimensional radar signal obtained by monitoring and the two-dimensional signal obtained by analysis on this basis, so as to realize the detection of the vital signs of the target person through the key information of the human vital signs contained in the signals of multiple dimensions, and obtain accurate and reliable human heart rate and respiratory rate, while avoiding the inconvenience of long-term monitoring, discomfort, and limited application to specific populations existing in traditional contact detection methods. The non-invasive monitoring method is convenient and fast, and is particularly beneficial to the elderly, patients with chronic diseases, and individuals who need long-term rehabilitation monitoring, laying the foundation for home medical care and remote health monitoring, and can be used as an important tool for health monitoring.

[0047] Figure 1: is a flow chart of the heart rate and respiratory rate millimeter wave radar detection method based on CNN fusion features provided by the present invention, such as Figure 1 As shown, the method includes:

[0048] Step 110, obtaining a real-time one-dimensional radar signal of the target person, where the one-dimensional radar signal is obtained by monitoring the vital signs of the target person through a millimeter-wave radar sensor;

[0049] Step 120, performing time-frequency analysis based on the one-dimensional radar signal to obtain a two-dimensional time-frequency signal;

[0050] Step 130 , based on the one-dimensional radar signal and the two-dimensional time-frequency signal, the heart rate and respiratory rate of the target person are detected to obtain the heart rate and respiratory rate of the target person.

[0051] Specifically, taking into account the defects of contact-based detection methods, such as restricting user activities, causing discomfort, and having limited application to specific groups of people, in an embodiment of the present invention, it is proposed to perform vital sign detection through a non-contact detection method. Compared with the contact-based detection method, the non-contact detection uses optical, radar, or infrared technologies to capture the physiological signals of the human body without contacting the human body, thereby realizing real-time monitoring of heart rate and respiratory rate. It not only avoids the inconvenience and discomfort of users wearing electrodes or other sensing devices, but is also suitable for special groups of people who cannot use traditional monitoring methods, and has a wide range of applications.

[0052] Based on this, in an embodiment of the present invention, before performing human heart rate and respiratory rate detection, it is also necessary to determine the person whose heart rate and respiratory rate are to be detected, that is, the target person, and the vital signs data of the target person, which are obtained by non-contact monitoring.

[0053] Compared with traditional contact detection, non-contact monitoring has significant advantages. First, non-contact monitoring avoids the discomfort that may be caused by contact sensing equipment and improves the user's monitoring compliance, which is especially valuable in long-term monitoring. Secondly, non-contact monitoring is suitable for various special populations, such as newborns, burn patients, skin allergies, infectious disease patients, etc., avoiding the risk of secondary injury or infection caused by traditional monitoring methods. Finally, non-contact monitoring is continuous and can continuously monitor the user's health status. For patients with chronic diseases such as cardiovascular disease and respiratory problems, it can track changes in the condition in a timely manner, detect abnormalities in a timely manner and respond.

[0054] As one of the non-contact monitoring devices, the millimeter wave radar sensor is used in the electromagnetic wave frequency of the millimeter wave band. It can emit electromagnetic waves to the target and receive the electromagnetic waves reflected by the target. When these electromagnetic waves encounter a moving target, the frequency changes due to the Doppler effect. By analyzing the Doppler frequency shift of the electromagnetic waves emitted by the millimeter wave radar sensor and the reflected electromagnetic waves, the speed and direction of movement of the target can be identified and determined. When the target is a human body, it can also detect the subtle vibrations of the chest wall in the human chest cavity. These vibrations are closely related to the chest wall fluctuations caused by the heartbeat and breathing of the human body. Therefore, the radar signal monitored by it can accurately reflect the vital signs of the human body.

[0055] In view of this, in an embodiment of the present invention, this millimeter wave radar sensor can be introduced in the process of detecting the human heart rate and respiratory rate, so as to monitor vital signs and obtain the vital sign data of the target person; that is, the vital signs of the target person can be monitored by the millimeter wave radar sensor, and specifically, a non-contact frequency modulated continuous wave (Frequency Modulated Continuous Wave, FMCW) radar sensor can be used to continuously monitor the physiological movement of the chest wall of the target person to obtain the vital sign data of the target person, and the data is presented by a dynamic wave, so it is also called a one-dimensional radar signal. By monitoring the vital signs of the target person through the millimeter wave radar sensor, a real-time one-dimensional radar signal of the target person can be obtained.

[0056] Furthermore, after obtaining the one-dimensional radar signal of the target person, considering that the information contained in the one-dimensional signal is relatively simple, it only provides a one-dimensional time series signal related to the vital signs in the vital signs detection. Although this signal captures the dynamic changes of the time series, it may miss the frequency information, resulting in the subtle changes in the human heart beat and breathing during the vital signs detection process. It is impossible to accurately capture, thereby reducing the precision and accuracy of human heart rate and respiratory rate detection in complex environments.

[0057] Therefore, in the embodiment of the present invention, after obtaining the one-dimensional radar signal, it is necessary to further analyze it to extract comprehensive information in the time-frequency domain related to the heart rate and the respiratory rate, so as to obtain a signal containing information in both time and frequency, that is, a two-dimensional time-frequency signal. Here, specifically, the one-dimensional radar signal can be subjected to time-frequency analysis to analyze the frequency characteristics of the one-dimensional radar signal that vary with time, and to extract the time-frequency domain information, so as to obtain a two-dimensional time-frequency signal.

[0058] Among them, the time-frequency analysis of one-dimensional radar signals can be achieved with the help of time-frequency analysis methods, such as short-time Fourier transform, wavelet transform, Wigner-Ville distribution, etc. The advantage of such analysis methods compared to fast Fourier transform is that they can analyze signals in both time and frequency dimensions at the same time, thereby revealing the changing characteristics of the signal at different times and frequencies, and providing information in both time and frequency dimensions at the same time; while the fast Fourier transform only provides frequency information, therefore, it is not suitable for processing one-dimensional radar signals that change with time, and cannot obtain comprehensive information in the time and frequency domains.

[0059] Furthermore, after obtaining the two-dimensional time-frequency signal, the embodiment of the present invention can combine the one-dimensional radar signal and the two-dimensional time-frequency signal to detect the human heart rate and respiratory rate of the target person, so as to obtain the heart rate and respiratory rate of the target person. That is, based on the one-dimensional radar signal and the two-dimensional time-frequency signal, the human heart rate and respiratory rate can be detected in real time through the rich time and frequency information related to the vital signs of the human body contained in the one-dimensional radar signal and the two-dimensional time-frequency signal, so as to obtain the heart rate and respiratory rate of the target person.

[0060] In an embodiment of the present invention, a one-dimensional radar signal and a two-dimensional time-frequency signal are used in combination in the process of detecting human heart rate and respiratory rate, and features in the one-dimensional radar signal and features in the two-dimensional time-frequency signal are extracted simultaneously for detection. Compared with the information extraction of a single type of signal, the detection method can obtain richer and more comprehensive information. The addition of the two-dimensional time-frequency signal is crucial for vital sign detection, which contains comprehensive information in the time-frequency domain, making up for the missing frequency information in the one-dimensional radar signal, thereby making the detection more comprehensive and accurate.

[0061] The heart rate and respiratory rate millimeter wave radar detection method based on CNN fusion features provided by the present invention uses the real-time one-dimensional radar signal obtained by monitoring the vital signs of a target person through a millimeter wave radar sensor to perform time-frequency analysis to obtain a two-dimensional time-frequency signal; based on the one-dimensional radar signal and the two-dimensional time-frequency signal, the human heart rate and respiratory rate of the target person are detected, and the key information of the human vital signs contained in the multi-dimensional signal is used to realize the detection of the vital signs of the target person, and obtain accurate and reliable heart rate and respiratory rate, while overcoming the problems of long-term monitoring inconvenience, discomfort, and limited application on specific populations existing in traditional contact detection methods, and realizing comprehensive and accurate human heart rate and respiratory rate detection through non-invasive monitoring means, and it is particularly beneficial for the elderly, patients with chronic diseases and individuals who need long-term rehabilitation monitoring, and provides support for home medical care and remote health monitoring.

[0062] Based on the above embodiment, step 130 includes:

[0063] The one-dimensional radar signal and the two-dimensional time-frequency signal are input into the vital sign detection model to obtain the heart rate and respiratory rate of the target person output by the vital sign detection model;

[0064] The vital sign detection model is trained on the basis of convolutional neural network using sample one-dimensional radar signals of sample personnel and their heart rate labels and respiratory rate labels;

[0065] The vital signs detection model is used to detect the heart rate and respiratory rate of the target person based on the time series characteristics of the one-dimensional radar signal and the frequency component characteristics of the two-dimensional time-frequency signal to obtain the heart rate and respiratory rate of the target person.

[0066] Specifically, in step 130, the process of detecting the heart rate and respiratory rate of the target person according to the one-dimensional radar signal and the two-dimensional time-frequency signal to obtain the heart rate and respiratory rate of the target person can be implemented by a vital sign detection model, and the specific process may include:

[0067] First, the one-dimensional radar signal and the two-dimensional time-frequency signal can be used as input and input into the vital signs detection model. Then, the vital signs detection model detects the human heart rate and respiratory rate based on the input one-dimensional radar signal and the two-dimensional time-frequency signal. Finally, the heart rate and respiratory rate of the target person can be obtained as output by the vital signs detection model.

[0068] Specifically, when performing human heart rate and respiratory rate detection, the information related to human vital signs contained in the multi-dimensional signal is first extracted to obtain the corresponding features, and then the human heart rate and respiratory rate detection is carried out based on this feature to obtain the heart rate and respiratory rate of the target person.

[0069] That is, after the one-dimensional radar signal and the two-dimensional time-frequency signal are input into the vital signs detection model, the vital signs detection model can first perform feature extraction on the one-dimensional radar signal and the two-dimensional time-frequency signal to extract the time series image information in the one-dimensional radar signal and the time-frequency domain information in the two-dimensional time-frequency signal, thereby obtaining the time series characteristics of the one-dimensional radar signal and the frequency component characteristics of the two-dimensional time-frequency signal. Then, the human heart rate and respiratory rate of the target person can be detected based on the time series characteristics and frequency component characteristics, so as to detect the heart rate and respiratory rate of the target person through comprehensive key information in time and frequency.

[0070] However, it should be noted that before inputting the one-dimensional radar signal and the two-dimensional time-frequency signal into the vital sign detection model, it is necessary to pre-train a vital sign detection model with accurate detection performance and capable of accurately and reliably detecting the human heart rate and respiratory rate. That is, the sample one-dimensional radar signal of the sample person, as well as the heart rate label and respiratory rate label of the sample person, can be used for training to obtain the vital sign detection model.

[0071] Here, in the training process of the vital sign detection model, for the initial model in the training stage, considering that the shallow convolutional layer in CNN (Convolutional Neural Networks) has a strong ability to represent geometric information, accurate feature extraction and feature fusion can be performed. Therefore, in the embodiment of the present invention, the initial model can be constructed based on CNN.

[0072] Based on the above embodiment, the vital sign detection model includes a feature extraction network, a feature fusion network and a perception detection network;

[0073] The feature extraction network is used to extract features from the one-dimensional radar signal and the two-dimensional time-frequency signal to obtain the time series features of the one-dimensional radar signal and the frequency component features of the two-dimensional time-frequency signal;

[0074] The feature fusion network is used to fuse the time series features and the frequency component features based on the correlation between the time series features and the frequency component features to obtain the fused features;

[0075] The perception detection network is used to detect the heart rate and respiratory rate of the target person based on the fusion features to obtain the heart rate and respiratory rate of the target person.

[0076] Specifically, Figure 2 is a flow chart of the process of detecting the human heart rate and respiratory rate based on the vital signs detection model provided by the present invention, such as Figure 2 As shown, the vital signs detection model takes the one-dimensional radar signal and the two-dimensional time-frequency signal obtained on this basis as input, detects the human heart rate and respiratory rate based on them, and outputs the heart rate and respiratory rate of the target person.

[0077] Specifically, the vital sign detection model includes a feature extraction network, a feature fusion network, and a perception detection network; specifically, when performing heart rate and respiratory rate detection, the feature extraction network takes a one-dimensional radar signal and a two-dimensional time-frequency signal as input, and is used to extract features from the input one-dimensional radar signal and the two-dimensional time-frequency signal to obtain the time series features of the one-dimensional radar signal and the frequency component features of the two-dimensional time-frequency signal. Among them, the feature extraction network includes a convolution layer and a pooling layer. The convolution layer is used to perform a convolution operation on the input signal to achieve feature extraction, and the pooling layer is used to find the peak value of the output of the convolution layer, which corresponds to the maximum value of the convolution result.

[0078] Correspondingly, during the model training process, the convolution operation can be understood as calculating the matching degree between the current part of the input signal and the convolution kernel in the convolution layer. The convolution kernel can be understood as a template. The convolution operation of the convolution layer is template extraction. The goal of the convolution layer is to learn a convolution kernel as a template fragment; the goal of the pooling layer is to find the peak value of the convolution layer output, which corresponds to the maximum value of the convolution result. Because this maximum value appears when the template is aligned with a single heartbeat or breath, its number can indicate the number of heartbeats or breaths. The optimization goal during the training process is to minimize the signal ∧ corresponding to the same input time of the template T and the input signal clipped according to the template length i , that is, the error e(i).

[0079] e(i)=|D i -T| 2 =|D i | 2 +|T| 2 -2|D i *T|

[0080] After learning the template segment T that minimizes e(i), T can be input back into the network to find the location of a single heartbeat or breath within a specified time in the cropped signal, and then calculate the heart rate and respiratory rate.

[0081] The feature fusion network takes time series features and frequency component features as input and is used to fuse the two features. Specifically, in order to make the model focus on information related to human vital signs and ignore irrelevant information, an attention mechanism is introduced to perform feature fusion based on the attention mechanism. Specifically, the correlation between time series features and frequency component features is used as a benchmark to fuse the time series features and frequency component features, thereby obtaining the fused features, i.e., the fused features.

[0082] The perception detection network uses the fused features as input, and is used to detect the heart rate and respiratory rate of the target person based on the fused features to obtain the heart rate and respiratory rate of the target person. Here, the perception detection network is built on the basis of MLP (Multi-layer Perceptron). The hierarchical structure and nonlinear ability of MLP enable it to accurately capture the complex patterns in the training data during the training process, so that the model has good detection capabilities and can accurately detect the heart rate and respiratory rate of the human body.

[0083] It should be noted that, based on the architecture of the vital signs detection model in the embodiment of the present invention, when performing human heart rate and respiratory rate detection in the training phase, the one-dimensional radar signal and the two-dimensional time-frequency signal are used in combination, and the mean square error obtained is significantly lower than that of using only a single one-dimensional radar signal or a two-dimensional time-frequency signal, which shows that the simultaneous extraction of the time series characteristics of the one-dimensional radar signal and the frequency component characteristics in the two-dimensional time-frequency signal can obtain richer and more comprehensive information than a single type of signal extraction method, because the frequency component information in the two-dimensional time-frequency signal is very critical for estimating human vital signs (such as heart rate and respiratory rate), which makes up for the missing frequency information in the one-dimensional radar signal, and these frequency information can be effectively captured in the frequency analysis of the two-dimensional time-frequency signal. Therefore, the fusion method in the embodiment of the present invention not only enhances the sensitivity of the model to subtle changes in the signal, but also improves the ability to accurately estimate vital signs in a complex environment, and can detect accurate and reliable human heart rate and respiratory rate.

[0084] Based on the above embodiment, step 120 includes:

[0085] Perform clutter separation on the one-dimensional radar signal to obtain the target radar signal;

[0086] Perform time-frequency analysis on the target radar signal to obtain a two-dimensional time-frequency signal.

[0087] Specifically, in step 120, the process of performing time-frequency analysis on the one-dimensional radar signal to obtain the two-dimensional time-frequency signal may specifically include:

[0088] It is understandable that when performing vital sign detection through millimeter-wave radar sensors to obtain real-time one-dimensional radar signals of target persons, it is necessary to consider that vital sign detection in indoor environments will be interfered by many static elements in the environment, such as walls, furniture, floors, various fixed objects, etc., which will affect the monitoring results.

[0089] In an indoor environment, these static elements constitute the cluttered part of the background, and the signals reflected back to the millimeter wave radar sensor appear as a static clutter, which is usually called a DC component in signal processing. These DC components appear as continuous and stable signals in the radar signal, which are significantly different from the signals generated by dynamic objects, such as human physiological characteristics. Therefore, in order to effectively detect human physiological activities and extract human vital signs information, such as breathing, heartbeat, etc., in the embodiment of the present invention, these static noises need to be removed from the monitored radar signal to ensure the accuracy and reliability of subsequent vital signs detection.

[0090] Specifically, after obtaining the real-time one-dimensional radar signal of the target person, the one-dimensional radar signal can first be clutter separated to remove the static clutter therein, to ensure the accuracy and reliability of the vital sign data, thereby obtaining the radar signal after clutter separation, that is, the target radar signal. That is, the noise can be eliminated by background subtraction and filtering, and the DC component can be removed, thereby achieving the purpose of effective denoising and removing other interference.

[0091] Then, the target radar signal can be subjected to time-frequency analysis to obtain a two-dimensional time-frequency signal; that is, the target radar signal can be simultaneously analyzed in both time and frequency dimensions with the help of time-frequency analysis methods, such as short-time Fourier transform, wavelet transform, Wigner-Ville distribution, etc., to determine the changing characteristics of the target radar signal at different times and frequencies, thereby extracting comprehensive time-frequency domain information related to heart rate and respiratory rate, and obtaining a signal that contains information on both time and frequency, that is, a two-dimensional time-frequency signal.

[0092] Based on the above embodiment, time-frequency analysis is performed on the target radar signal to obtain a two-dimensional time-frequency signal, including:

[0093] Perform wavelet transform on the target radar signal to obtain the wavelet spectrum diagram;

[0094] Phase extraction is performed based on the wavelet spectrum to obtain a two-dimensional time-frequency signal.

[0095] Specifically, the process of performing time-frequency analysis on the target radar signal to obtain a two-dimensional time-frequency signal may include:

[0096] Firstly, the target radar signal is subjected to wavelet transform to extract the comprehensive time-frequency domain information related to the heart rate and respiratory rate through wavelet transform, thereby obtaining a wavelet spectrum diagram containing the time-frequency domain information of the vital signs of the target person.

[0097] Here, in the process of time-frequency analysis, wavelet transform is used to analyze the frequency characteristics of non-stable signals (i.e., target radar signals whose statistical characteristics change over time) that change over time, and a wavelet spectrum containing time-frequency domain information can be obtained. The advantage of wavelet transform over fast Fourier transform is that it can provide information on both time and frequency; while fast Fourier transform only provides frequency information, so it is not suitable for processing real-time changing signals of heart rate and respiratory rate.

[0098] Then, the two-dimensional time-frequency signal can be determined based on the wavelet spectrum, that is, the phase extraction can be performed based on the wavelet spectrum to extract the time and frequency information, thereby obtaining the two-dimensional time-frequency signal.

[0099] During the training of the model, the sample one-dimensional radar signal of the sample personnel can be subjected to continuous wavelet transform, specifically, the Morse wavelet can be used for wavelet transform, and the frequency limit is set to 0-4Hz to focus on the frequency range of vital signs, and the window size is set to 12s to include at least 2 to 3 breathing cycles, and the window is segmented by moving 1s. After data segmentation, a data set is formed for initial model training and testing. When configuring the training data set, segmentation is performed based on the part without artifacts to prevent artifacts from causing erroneous learning of the model.

[0100] Based on the above embodiment, clutter separation is performed on the one-dimensional radar signal to obtain the target radar signal, including:

[0101] Obtain the background signal of the target area where the target person is located. The background signal is obtained by monitoring the target area through the millimeter wave radar sensor when there is no one in the target area;

[0102] Perform median calculation based on the background signal to obtain the background clutter corresponding to the target area, and perform background subtraction on the one-dimensional radar signal based on the background clutter;

[0103] Artifacts are filtered out based on the radar signal after background subtraction to obtain the target radar signal.

[0104] Specifically, the above process of performing clutter separation on the one-dimensional radar signal to obtain the target radar signal may include the following two steps:

[0105] The first step is to collect signals through the millimeter-wave radar sensor when there is no one in the detection area of ​​the millimeter-wave radar sensor, that is, when there is no one in the target area where the target person is located. The collected radar signal is the background signal of the target area. Then, the median calculation can be performed through multiple frames of static background signals to obtain the median of the background signal, which is used as the background clutter corresponding to the target area. After that, the background clutter can be subtracted from the real-time one-dimensional radar signal, that is, the one-dimensional radar signal is background subtracted based on the background clutter to eliminate the influence of the static background. The formula is as follows:

[0106] R h (t) = R(t) - R m (t)

[0107] Among them, R h (t) is the signal after background subtraction, R(t) is the one-dimensional radar signal, R m (t) is the median of multiple frames of static background signals, i.e., background clutter.

[0108] In the second step, the radar signal after background subtraction is filtered to remove unnecessary artifact components other than heart rate and respiratory rate. That is, since the respiratory rate of a healthy human body at rest is 12-20 times per minute, corresponding to a frequency range of 0.2-0.4 Hz, and the resting heart rate is 60-100 times per minute, corresponding to a frequency range of 1.0-1.7 Hz, an IIR (Infinite Impulse Response) bandpass filter can be used here to filter the radar signal after background subtraction to filter out the remaining clutter, thereby obtaining the target radar signal.

[0109] In the embodiment of the present invention, through the above two-step processing, the static clutter components in the background can be effectively separated, so that the dynamic signal including the human heart rate and respiratory rate is clearer, and can be used to accurately detect the human heart rate and respiratory rate.

[0110] The heart rate and breathing rate millimeter wave radar detection device based on CNN fusion features provided by the present invention is described below. The heart rate and breathing rate millimeter wave radar detection device based on CNN fusion features described below and the heart rate and breathing rate millimeter wave radar detection method based on CNN fusion features described above can be referenced to each other.

[0111] Figure 3 : is a structural schematic diagram of a heart rate and respiratory rate millimeter wave radar detection device based on CNN fusion features provided by the present invention, such as Figure 3 As shown, the device comprises:

[0112] An acquisition unit 310 is used to acquire a real-time one-dimensional radar signal of a target person, wherein the one-dimensional radar signal is obtained by monitoring the vital signs of the target person through a millimeter-wave radar sensor;

[0113] An analysis unit 320, configured to perform time-frequency analysis based on the one-dimensional radar signal to obtain a two-dimensional time-frequency signal;

[0114] The detection unit 330 is used to detect the heart rate and respiratory rate of the target person based on the one-dimensional radar signal and the two-dimensional time-frequency signal to obtain the heart rate and respiratory rate of the target person.

[0115] The heart rate and respiratory rate millimeter wave radar detection device based on CNN fusion features provided by the present invention uses the real-time one-dimensional radar signal obtained by monitoring the vital signs of a target person through a millimeter wave radar sensor to perform time-frequency analysis to obtain a two-dimensional time-frequency signal; based on the one-dimensional radar signal and the two-dimensional time-frequency signal, the human heart rate and respiratory rate of the target person are detected, and the key information of the human vital signs contained in the multi-dimensional signal is used to realize the detection of the vital signs of the target person, and obtain accurate and reliable heart rate and respiratory rate, while overcoming the problems of long-term monitoring inconvenience, discomfort, and limited application on specific populations existing in traditional contact detection methods, and realizing comprehensive and accurate human heart rate and respiratory rate detection through non-invasive monitoring means, and is particularly beneficial to the elderly, patients with chronic diseases, and individuals who need long-term rehabilitation monitoring, and provides support for home medical care and remote health monitoring.

[0116] Based on the above embodiment, the detection unit 330 is used for:

[0117] Inputting the one-dimensional radar signal and the two-dimensional time-frequency signal into a vital sign detection model to obtain a heart rate and a respiratory rate of a target person output by the vital sign detection model;

[0118] The vital sign detection model is obtained by training the sample one-dimensional radar signal of the sample personnel and their heart rate label and respiratory rate label based on the convolutional neural network;

[0119] The vital sign detection model is used to detect the human heart rate and respiratory rate of the target person based on the time series characteristics of the one-dimensional radar signal and the frequency component characteristics of the two-dimensional time-frequency signal to obtain the heart rate and respiratory rate of the target person.

[0120] Based on the above embodiment, the vital sign detection model includes a feature extraction network, a feature fusion network and a perception detection network;

[0121] The feature extraction network is used to extract features from the one-dimensional radar signal and the two-dimensional time-frequency signal to obtain time series features of the one-dimensional radar signal and frequency component features of the two-dimensional time-frequency signal;

[0122] The feature fusion network is used to perform feature fusion on the time series feature and the frequency component feature based on the correlation between the time series feature and the frequency component feature to obtain a fused feature;

[0123] The perception detection network is used to detect the human heart rate and respiratory rate of the target person based on the fusion features to obtain the heart rate and respiratory rate of the target person.

[0124] Based on the above embodiment, the analysis unit 320 is used for:

[0125] Performing clutter separation on the one-dimensional radar signal to obtain a target radar signal;

[0126] Performing time-frequency analysis on the target radar signal to obtain a two-dimensional time-frequency signal.

[0127] Based on the above embodiment, the analysis unit 320 is used for:

[0128] Performing wavelet transform on the target radar signal to obtain a wavelet spectrum diagram;

[0129] Phase extraction is performed based on the wavelet spectrum to obtain a two-dimensional time-frequency signal.

[0130] Based on the above embodiment, the analysis unit 320 is used for:

[0131] Acquire a background signal of a target area where the target person is located, wherein the background signal is obtained by monitoring the target area through a millimeter wave radar sensor when no one is in the target area;

[0132] Performing median calculation based on the background signal to obtain background clutter corresponding to the target area, and performing background subtraction on the one-dimensional radar signal based on the background clutter;

[0133] Artifacts are filtered out based on the radar signal after background subtraction to obtain the target radar signal.

[0134] Figure 4 : is a structural schematic diagram of a heart rate and respiratory rate millimeter wave radar detection system based on CNN fusion features provided by the present invention, such as Figure 4 As shown, the system includes: a millimeter wave radar sensor 410 and a processor 420;

[0135] The processor 420 is used to obtain a real-time one-dimensional radar signal of the target person obtained by the millimeter-wave radar sensor 410 for monitoring the vital signs of the target person, and perform time-frequency analysis based on the one-dimensional radar signal to obtain a two-dimensional time-frequency signal; based on the one-dimensional radar signal and the two-dimensional time-frequency signal, perform human heart rate and respiratory rate detection on the target person to obtain the heart rate and respiratory rate of the target person.

[0136] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the heart rate and respiratory rate millimeter wave radar detection method based on CNN fusion features, the method comprising: obtaining a real-time one-dimensional radar signal of a target person, the one-dimensional radar signal being obtained by monitoring the vital signs of the target person through a millimeter wave radar sensor; performing time-frequency analysis based on the one-dimensional radar signal to obtain a two-dimensional time-frequency signal; performing human heart rate and respiratory rate detection on the target person based on the one-dimensional radar signal and the two-dimensional time-frequency signal to obtain the heart rate and respiratory rate of the target person.

[0137] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0138] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by the computer, the computer can execute the heart rate and respiratory rate millimeter wave radar detection method based on CNN fusion features provided by the above methods, and the method includes: obtaining a real-time one-dimensional radar signal of a target person, and the one-dimensional radar signal is obtained by monitoring the vital signs of the target person through a millimeter wave radar sensor; performing time-frequency analysis based on the one-dimensional radar signal to obtain a two-dimensional time-frequency signal; based on the one-dimensional radar signal and the two-dimensional time-frequency signal, performing human heart rate and respiratory rate detection on the target person to obtain the heart rate and respiratory rate of the target person.

[0139] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the heart rate and respiratory rate millimeter wave radar detection method based on CNN fusion features provided by the above-mentioned methods, the method comprising: obtaining a real-time one-dimensional radar signal of a target person, the one-dimensional radar signal being obtained by monitoring the vital signs of the target person through a millimeter wave radar sensor; performing time-frequency analysis based on the one-dimensional radar signal to obtain a two-dimensional time-frequency signal; performing human heart rate and respiratory rate detection on the target person based on the one-dimensional radar signal and the two-dimensional time-frequency signal to obtain the heart rate and respiratory rate of the target person.

[0140] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0141] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A heart rate and respiratory rate millimeter wave radar detection method based on CNN fusion features, characterized in that: include: Acquire a real-time one-dimensional radar signal of a target person, wherein the one-dimensional radar signal is obtained by monitoring the vital signs of the target person through a millimeter-wave radar sensor; Performing time-frequency analysis based on the one-dimensional radar signal to obtain a two-dimensional time-frequency signal; Based on the one-dimensional radar signal and the two-dimensional time-frequency signal, the heart rate and respiratory rate of the target person are detected to obtain the heart rate and respiratory rate of the target person; The detecting of the human heart rate and respiratory rate of the target person based on the one-dimensional radar signal and the two-dimensional time-frequency signal to obtain the heart rate and respiratory rate of the target person includes: Inputting the one-dimensional radar signal and the two-dimensional time-frequency signal into a vital sign detection model to obtain a heart rate and a respiratory rate of a target person output by the vital sign detection model; The vital sign detection model is obtained by training the sample one-dimensional radar signal of the sample personnel and their heart rate label and respiratory rate label based on the convolutional neural network; The vital sign detection model is used to detect the human heart rate and respiratory rate of the target person based on the time series characteristics of the one-dimensional radar signal and the frequency component characteristics of the two-dimensional time-frequency signal to obtain the heart rate and respiratory rate of the target person.

2. The heart rate and respiratory rate millimeter wave radar detection method based on CNN fusion features according to claim 1 is characterized in that: The vital sign detection model includes a feature extraction network, a feature fusion network and a perception detection network; The feature extraction network is used to extract features from the one-dimensional radar signal and the two-dimensional time-frequency signal to obtain time series features of the one-dimensional radar signal and frequency component features of the two-dimensional time-frequency signal; The feature fusion network is used to perform feature fusion on the time series feature and the frequency component feature based on the correlation between the time series feature and the frequency component feature to obtain a fused feature; The perception detection network is used to detect the human heart rate and respiratory rate of the target person based on the fusion features to obtain the heart rate and respiratory rate of the target person.

3. The heart rate and respiratory rate millimeter wave radar detection method based on CNN fusion features according to claim 1 or 2 is characterized in that: The step of performing time-frequency analysis based on the one-dimensional radar signal to obtain a two-dimensional time-frequency signal includes: Performing clutter separation on the one-dimensional radar signal to obtain a target radar signal; Performing time-frequency analysis on the target radar signal to obtain a two-dimensional time-frequency signal.

4. The heart rate and respiratory rate millimeter wave radar detection method based on CNN fusion features according to claim 3 is characterized in that: The step of performing time-frequency analysis on the target radar signal to obtain a two-dimensional time-frequency signal includes: Performing wavelet transform on the target radar signal to obtain a wavelet spectrum diagram; Phase extraction is performed based on the wavelet spectrum to obtain a two-dimensional time-frequency signal.

5. The heart rate and respiratory rate millimeter wave radar detection method based on CNN fusion features according to claim 3 is characterized in that: The step of performing clutter separation on the one-dimensional radar signal to obtain a target radar signal includes: Acquire a background signal of a target area where the target person is located, wherein the background signal is obtained by monitoring the target area through a millimeter wave radar sensor when no one is in the target area; Performing median calculation based on the background signal to obtain background clutter corresponding to the target area, and performing background subtraction on the one-dimensional radar signal based on the background clutter; Artifacts are filtered out based on the radar signal after background subtraction to obtain the target radar signal.

6. A heart rate and respiratory rate millimeter wave radar detection device based on CNN fusion features, characterized in that: include: An acquisition unit, used for acquiring a real-time one-dimensional radar signal of a target person, wherein the one-dimensional radar signal is obtained by monitoring the vital signs of the target person through a millimeter-wave radar sensor; An analysis unit, configured to perform time-frequency analysis based on the one-dimensional radar signal to obtain a two-dimensional time-frequency signal; A detection unit, configured to detect the heart rate and respiratory rate of the target person based on the one-dimensional radar signal and the two-dimensional time-frequency signal, so as to obtain the heart rate and respiratory rate of the target person; The detection unit is specifically used for: Inputting the one-dimensional radar signal and the two-dimensional time-frequency signal into a vital sign detection model to obtain a heart rate and a respiratory rate of a target person output by the vital sign detection model; The vital sign detection model is obtained by training the sample one-dimensional radar signal of the sample personnel and their heart rate label and respiratory rate label based on the convolutional neural network; The vital sign detection model is used to detect the human heart rate and respiratory rate of the target person based on the time series characteristics of the one-dimensional radar signal and the frequency component characteristics of the two-dimensional time-frequency signal to obtain the heart rate and respiratory rate of the target person.

7. A heart rate and respiratory rate millimeter wave radar detection system based on CNN fusion features, characterized in that: include: Millimeter wave radar sensors and processors; The processor is used to obtain a real-time one-dimensional radar signal of the target person obtained by the millimeter-wave radar sensor for monitoring the vital signs of the target person, and perform time-frequency analysis based on the one-dimensional radar signal to obtain a two-dimensional time-frequency signal; input the one-dimensional radar signal and the two-dimensional time-frequency signal into a vital sign detection model to obtain the heart rate and respiratory rate of the target person output by the vital sign detection model; The vital sign detection model is obtained by training the sample one-dimensional radar signal of the sample personnel and their heart rate label and respiratory rate label based on the convolutional neural network; The vital sign detection model is used to detect the human heart rate and respiratory rate of the target person based on the time series characteristics of the one-dimensional radar signal and the frequency component characteristics of the two-dimensional time-frequency signal to obtain the heart rate and respiratory rate of the target person.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the heart rate and breathing rate millimeter wave radar detection method based on CNN fusion features as described in any one of claims 1 to 5 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting heart rate and respiratory rate by millimeter-wave radar based on CNN fusion features as described in any one of claims 1 to 5 is implemented.

Images