Information output method and device based on millimeter wave radar signal

Abstract

The present invention provides an information output method and device based on millimeter wave radar signals, the method comprising: transmitting a millimeter wave radar transmission signal to a measured object, and receiving a plurality of millimeter wave radar echo signals reflected by a irradiated area of ​​the measured object and defined by a plurality of channels composed of N transmitting antennas and M receiving antennas; performing coherent accumulation operation processing on the plurality of millimeter wave radar echo signals to obtain a plurality of initial signals, the initial signals comprising three-dimensional spatial features and time features, the plurality of initial signals corresponding to a plurality of spatial position points, the initial signal of each spatial position point corresponding to a plurality of time windows; performing filtering processing on the plurality of initial signals to eliminate the influence of mechanical movement caused by breathing of the measured object, and obtaining a plurality of reference signals related to the heartbeat movement of the heart part of the measured object; inputting the plurality of reference signals into a trained information prediction model, and outputting electrocardiogram representation prediction information of the measured object.

Description

Technical Field

[0001] The present invention relates to the field of intelligent sensing technology, and in particular to an information output method and device based on millimeter-wave radar signals. Background Art

[0002] The number of deaths caused by chronic diseases and cardiovascular diseases (CVD) is increasing worldwide. The electrocardiogram (ECG) is one of the most important biomedical signals used to describe the activity of the human heart and provides important information for the diagnosis of heart disease. Experimental evidence shows that many cardiovascular diseases can be better diagnosed, controlled and prevented by continuous monitoring and analysis of the ECG.

[0003] Currently, electrocardiogram monitoring methods are based on body electrodes, which detect tiny electrical changes caused by the depolarization and repolarization of the myocardium during each cardiac cycle. Existing detection methods have many limitations, such as the need to attach sensors, restrict the user's mobility, and involve privacy issues.

[0004] The introduction of millimeter-wave radar technology in the field of vital signs monitoring provides innovative possibilities for non-invasive and non-intrusive monitoring. It enables contactless vital signs monitoring. Traditional vital signs monitoring methods usually require sensors to be attached directly to the human body, such as breathing belts, ECG patches, nasal probes, etc. These sensors can be bulky, limit the user's mobility, and may cause allergic reactions and other problems. In contrast, millimeter-wave radar-based vital signs monitoring technology can obtain the target's vital signs information, such as breathing and heartbeat, by scanning the target without attaching sensors, thus avoiding the above problems. This method can provide more comprehensive monitoring results and provide more information for medical research and clinical diagnosis.

[0005] In the process of implementing the present disclosure, it was found that the application of millimeter wave radar makes it possible to obtain a human electrocardiogram without contact. However, how to completely and accurately extract the mechanical activity information of the heart in the chest and how to accurately convert the mechanical activity of the heart into electrical activity are technical problems that need to be solved urgently. Summary of the invention

[0006] In view of the above problems, the present invention provides a method and device for outputting information based on millimeter-wave radar signals performed by an electronic device.

[0007] In one aspect of the present invention, there is provided a method for outputting information based on a millimeter-wave radar signal performed by an electronic device, comprising:

[0008] Transmitting a millimeter-wave radar transmission signal to the object under test, and receiving a plurality of millimeter-wave radar echo signals reflected by the irradiated area of ​​the object under test and defined by a plurality of channels composed of N transmitting antennas and M receiving antennas;

[0009] Performing coherent accumulation operation processing on multiple millimeter-wave radar echo signals to obtain multiple initial signals, the initial signals contain three-dimensional spatial features and time features, the multiple initial signals correspond to multiple spatial position points, and the initial signal of each spatial position point corresponds to multiple time windows;

[0010] Filtering the multiple initial signals to eliminate the influence of mechanical motion caused by the breathing of the measured object, and obtaining multiple reference signals related to the heartbeat motion of the heart of the measured object;

[0011] Multiple reference signals are input into the trained information prediction model to output the ECG representation prediction information of the subject.

[0012] According to an embodiment of the present invention, filtering the multiple initial signals includes:

[0013] Performing multi-level transformation decomposition on multiple initial signals using a continuous wavelet transform decomposition algorithm to generate a predetermined number of N groups of primary decomposition signals;

[0014] Based on the magnitude of the signal frequency, each group of primary decomposition signals is decomposed into a first decomposition signal and a second decomposition signal, wherein the frequency of the first decomposition signal is smaller than the frequency of the second decomposition signal;

[0015] A target group of decomposed signals is determined from the N groups of second decomposed signals as a plurality of reference signals related to the heartbeat motion of the heart of the subject.

[0016] According to an embodiment of the present invention, determining a target group decomposition signal from N groups of second decomposition signals comprises:

[0017] Acquiring heartbeat rhythm information of the subject, wherein the heartbeat rhythm information includes heartbeat time interval and heartbeat amplitude;

[0018] Comparing the signal pulse interval and the signal pulse amplitude of each group of the N groups of second decomposition signals with the heartbeat rhythm information of the subject;

[0019] According to the comparison result, a target group of decomposition signals is determined from the N groups of second decomposition signals.

[0020] According to an embodiment of the present invention, determining a target group decomposition signal from N groups of second decomposition signals comprises:

[0021] Using an electrocardiogram measuring device to collect a real electrocardiogram of the subject;

[0022] Calculating the correlation between each group of second decomposition signals in the N groups of second decomposition signals and the true electrocardiogram;

[0023] The second decomposition signal having the greatest correlation with the true electrocardiogram is determined as the target group decomposition signal.

[0024] According to an embodiment of the present invention, wherein:

[0025] The information prediction model includes the first stage model and the second stage model;

[0026] The first stage model includes feature extraction layer, Gaussian distribution mapping layer, and feature restoration layer;

[0027] The second stage model includes the electrical feature mapping layer.

[0028] According to an embodiment of the present invention, inputting a plurality of reference signals into a trained information prediction model and outputting electrocardiogram prediction information of a measured object comprises:

[0029] Input multiple reference signals into the feature extraction layer to extract the characteristics of heartbeat movement law and generate original heartbeat features;

[0030] The original heartbeat features are input into the Gaussian distribution mapping layer so as to constrain the original heartbeat features using Gaussian distribution parameters and generate target heartbeat features that conform to Gaussian distribution characteristics;

[0031] Some randomly sampled features in the target heartbeat features are input into the electrical feature mapping layer for electrical feature conversion to obtain electrocardiographic representation prediction information for representing the heartbeat movement law of the measured object.

[0032] According to an embodiment of the present invention, the first stage model is trained by the following method:

[0033] Acquire multiple training samples and a first initial model to be trained, wherein the training samples have the same data type as the reference signal and are obtained by processing multiple millimeter-wave radar echo signals reflected by the sample object, and the first initial model has the same network structure as the first-stage model;

[0034] Inputting a plurality of training samples into a feature extraction layer of a first initial model to extract features of heartbeat motion patterns, thereby generating original heartbeat feature samples;

[0035] Inputting the original heartbeat feature sample into the Gaussian distribution mapping layer of the first initial model, so as to constrain the original heartbeat feature by using Gaussian distribution parameters and generate a target heartbeat feature sample that conforms to the Gaussian distribution characteristics;

[0036] Inputting some randomly sampled feature samples in the target heartbeat feature samples into the feature restoration layer of the first initial model for feature restoration to generate restored samples;

[0037] Calculate the error loss between the restored sample and the training sample, and adjust the network parameters of the feature extraction layer according to the error loss;

[0038] When the error loss meets the first preset threshold condition, a trained first-stage model is obtained.

[0039] According to an embodiment of the present invention, the second stage model is trained by the following method:

[0040] Using an electrocardiogram measuring device to obtain an electrocardiogram sample of the sample object;

[0041] Obtaining a second initial model, wherein the second initial model has the same network structure as the second-stage model;

[0042] Inputting some randomly sampled feature samples in the target heartbeat feature samples and the electrocardiogram samples into the electrical feature mapping layer of the second initial model to obtain an electrocardiogram representation prediction sample for representing the heartbeat movement law of the sample object;

[0043] Calculate the error loss between the ECG representation prediction sample and the ECG sample, and adjust the network parameters of the electrical feature mapping layer according to the error loss;

[0044] When the error loss meets the second preset threshold condition, a trained second-stage model is obtained.

[0045] According to an embodiment of the present invention, performing coherent accumulation operation processing on multiple millimeter wave radar echo signals includes adopting the following method:

[0046]

[0047] in: The i-th signal corresponding to the spatial position point (x, y, z) and the signal acquisition time window t among the multiple initial signals obtained after the coherent accumulation operation;

[0048] is the number of receiving antennas;

[0049] is the number of transmitting antennas;

[0050] is a plurality of millimeter wave radar echo signals, corresponding to the The signal of the channel defined by the mth receiving antenna and the mth transmitting antenna in the acquisition period t;

[0051] is the rate of change of the transmission frequency of the millimeter wave radar transmission signal;

[0052] is the wavelength of the signal transmitted by the millimeter wave radar;

[0053] For the transmitting antenna to the target location And return The round trip distance of the receiving antennas;

[0054] is the speed of light;

[0055] T is the sampling period.

[0056] Another aspect of the present invention provides an information output device based on millimeter wave radar signals, comprising:

[0057] A transmitting and receiving module, used to transmit a millimeter-wave radar transmission signal to the object under test, and receive a plurality of millimeter-wave radar echo signals reflected by the irradiated area of ​​the object under test and defined by a plurality of channels composed of N transmitting antennas and M receiving antennas;

[0058] An operation processing module is used to perform coherent accumulation operation processing on multiple millimeter wave radar echo signals to obtain multiple initial signals, the initial signals contain three-dimensional spatial features and time features, the multiple initial signals correspond to multiple spatial position points, and the initial signal of each spatial position point corresponds to multiple time windows;

[0059] A filtering processing module, used for filtering the multiple initial signals to eliminate the influence of mechanical motion caused by the breathing of the measured object, and obtain multiple reference signals related to the heartbeat motion of the heart part of the measured object;

[0060] The information prediction module is used to input multiple reference signals into the trained information prediction model and output the electrocardiogram representation prediction information of the object under test.

[0061] According to an embodiment of the present invention, through the above method, signal processing technology is used in combination with an end-to-end deep learning network to realize the perception of mechanical activity of human chest heart micro-movements based on millimeter wave radar, and to realize the spatiotemporal relationship mapping with cardiac electrical activity, and complete the mapping of human chest heart mechanical activity to various indicators of electrocardiogram electrical activity, thereby realizing non-contact fine-grained electrocardiogram monitoring.

[0062] According to an embodiment of the present invention, the above method performs contactless detection based on millimeter-wave radar signals, which can avoid the discomfort caused by wearing electrodes and sensor devices, and can avoid cross infection. Compared with the traditional contact measurement method, it greatly improves the user experience and scope of application (for example, due to poor physical health, it is impossible to directly touch the body part, which makes the traditional measurement method impossible to implement, but this method is not limited by this). The current electrocardiogram monitoring method can not only obtain the electrocardiogram waveform when the radar is facing the human chest, but also in more complex environments, such as when the human body is sitting, while ensuring that the human body is motionless.

[0063] According to an embodiment of the present invention, by filtering the original collected signals to eliminate the influence of mechanical movement caused by the breathing of the subject, the obtained multiple reference signals more accurately characterize the law of heartbeat movement of the subject's heart, thereby improving the accuracy of detection.

[0064] Furthermore, the mapping of cardiac mechanical activity to cardiac electrical activity is completed based on an end-to-end deep learning network, realizing the automated mapping processing of artificial intelligence. The prediction results are provided for user reference, providing a new information dimension and improving the user experience.

[0065] The current radar technology for extracting cardiac activity is limited to the phase difference aspect, which is greatly affected by the environment, resulting in inaccurate extraction of cardiac activity. And due to the different fluctuations in breathing size, the extraction of cardiac activity signals is also limited; this method is based on the human heartbeat pattern, and uses the wavelet transform method to decompose the phase signal in multiple levels, which can effectively extract a relatively clean heartbeat motion signal, eliminate respiratory interference and environmental noise to a large extent, and make signal extraction more stable. The continuous wavelet decomposition method can largely eliminate the respiratory motion representation, amplify the cardiac signal representation, accurately identify the cardiac signal, improve the accuracy of signal processing, and facilitate the subsequent accurate prediction results. Compared with other filtering processes, the continuous wavelet decomposition method has higher accuracy, and the calculation process is simple and easy to implement, and the filtering process is faster.

[0066] According to an embodiment of the present invention, in terms of the network architecture of deep learning, when the network architecture is used to extract features, it includes a feature extraction layer and a Gaussian distribution mapping layer. The features extracted by the feature extraction layer are distributed in a relatively wide space. If the feature is directly used as the final extracted feature, it is likely that the expected effect will not be achieved, for example, the heartbeat pattern cannot be well reflected. An obvious advantage of the network is that by setting a Gaussian distribution mapping layer, the heartbeat features in the cardiac signal can be mapped to a fixed distribution, thereby making the feature constraints more obvious, and the network has a stronger ability to extract features, thereby increasing generalization, and can be better applied to the scenarios of the present invention.

[0067] According to an embodiment of the present invention, by setting a feature restoration layer to perform feature restoration and performing constraints during the training process, it can be ensured that the Encoder part of the network can accurately learn the common features in the radar signal, thereby improving the accuracy of feature extraction.

[0068] In summary, in terms of signal processing, this method can eliminate respiratory interference more stably and accurately, and extract the most obvious characteristic signal of the heartbeat; in terms of deep learning networks, due to the constraints on the feature space, the feature extractor can extract more effective features for restoring the ECG, and prevent the network from overfitting, thereby improving generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0070] Figure 1 A flowchart of a method for outputting information based on millimeter-wave radar signals according to an embodiment of the present invention is schematically shown;

[0071] Figure 2 The appearance of a millimeter wave radar according to an embodiment of the present invention is schematically shown;

[0072] Figure 3 A diagram schematically shows a decomposition result obtained after performing multi-level transformation decomposition on multiple initial signals using a continuous wavelet transform decomposition algorithm according to an embodiment of the present invention;

[0073] Figure 4 Schematically shows the appearance of an electrocardiogram detection device according to an embodiment of the present invention;

[0074] Figure 5 A schematic diagram of a network structure of an information prediction model according to an embodiment of the present invention is shown;

[0075] Figure 6 A schematic diagram schematically shows a process of testing a subject using the information output method of an embodiment of the present invention;

[0076] Figure 7 Schematically showing electrocardiogram waveforms generated for the same subject under different breathing modes according to an embodiment of the present invention, as well as correlations with true values ​​and RMS (root mean square) index diagrams;

[0077] Figure 8 A schematic diagram showing the correlation comparison between the electrocardiogram representation prediction information output by the information output method according to an embodiment of the present invention and the electrocardiogram collected by the electrocardiogram detection device for multiple different subjects;

[0078] Fig. 9 A schematic diagram showing a comparison of the RMS (root mean square) errors between the electrocardiogram (ECG) representation prediction information output by the information output method according to an embodiment of the present invention and the electrocardiogram (ECG) collected by an ECG detection device for a plurality of different subjects;

[0079] Fig.10 The structure block diagram of the information output device based on millimeter-wave radar signals according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0080] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0081] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0082] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0083] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0084] In one aspect of the present invention, a method for outputting information based on a millimeter-wave radar signal performed by an electronic device is provided. Figure 1 The flowchart of the information output method based on the millimeter wave radar signal according to an embodiment of the present invention is schematically shown. Figure 1 As shown, the method includes operations S101-S104.

[0085] In operation S101 , a millimeter wave radar transmission signal is transmitted to an object under test, and a plurality of millimeter wave radar echo signals reflected by an irradiated area of ​​the object under test and defined by a plurality of channels composed of N transmitting antennas and M receiving antennas are received.

[0086] Figure 2The appearance diagram of the millimeter wave radar according to an embodiment of the present invention is schematically shown. The millimeter wave radar may be a multi-transmit and multi-receive device, including N transmitting antennas and M receiving antennas. For example, it includes 3 transmitting antennas and 4 receiving antennas.

[0087] The specific signal collection method can be, for example, a human body lying flat on the bed, using Figure 2 The millimeter wave radar shown transmits millimeter wave signals directly at the chest position of the human body and receives echo signals. Specifically, the radiation area can be the radiation range of the millimeter wave radar, for example, including the chest area of ​​the object being measured.

[0088] In operation S102, a plurality of millimeter-wave radar echo signals are processed by coherent accumulation operation to obtain a plurality of initial signals, wherein the initial signals include three-dimensional spatial features and time features, the plurality of initial signals correspond to a plurality of spatial position points, and the initial signal of each spatial position point corresponds to a plurality of time windows.

[0089] The purpose of this operation is to extract the motion data from the echo signal of the whole chest area in space by processing the received echo signal. The details are as follows:

[0090] By utilizing the array antenna and frequency-modulated continuous wave system of the radio frequency equipment, the propagation distance of the reflected signal at different positions on the chest can be mapped to the phase change. Finally, the reflected signal at all positions in the entire chest can be restored by performing coherent accumulation operations on the received signals in the virtual channel composed of different transmitting and receiving antennas.

[0091] Specifically, the coherent accumulation operation processing of multiple millimeter wave radar echo signals includes adopting the following method with reference to the following formula (1):

[0092] (1)

[0093] in: The i-th signal corresponding to the spatial position point (x, y, z) and the signal acquisition time window t among the multiple initial signals obtained after the coherent accumulation operation;

[0094] is the number of receiving antennas;

[0095] is the number of transmitting antennas;

[0096] is a plurality of millimeter wave radar echo signals, corresponding to the The signal of the channel defined by the mth receiving antenna and the mth transmitting antenna in the acquisition period t;

[0097] is the rate of change of the transmission frequency of the millimeter wave radar transmission signal;

[0098] is the wavelength of the signal transmitted by the millimeter wave radar;

[0099] For the transmitting antenna to the target location And return The round trip distance of the receiving antennas;

[0100] is the speed of light;

[0101] T is the sampling period.

[0102] In operation S103, a plurality of initial signals are filtered to eliminate the influence of mechanical motion caused by the breathing of the subject, and a plurality of reference signals related to the heartbeat motion of the heart of the subject are obtained.

[0103] In addition to the target signal required for detection, that is, the millimeter-wave radar echo signal generated by the body surface vibration caused by heartbeat movement, the initial signal also contains implicit interference signals, such as the millimeter-wave radar echo signal generated by the body surface vibration caused by breathing. Therefore, the collected signal needs to be filtered to obtain multiple reference signals for characterizing the heartbeat movement patterns of the heart part of the subject.

[0104] In operation S104, a plurality of reference signals are input into a trained information prediction model, and electrocardiogram representation prediction information of the measured object is output.

[0105] According to an embodiment of the present invention, the chest movement caused by breathing or heartbeat of the human body can be captured by the phase of the radar. When the radar is facing the chest of the human body, the breathing and heartbeat changes can be detected by the radar because there is no interference from other parts of the human body. However, the radar captures the mechanical movement of the human body surface, while most of the current information used is the electrocardiogram, which reflects the electrical activity of the human heart. Experiments have shown that there is a certain nonlinear relationship between the electrical activity and mechanical activity of the human heart, and this relationship can be learned through a deep learning network to achieve the conversion from the mechanical activity of the heart on the surface of the human body to the electrical activity.

[0106] Therefore, after the micro-motion characterization of the heart is amplified, the mechanical motion characteristics similar to the heartbeat can be clearly identified, and then the mechanical activity of the heart can be mapped to the electrical activity. Specifically, the extracted cardiac mechanical activity signal can be used to build an end-to-end deep learning network, that is, to build the above-mentioned information prediction model, and the information prediction model can be used to complete the mapping of cardiac mechanical activity to cardiac electrical activity, and obtain electrocardiogram characterization prediction information, specifically, the waveform of the electrocardiogram can be obtained.

[0107] According to an embodiment of the present invention, through the above method, signal processing technology is used in combination with an end-to-end deep learning network to realize the perception of mechanical activity of human chest heart micro-movements based on millimeter wave radar, and to realize the spatiotemporal relationship mapping with cardiac electrical activity, and complete the mapping of human chest heart mechanical activity to various indicators of electrocardiogram electrical activity, thereby realizing non-contact fine-grained electrocardiogram monitoring.

[0108] According to an embodiment of the present invention, the above method performs contactless detection based on millimeter-wave radar signals, which can avoid the discomfort caused by wearing electrodes and sensor devices, and can avoid cross infection. Compared with the traditional contact measurement method, it greatly improves the user experience and scope of application (for example, due to poor physical health, it is impossible to directly touch the body part, which makes the traditional measurement method impossible to implement, but this method is not limited by this). The current electrocardiogram monitoring method can not only obtain the electrocardiogram waveform when the radar is facing the human chest, but also in more complex environments, such as when the human body is sitting, while ensuring that the human body is motionless.

[0109] According to an embodiment of the present invention, by filtering the original collected signals to eliminate the influence of mechanical movement caused by the breathing of the subject, the obtained multiple reference signals more accurately characterize the law of heartbeat movement of the subject's heart, thereby improving the accuracy of detection.

[0110] Furthermore, the mapping of cardiac mechanical activity to cardiac electrical activity is completed based on an end-to-end deep learning network, realizing the automated mapping processing of artificial intelligence. The prediction results are provided for user reference, providing a new information dimension and improving the user experience.

[0111] It should be noted here that the ECG representation prediction information output by the information prediction model has a similar information representation function to the electrocardiogram. The prediction result is only used as reference information for medical staff to make a diagnosis in order to obtain the intermediate result of processing information parameters, and is not used as a direct diagnosis result.

[0112] Furthermore, all steps of the above method are information processing methods implemented by computers and other devices. The entire process does not require the participation of medical personnel, and the direct purpose is not to obtain a diagnosis result or health status. The information obtained itself cannot directly lead to a diagnosis result of the disease or health status.

[0113] According to an embodiment of the present invention, after extracting the phase change of the reflected signal from the chest position, since the phase change is modulated by breathing and heartbeat, it is necessary to filter the phase change to eliminate the influence of breathing. In the above operation S103, filtering the multiple initial signals includes the following operations:

[0114] Operation 11: Perform multi-level transformation decomposition on the multiple initial signals using a continuous wavelet transform decomposition algorithm to generate a predetermined number of N groups of primary decomposition signals.

[0115] Although the amplitude of breathing is large and the frequency is close to the heartbeat frequency, the representation form of breathing movement itself is quite different from that of heartbeat movement, and heartbeat movement has a fixed vibration mode. Therefore, the continuous wavelet transform decomposition method is used to decompose the phase signal through multi-level wavelet decomposition to extract signals of different frequency bands.

[0116] The specific decomposition level of the multi-level decomposition is determined according to the signal extraction accuracy. After experimental determination, the heartbeat frequency band can be more clearly identified after 6 levels of decomposition. If it is less than 6 levels, the recognition accuracy cannot be achieved. After more than 6 levels of decomposition, the heartbeat frequency band can also be clearly identified, but the accuracy difference with the 6-level decomposition result is not much. Therefore, more than 6 decompositions will cause a waste of computing resources.

[0117] Operation 12: Decompose each group of primary decomposed signals into a first decomposed signal and a second decomposed signal based on the magnitude of the signal frequency, wherein the frequency of the first decomposed signal is smaller than the frequency of the second decomposed signal.

[0118] Figure 3 The diagram schematically shows a decomposition result obtained after performing multi-level transformation decomposition on multiple initial signals using a continuous wavelet transform decomposition algorithm according to an embodiment of the present invention.

[0119] After the phase signal is decomposed by 6 levels of wavelet, 6 groups of primary decomposition signals are obtained. The high-frequency signal and the low-frequency signal in each group of primary decomposition signals are separated respectively, and the following are obtained: Figure 3 The 12 groups of decomposition results are shown.

[0120] in, Figure 3 The left side is the first decomposition signal, which corresponds to the mechanical motion signal representation caused by breathing. Figure 3 The right side is the second decomposed signal, which corresponds to the mechanical motion signal representation caused by the heartbeat.

[0121] Operation 13: Determine a target group of decomposed signals from the N groups of second decomposed signals as a plurality of reference signals related to the heartbeat motion of the heart of the subject.

[0122] According to an embodiment of the present invention, determining a target group decomposition signal from N groups of second decomposition signals comprises the following operations:

[0123] Operation 131: obtaining heartbeat rhythm information of the subject, wherein the heartbeat rhythm information includes heartbeat time interval and heartbeat amplitude;

[0124] Operation 132, comparing the signal pulse interval and the signal pulse amplitude of each group of the N groups of second decomposed signals with the heartbeat rhythm information of the subject;

[0125] Operation 133 : Determine a target group of decomposed signals from the N groups of second decomposed signals according to the comparison result.

[0126] like Figure 3 As shown, each group of signals in the second decomposed signal includes multiple pulses, one heartbeat corresponds to one pulse, the time interval between adjacent pulses corresponds to the heartbeat time interval, and the signal pulse amplitude corresponds to the heartbeat amplitude. The signal pulse interval and signal pulse amplitude of each group of second decomposed signals are compared with the heartbeat rhythm information of the object under test, specifically, the signal pulse interval and the heartbeat time interval are compared for similarity, and the signal pulse amplitude and the heartbeat amplitude are compared for similarity, and the signal with both similarities is determined as the target group decomposed signal.

[0127] like Figure 3 In the present invention, the signal pulse interval and signal pulse amplitude of the third group of signals are similar to the heartbeat interval and heartbeat amplitude of a real human body, and are determined as the final target signals.

[0128] The current radar technology for extracting cardiac activity is limited to the phase difference, which is greatly affected by the environment, resulting in inaccurate extraction of cardiac activity. And due to the different fluctuations in breathing, the extraction of cardiac activity signals is also limited; this method is based on the human heartbeat pattern and uses the wavelet transform method to decompose the phase signal at multiple levels, which can effectively extract a relatively clean heartbeat motion signal, eliminate breathing interference and environmental noise to a large extent, and make the signal extraction more stable.

[0129] Further, the method for determining the target group decomposition signal from the N groups of second decomposition signals may also be:

[0130] Operation 1311: using an electrocardiogram measuring device to collect a real electrocardiogram of the subject;

[0131] Figure 4 The external view of the electrocardiogram detection device according to an embodiment of the present invention is schematically shown. Figure 4 The device shown acquires an electrocardiogram.

[0132] Operation 1321, calculating the correlation between each group of second decomposition signals in the N groups of second decomposition signals and the real electrocardiogram;

[0133] Operation 1331: determine the second decomposition signal having the greatest correlation with the true electrocardiogram as the target group decomposition signal.

[0134] According to the embodiments of the present invention, the method of continuous wavelet decomposition can largely eliminate the respiratory motion representation, amplify the cardiac signal representation, accurately identify the cardiac signal, improve the accuracy of signal processing, and facilitate the subsequent accurate prediction results. The method of continuous wavelet decomposition has higher accuracy than other filtering processes, and the calculation process is simple and easy to implement, and the filtering process efficiency is faster.

[0135] Figure 5 A schematic diagram of the network structure of an information prediction model according to an embodiment of the present invention is shown schematically.

[0136] like Figure 5 As shown, the information prediction model consists of two parts, including the first stage model and the second stage model.

[0137] Among them, the first stage model includes a feature extraction layer (such as the encoder in 5, which can be a one-dimensional convolutional TCN network), a Gaussian distribution mapping layer (two fully connected networks), and a feature restoration layer.

[0138] The second stage model includes the electrical feature mapping layer (e.g. Figure 5 Wavenet network in ).

[0139] According to an embodiment of the present invention, the prediction process is also divided into two different stages. Specifically, multiple reference signals are input into a trained information prediction model, and the output of the ECG prediction information of the measured object includes:

[0140] The first stage: multiple reference signals are input into the feature extraction layer to extract the characteristics of the heartbeat movement law and generate the original heartbeat features; the original heartbeat features are then input into the Gaussian distribution mapping layer so as to constrain the original heartbeat features using Gaussian distribution parameters and generate target heartbeat features that conform to the Gaussian distribution characteristics;

[0141] For example, first, the reference signal features of the mechanical motion of the human chest are input into the one-dimensional convolutional TCN network to extract the features of the heart mechanical motion data and generate the original heartbeat features; then the original heartbeat features are input into the Gaussian distribution mapping layer, and the features are mapped to the mean through two fully connected networks. , the variance is in the Gaussian distribution.

[0142] The second stage: some randomly sampled features of the target heartbeat features are input into the electrical feature mapping layer for electrical feature conversion to obtain the electrocardiogram representation prediction information used to characterize the heartbeat movement law of the object under test.

[0143] For example, the features output by the encoder of the first-stage network are input into Wavenet for electrical feature conversion, mapping of cardiac mechanical activity to electrical activity, and generation of waveform representing the information of the cardiac electrical activity.

[0144] According to an embodiment of the present invention, based on the learned deep learning architecture of human body surface heart mechanical activity, the extracted chest radar-sensed heart mechanical activity data is input, and the generation result of the heart electrocardiogram waveform is output, ultimately realizing non-contact electrocardiogram monitoring.

[0145] According to an embodiment of the present invention, in terms of the network architecture of deep learning, when the above-mentioned network architecture is used to extract features, it includes a feature extraction layer and a Gaussian distribution mapping layer. The features extracted by the feature extraction layer are distributed in a relatively wide space. If the feature is directly used as the final extracted feature, it is very likely that the expected effect will not be achieved, for example, the heartbeat pattern cannot be well reflected. An obvious advantage of the network is that by setting a Gaussian distribution mapping layer, the heartbeat features in the cardiac signal can be mapped to a fixed distribution, thereby making the feature constraints more obvious, and the network has a stronger ability to extract features, thereby increasing generalization, and can be better applied to the scenarios of the present invention.

[0146] According to an embodiment of the present invention, the first stage model is trained by the following method:

[0147] Operation 21: obtaining a plurality of training samples and a first initial model to be trained, wherein the training samples have the same data type as the reference signal and are obtained by processing a plurality of millimeter-wave radar echo signals reflected by the sample object, and the first initial model has the same network structure as the first-stage model;

[0148] Operation 22: inputting the plurality of training samples into the feature extraction layer of the first initial model to extract the features of the heartbeat movement law, and generating original heartbeat feature samples;

[0149] Operation 23: inputting the original heartbeat feature sample into the Gaussian distribution mapping layer of the first initial model, so as to constrain the original heartbeat feature by using Gaussian distribution parameters and generate a target heartbeat feature sample that conforms to Gaussian distribution characteristics;

[0150] The method of operation 21 to operation 23 is the same as the prediction method of the first stage. For example, firstly, the reference signal feature of the mechanical motion of the human chest is input into the one-dimensional convolutional TCN network to extract the features of the mechanical motion data of the heart and generate the original heartbeat feature; then the original heartbeat feature is input into the Gaussian distribution mapping layer, and the feature is mapped to the mean through two fully connected networks. , the variance is in the Gaussian distribution.

[0151] During the training phase, in order to ensure that the Encoder part of the network accurately learns the common features of the radar signal, including the frequency of the heartbeat, the pattern of the heartbeat vibration, etc., the original radar heartbeat signal is restored through one-dimensional deconvolution TCN. Specifically, the following operations 24-26 are included.

[0152] Operation 24: inputting some randomly sampled feature samples in the target heartbeat feature samples into the feature restoration layer (one-dimensional deconvolution TCN) of the first initial model for feature restoration to generate restored samples;

[0153] Operation 25: calculating the error loss between the restored sample and the training sample, and adjusting the network parameters of the feature extraction layer according to the error loss;

[0154] Operation 26: When the error loss satisfies a first preset threshold condition, a trained first-stage model is obtained.

[0155] According to an embodiment of the present invention, by setting a feature restoration layer to perform feature restoration and performing constraints during the training process, it can be ensured that the Encoder part of the network can accurately learn the common features in the radar signal, thereby improving the accuracy of feature extraction.

[0156] According to an embodiment of the present invention, the second stage model is trained by the following method:

[0157] Operation 31: obtaining an electrocardiogram sample of the sample object by using an electrocardiogram measuring device;

[0158] Operation 32: obtaining a second initial model, wherein the second initial model has the same network structure as the second stage model;

[0159] Operation 33: inputting some randomly sampled feature samples in the target heartbeat feature samples and the electrocardiogram samples into the electrical feature mapping layer of the second initial model to obtain an electrocardiogram representation prediction sample for representing the heartbeat movement law of the sample object;

[0160] Operation 34, calculating the error loss between the ECG representation prediction sample and the ECG sample, and adjusting the network parameters of the electrical feature mapping layer according to the error loss;

[0161] Operation 35: When the error loss satisfies the second preset threshold condition, a trained second-stage model is obtained.

[0162] According to an embodiment of the present invention, by using real electrocardiogram samples for constrained training, the network can learn a mapping method from heartbeat motion signals to electrical signals.

[0163] The second training phase uses autoregressive learning.

[0164] The features of the encoder output of the first-stage network are input into Wavenet, and the modal prior of the electrocardiogram (ECG) is added during the training process, so that the network can learn the mapping of cardiac mechanical activity to electrical activity by itself. At the same time, the encoder part is constrained to learn high-dimensional features to complete the generation of ECG waveforms of the heart's electrical activity.

[0165] The present invention is further described below by embodiments and related test experiments. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present invention. However, it is obvious that one or more embodiments may also be implemented without these specific details. And, in the absence of conflict, the details in the following embodiments may be combined into other feasible embodiments at will.

[0166] Figure 6 The schematic diagram shows the process of testing a subject using the information output method according to the embodiment of the present invention.

[0167] like Figure 6 As shown, the subject lies flat on the bed, and the millimeter wave radar transmits and collects signals 0.4-0.5m above the chest of the human body. In the experiment, the radar is in a 3-transmit 4-receive working state, and the starting frequency of the signal is set to 60GHz, the bandwidth is 4GHz, and the frame rate is 100Hz. The collected signal is output using the information output method of the embodiment of the present invention to output the ECG representation prediction information.

[0168] In addition, the subjects' electrocardiograms were collected by an electrocardiogram device, and the output of the electrocardiogram device was 500HZ.

[0169] In the experiment, the subjects lay flat on the bed and collected data for two minutes. A total of 10 people were tested, and the index results obtained are as follows: Figure 7 , 8 , as shown in Figure 9.

[0170] The performance of the algorithm in generating ECG representation waveforms is judged by the correlation between the generated ECG representation prediction waveform and the real ECG collected by the ECG device, and the overall performance of the algorithm is measured by calculating the similarity between the two and the RMS sum. At the same time, the similarity of the two waveforms under different breathing modes and the changes in heart rate are calculated to measure the stability of the ECG representation information generated by the algorithm.

[0171] The correlation is calculated as follows:

[0172] (2)

[0173] in, and is the value of the data point, and is the corresponding mean, is the number of data points.

[0174] Figure 7 The diagram schematically shows the correlation and RMS (root mean square) index between the electrocardiogram representation prediction information output by the information output method of the embodiment of the present invention under different breathing modes for the same subject and the electrocardiogram collected by the electrocardiogram detection device according to the embodiment of the present invention.

[0175] Among them, the first row of data is the comparison result under normal breathing mode, the second row of data is the comparison result under abnormal breathing mode, and the third row of data is the comparison result under the breathing mode after exercise (there may be an increased heart rate and rapid breathing).

[0176] like Figure 7 As shown, under various breathing modes, the ECG representation prediction information output by the information output method according to the embodiment of the present invention has a strong correlation (similarity) with the electrocardiogram collected by the electrocardiogram detection device.

[0177] Figure 8 A schematic diagram shows the correlation comparison between the electrocardiogram representation prediction information output by the information output method of an embodiment of the present invention and the electrocardiogram collected by an electrocardiogram detection device for multiple different subjects (IDs in the figure represent the tester numbers).

[0178] Fig. 9 A schematic diagram shows a comparison of the RMS (root mean square) errors between the electrocardiogram (ECG) prediction information output by the information output method of an embodiment of the present invention and the electrocardiogram (ECG) collected by an ECG detection device for multiple different subjects (IDs in the figure represent the tester numbers).

[0179] Figure 8 , Fig. 9 The comparison results for 10 subjects are shown in the figure. As shown in the figure, the electrocardiogram prediction information output by the information output method of the embodiment of the present invention has a strong correlation with the electrocardiogram collected by the electrocardiogram detection device, and the error between the two is small.

[0180] It can be seen that the method of the embodiment of the present invention achieves relatively accurate monitoring performance in actual electrocardiogram monitoring.

[0181] Based on the above-mentioned information output method based on millimeter wave radar signal, the present disclosure also provides an information output device based on millimeter wave radar signal. Fig.10 The device is described in detail.

[0182] Fig.10The structure block diagram of the information output device based on millimeter-wave radar signals according to an embodiment of the present invention is schematically shown.

[0183] like Fig.10 As shown, the information output device 1000 of this embodiment includes a transmitting and receiving module 1001, an operation processing module 1002, a filtering processing module 1003, and an information prediction module 1004.

[0184] The transmitting and receiving module 1001 is used to transmit a millimeter wave radar transmission signal to the object under test, and receive a plurality of millimeter wave radar echo signals reflected by the irradiated area of ​​the object under test and defined by a plurality of channels composed of N transmitting antennas and M receiving antennas;

[0185] The operation processing module 1002 is used to perform coherent accumulation operation processing on multiple millimeter wave radar echo signals to obtain multiple initial signals, where the initial signals include three-dimensional spatial features and time features, the multiple initial signals correspond to multiple spatial position points, and the initial signal of each spatial position point corresponds to multiple time windows;

[0186] The filtering processing module 1003 is used to filter the multiple initial signals to eliminate the influence of the mechanical movement caused by the breathing of the measured object, and obtain multiple reference signals related to the heartbeat movement of the heart part of the measured object;

[0187] The information prediction module 1004 is used to input multiple reference signals into the trained information prediction model and output the ECG representation prediction information of the measured object.

[0188] According to an embodiment of the present disclosure, any multiple modules among the transmitting and receiving module 1001, the operation processing module 1002, the filtering processing module 1003, and the information prediction module 1004 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0189] According to an embodiment of the present disclosure, at least one of the transmitting and receiving module 1001, the operation processing module 1002, the filtering processing module 1003, and the information prediction module 1004 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a proper combination of any of them. Alternatively, at least one of the transmitting and receiving module 1001, the operation processing module 1002, the filtering processing module 1003, and the information prediction module 1004 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0190] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for outputting information based on a millimeter-wave radar signal performed by an electronic device, comprising: Transmitting a millimeter-wave radar transmission signal to the object under test, and receiving a plurality of millimeter-wave radar echo signals reflected by the irradiated area of ​​the object under test and defined by a plurality of channels composed of N transmitting antennas and M receiving antennas; Performing coherent accumulation operation processing on the multiple millimeter-wave radar echo signals to obtain multiple initial signals, wherein the initial signals include three-dimensional spatial features and time features, the multiple initial signals correspond to multiple spatial position points, and the initial signal of each spatial position point corresponds to multiple time windows; Filtering the multiple initial signals to eliminate the influence of mechanical motion caused by the breathing of the subject, and obtaining multiple reference signals related to the heartbeat motion of the heart of the subject; Inputting the multiple reference signals into a trained information prediction model, and outputting the electrocardiogram representation prediction information of the subject; The coherent accumulation operation processing of the multiple millimeter wave radar echo signals includes adopting the following method: in: The i-th signal corresponding to the spatial position point (x, y, z) and the signal acquisition time window t among the multiple initial signals obtained after the coherent accumulation operation; N is the number of receiving antennas; M is the number of transmitting antennas; is a signal corresponding to a channel defined by an nth receiving antenna and an mth transmitting antenna in a collection period t among multiple millimeter-wave radar echo signals; is the rate of change of the transmission frequency of the millimeter wave radar transmission signal; is the wavelength of the signal transmitted by the millimeter wave radar; For the transmitting antenna to the target location And return The round trip distance of the receiving antennas; is the speed of light; T is the sampling period.

2. The method according to claim 1, wherein: The filtering process of the multiple initial signals includes: Performing multi-level transformation decomposition on the multiple initial signals using a continuous wavelet transform decomposition algorithm to generate a predetermined number of N groups of primary decomposition signals; Based on the magnitude of the signal frequency, each group of primary decomposition signals is decomposed into a first decomposition signal and a second decomposition signal, wherein the frequency of the first decomposition signal is smaller than the frequency of the second decomposition signal; A target group of decomposed signals is determined from the N groups of second decomposed signals as a plurality of reference signals related to the heartbeat motion of the heart of the subject.

3. The method according to claim 2, wherein: Determining a target group of decomposed signals from the N groups of second decomposed signals comprises: Acquiring heartbeat rhythm information of the subject, wherein the heartbeat rhythm information includes heartbeat time interval and heartbeat amplitude; Comparing the signal pulse interval and the signal pulse amplitude of each group of the second decomposed signals in the N groups of second decomposed signals with the heartbeat rhythm information of the measured object; According to the comparison result, a target group of decomposition signals is determined from the N groups of second decomposition signals.

4. The method according to claim 2, wherein: Determining a target group of decomposed signals from the N groups of second decomposed signals comprises: Using an electrocardiogram measuring device to collect a real electrocardiogram of the subject; Calculating the correlation between each group of the N groups of second decomposed signals and the real electrocardiogram; The second decomposition signal having the greatest correlation with the real electrocardiogram is determined as the target group decomposition signal.

5. The method according to claim 1, wherein: The information prediction model includes a first-stage model and a second-stage model; The first stage model includes a feature extraction layer, a Gaussian distribution mapping layer, and a feature restoration layer; The second stage model includes an electrical signature mapping layer.

6. The method according to claim 5, wherein: Inputting the plurality of reference signals into a trained information prediction model, and outputting the electrocardiogram prediction information of the measured object comprises: Inputting the multiple reference signals into a feature extraction layer to extract features of heartbeat motion patterns, thereby generating original heartbeat features; Inputting the original heartbeat feature into a Gaussian distribution mapping layer so as to constrain the original heartbeat feature using Gaussian distribution parameters to generate a target heartbeat feature that conforms to Gaussian distribution characteristics; Some randomly sampled features in the target heartbeat features are input into the electrical feature mapping layer for electrical feature conversion to obtain electrocardiographic representation prediction information for representing the heartbeat movement law of the measured object.

7. The method according to claim 5, wherein: The first stage model is trained by the following method: Acquire multiple training samples and a first initial model to be trained, wherein the training samples have the same data type as the reference signal and are obtained by processing multiple millimeter-wave radar echo signals reflected by the sample object, and the first initial model has the same network structure as the first-stage model; Input the plurality of training samples into the feature extraction layer of the first initial model to extract the features of the heartbeat movement law, and generate original heartbeat feature samples; Inputting the original heartbeat feature sample into the Gaussian distribution mapping layer of the first initial model, so as to constrain the original heartbeat feature by using Gaussian distribution parameters and generate a target heartbeat feature sample that conforms to Gaussian distribution characteristics; Inputting some randomly sampled feature samples in the target heartbeat feature samples into the feature restoration layer of the first initial model for feature restoration to generate restored samples; Calculating the error loss between the restored sample and the training sample, and adjusting the network parameters of the feature extraction layer according to the error loss; When the error loss satisfies a first preset threshold condition, a trained first-stage model is obtained.

8. The method according to claim 7, wherein: The second stage model is trained by the following method: Using an electrocardiogram measuring device to obtain an electrocardiogram sample of the sample object; Acquire a second initial model, wherein the second initial model has the same network structure as the second stage model; Inputting some randomly sampled feature samples in the target heartbeat feature samples and the electrocardiogram samples into the electrical feature mapping layer of the second initial model to obtain an electrocardiogram representation prediction sample for representing the heartbeat movement law of the sample object; Calculating the error loss between the electrocardiogram representation prediction sample and the electrocardiogram sample, and adjusting the network parameters of the electrical feature mapping layer according to the error loss; When the error loss satisfies the second preset threshold condition, a trained second-stage model is obtained.

9. An information output device based on millimeter wave radar signals, comprising: A transmitting and receiving module, used to transmit a millimeter-wave radar transmission signal to the object under test, and receive a plurality of millimeter-wave radar echo signals reflected by the irradiated area of ​​the object under test and defined by a plurality of channels composed of N transmitting antennas and M receiving antennas; An operation processing module, used for performing coherent accumulation operation processing on the multiple millimeter wave radar echo signals to obtain multiple initial signals, wherein the initial signals include three-dimensional spatial features and time features, the multiple initial signals correspond to multiple spatial position points, and the initial signal of each spatial position point corresponds to multiple time windows; A filtering processing module, used for filtering the multiple initial signals to eliminate the influence of mechanical motion caused by the breathing of the measured object, and obtain multiple reference signals related to the heartbeat motion of the heart part of the measured object; An information prediction module, used for inputting the plurality of reference signals into a trained information prediction model and outputting the electrocardiogram representation prediction information of the measured object; The coherent accumulation operation processing of the multiple millimeter wave radar echo signals includes adopting the following method: in: The i-th signal corresponding to the spatial position point (x, y, z) and the signal acquisition time window t among the multiple initial signals obtained after the coherent accumulation operation; N is the number of receiving antennas; M is the number of transmitting antennas; is a signal corresponding to a channel defined by an nth receiving antenna and an mth transmitting antenna in a collection period t among multiple millimeter-wave radar echo signals; is the rate of change of the transmission frequency of the millimeter wave radar transmission signal; is the wavelength of the signal transmitted by the millimeter wave radar; For the transmitting antenna to the target location And return The round trip distance of the receiving antennas; is the speed of light; T is the sampling period.

Images