Heart rate variability prediction method and device based on millimeter wave radar signal

Through electronic devices based on millimeter wave radar signals, using multi-dimensional feature processing and deep learning models, the contact restriction and signal-to-noise ratio problems of existing HRV monitoring equipment are solved, and the heart rate variability prediction with non-contact fine-grained size is realized, which improves the Monitoring accuracy and user experience.

CN118592917BActive Publication Date: 2024-12-03HE FEI ZHONG KE ZHI QI XIN XI KE JI YOU XIAN GONG SI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410801965.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-20
Publication Date
2024-12-03
Estimated Expiration
2044-06-20

AI Technical Summary

Technical Problem

The existing HRV monitoring equipment relies on contact electrocardiogram equipment, which has problems with daily monitoring difficulties, limited application scenarios and contact skin damage. At the same time, based on the non-contact method, only single-point chest motion information is analyzed, resulting in signal-to-noise ratio of signal quality. Instability affects the accuracy of rate variability prediction.

Method used

Using electronic devices based on millimeter-wave radar signals, by reading multiple reference signals containing three-dimensional spatial characteristics and temporal characteristics, the trained heartbeat rhythm prediction model is used to predict, including spatiotemporal feature extraction, spatial feature compression, reduction, fusion and Feature mapping network generates accurate heartbeat rhythm prediction results.

Benefits of technology

The non-contact fine-grained heart rate variability prediction is achieved, the signal-to-noise ratio is improved, the discomfort and cross-infection of contact measurements is avoided, the scope of application is expanded, and a more accurate mapping of cardiac mechanical activity to electrical activity is provided.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118592917B_ABST
    Figure CN118592917B_ABST
Patent Text Reader

Abstract

The present invention provides a method for predicting heart rate variability based on millimeter wave radar signals, the method comprising: reading multiple reference signals related to the heartbeat movement of the heart part of the measured object from a data storage area, wherein the reference signals include three-dimensional spatial features and time features, the multiple reference signals correspond to multiple spatial position points, the reference signal of each spatial position point corresponds to multiple time windows, the multiple reference signals are obtained by processing multiple original signals, and the multiple original signals are: multiple millimeter wave radar echo signals reflected by the irradiated area of ​​the measured object; using a trained heartbeat rhythm prediction model to predict the multiple reference signals, and outputting a target prediction result for characterizing the heartbeat rhythm of the measured object, wherein the heartbeat rhythm prediction model comprises a spatiotemporal feature extraction basic network, a spatial feature compression network, a spatial feature restoration network, a temporal feature compression network, a spatiotemporal feature fusion network, and a feature mapping network.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of intelligent sensing technology, and in particular to a method and device for predicting heart rate variability based on millimeter wave radar signals. Background Art

[0002] Heart Rate Variability (HRV) refers to the natural fluctuations in the time interval between heartbeats (i.e., RR intervals). This fluctuation reflects the regulatory effect of the autonomic nervous system on the heart, in which the activity of the sympathetic and parasympathetic nerves can affect the speed of the heart rate. HRV is often used to assess the autonomic regulatory function of the heart and overall health, and is related to a variety of cardiovascular diseases. A normal HRV indicates that the heart is able to adapt to changes in the body's needs, while a decrease in HRV may be associated with a variety of heart diseases and adverse health outcomes. Therefore, it is particularly important to accurately measure heart rate variability.

[0003] In the process of realizing the concept of the present invention, the inventors found that there are at least the following problems in the related technology: the currently commonly used HRV monitoring equipment relies on contact ECG equipment, which has diagnostic and treatment defects such as difficulty in daily monitoring, limited application scenarios, and contact skin damage. In addition, the existing non-contact-based heart rate variability prediction method only considers the analysis of single-point chest movement information, which does not conform to the actual spatial attributes of the HRV monitoring task, and will lead to unstable signal-to-noise ratio of the acquired signal quality, thereby affecting the accuracy of the rate variability prediction method. Summary of the invention

[0004] In view of the above problems, the present invention provides a heart rate variability prediction method based on millimeter wave radar signals performed by an electronic device.

[0005] A first aspect of the present invention provides a heart rate variability prediction method based on millimeter wave radar signals performed by an electronic device, comprising:

[0006] Reading multiple reference signals related to the heartbeat movement of the heart of the object under test from the data storage area, wherein the reference signals include three-dimensional spatial features and time features, the multiple reference signals correspond to multiple spatial position points, the reference signal of each spatial position point corresponds to multiple time windows, the multiple reference signals are obtained by processing multiple original signals, and the multiple original signals are: multiple millimeter wave radar echo signals reflected by the irradiated area of ​​the object under test;

[0007] The trained heart rhythm prediction model is used to predict multiple reference signals and output a target prediction result for characterizing the heart rhythm of the subject. The heart rhythm prediction model includes a spatiotemporal feature extraction basic network, a spatial feature compression network, a spatial feature restoration network, a temporal feature compression network, a spatiotemporal feature fusion network, and a feature mapping network.

[0008] According to an embodiment of the present invention, predicting multiple reference signals using a heartbeat rhythm prediction model includes performing multiple prediction operations, in which the heartbeat rhythm prediction model outputs a set of local prediction results based on original input features, and the multiple sets of local prediction results corresponding to the multiple prediction operations are collectively used as target prediction results, and the feature dimensions of the original input features include three-dimensional spatial feature dimensions, time feature dimensions, and channel feature dimensions;

[0009] The Kth prediction operation in multiple prediction operations includes:

[0010] Selecting some reference signals from multiple reference signals as K-th selected signals, and performing feature concatenation of the K-th selected signals and the K-1-th local prediction results output by the heartbeat rhythm prediction model to generate K-th original input features;

[0011] The K-th original input features are input into the heart rhythm prediction model, and the K-th local prediction results are output.

[0012] According to an embodiment of the present invention, feature splicing of the Kth selection signal and the K-1th local prediction result output by the heartbeat rhythm prediction model includes:

[0013] The K-th selection signal is feature concatenated based on the three-dimensional spatial feature dimension and the time feature dimension, and the K-1-th local prediction result is feature concatenated based on the channel feature dimension to generate the K-th original input feature.

[0014] According to an embodiment of the present invention, inputting the Kth original input feature into the heart rhythm prediction model, and outputting the Kth local prediction result includes:

[0015] Input the Kth original input features into the spatiotemporal feature extraction basic network to perform preliminary extraction of spatiotemporal features and generate initial spatiotemporal features;

[0016] The time features in the initial space-time features are isolated, and the space features in the initial space-time features are input into the space feature compression network for feature compression to generate compressed space features;

[0017] The compressed spatial features are input into the spatial feature restoration network for feature extraction and restoration to generate restored spatial features;

[0018] Input the time feature in the initial spatiotemporal feature into the time feature compression network for feature compression to generate compressed time feature;

[0019] The restored spatial features and compressed temporal features are input into the spatiotemporal feature fusion network for feature fusion to generate fused spatiotemporal features;

[0020] The fused spatiotemporal features are input into the feature mapping network for feature mapping, and the Kth local prediction result is output.

[0021] According to an embodiment of the present invention, the spatiotemporal feature extraction basic network includes:

[0022] A spatial feature extraction unit, used for extracting three-dimensional spatial features from original input features;

[0023] A channel feature extraction unit, used to extract channel features from original input features;

[0024] The time feature extraction unit is used to extract the time features from the original input features.

[0025] According to an embodiment of the present invention, it also includes:

[0026] Transmitting a millimeter-wave radar transmission signal to the object under test, and receiving a plurality of millimeter-wave radar echo signals reflected by the irradiated area of ​​the object under test;

[0027] Performing first signal processing on the multiple millimeter-wave radar echo signals to obtain multiple first-stage signals, wherein the first signal processing includes: performing coherent accumulation operations on the multiple millimeter-wave radar echo signals defined by multiple channels composed of N transmitting antennas and M receiving antennas, and the first-stage signals include three-dimensional spatial features and time features;

[0028] Performing second signal processing on the plurality of first-stage signals to generate a second-stage signal for characterizing the center position of the human body of the measured object, wherein the second-stage signal includes a three-dimensional spatial feature;

[0029] Taking the predetermined neighborhood range of the spatial position point of the second-stage signal as the target three-dimensional spatial region, performing third signal processing on multiple millimeter-wave radar echo signals within the target three-dimensional spatial region to obtain multiple third-stage signals related to the mechanical movement of the heart part of the measured object, wherein the third-stage signals include three-dimensional spatial features and time features;

[0030] The fourth signal processing is performed on the plurality of third stage signals to eliminate the influence of the mechanical movement caused by the breathing of the measured object, and a plurality of reference signals related to the heartbeat movement of the heart part of the measured object are obtained.

[0031] According to an embodiment of the present invention, performing second signal processing on a plurality of first-stage signals to generate a second-stage signal for characterizing the center position of the human body of the measured object includes:

[0032] Accumulate the signal values ​​corresponding to each spatial position point in the multiple first-stage signals based on multiple time windows to generate multiple time accumulation signals corresponding to the multiple spatial position points one by one, wherein the time accumulation signals contain three-dimensional spatial features;

[0033] Calculating a signal difference value between each time-accumulated signal and a noise signal group, wherein the noise signal group is composed of a plurality of target signals within a predetermined three-dimensional neighborhood range of the time-accumulated signal, and the plurality of target signals are not directly adjacent to the time-accumulated signal;

[0034] According to the signal difference value corresponding to each time-accumulated signal, a second-stage signal for characterizing the center position of the human body of the measured object is determined from the multiple time-accumulated signals.

[0035] According to an embodiment of the present invention, performing coherent accumulation operation on a plurality of millimeter-wave radar echo signals defined by a plurality of channels includes adopting the following method:

[0036]

[0037] in: The i-th signal corresponding to the spatial position (x, y, z) and the signal acquisition period t among the multiple first-stage signals obtained after the coherent accumulation operation;

[0038] N is the number of receiving antennas;

[0039] M is the number of transmitting antennas;

[0040] T is the total number of sampling points in each frame period of the radar;

[0041] is the sampling point within the cycle;

[0042] is a plurality of millimeter wave radar echo signals, corresponding to the The channel defined by the mth receiving antenna and the mth transmitting antenna is in the acquisition period signal;

[0043] is the sampling period;

[0044] is the rate of change of the transmission frequency of the millimeter wave radar transmission signal;

[0045] For the transmitting antenna to the target location And return The round trip distance of each receiving antenna;

[0046] is the speed of light;

[0047] is the wavelength of the signal transmitted by the millimeter-wave radar.

[0048] According to an embodiment of the present invention, performing fourth signal processing on a plurality of third-stage signals comprises:

[0049] Performing second-order differential processing on the plurality of third-stage signals to eliminate the influence of mechanical movement caused by breathing of the measured object, wherein performing second-order differential processing on the plurality of third-stage signals includes:

[0050]

[0051] in:

[0052] It is the third stage signal;

[0053] For the i-th The i-th reference signal obtained by performing second-order difference processing; h is the period of each second-stage signal;

[0054] for The third stage signal of the nth sample before, n=1, 2, 3;

[0055] for Then the third stage signal of the nth sampling;

[0056] is the value of the time series that is i samples away.

[0057] A second aspect of the present invention provides a heart rate variability prediction device based on millimeter wave radar signals, comprising:

[0058] A reading module is used to read multiple reference signals related to the heartbeat movement of the heart part of the measured object from the data storage area, wherein the reference signals include three-dimensional spatial features and time features, the multiple reference signals correspond to multiple spatial position points, the reference signal of each spatial position point corresponds to multiple time windows, the multiple reference signals are obtained by processing multiple original signals, and the multiple original signals are: multiple millimeter wave radar echo signals reflected by the irradiated area of ​​the measured object;

[0059] The prediction module is used to predict multiple reference signals using a trained heart rhythm prediction model and output a target prediction result for characterizing the heart rhythm of the subject. The heart rhythm prediction model includes a spatiotemporal feature extraction basic network, a spatial feature compression network, a spatial feature restoration network, a temporal feature compression network, a spatiotemporal feature fusion network, and a feature mapping network.

[0060] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0061] The fourth aspect of the present invention further provides a computer-readable storage medium on which a computer program or instruction is stored, and the steps of the above method are implemented when the above computer program or instruction is executed by a processor.

[0062] The fifth aspect of the present invention also provides a computer program product, including a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0063] According to an embodiment of the present invention, the multiple reference signals stored in the data storage area are obtained by processing the original signal, wherein, in the signal processing stage, the first-stage signal is subjected to coarse-grained filtering to determine the position of the human body within a spatial range. Then, the signal is further fine-grained spatially positioned within a predetermined neighborhood range of the spatial position point of the second-stage signal to more accurately obtain the signal conditions of the area around the human body. It can be seen that the input signal is spatially filtered twice, firstly coarse-grained spatial positioning is performed to determine the position of the human body, and then fine-grained spatial positioning is performed, so that the signal conditions of the area around the human body can be more accurately obtained, thereby improving the signal-to-noise ratio of the input signal.

[0064] According to an embodiment of the present invention, multiple reference signals stored in a data storage area are read, and a spatiotemporal relationship between body reflection signals and heartbeats is established based on a pre-trained heart rhythm prediction model. By extracting cardiac mechanical activity signals and using the heart rhythm prediction model to map cardiac mechanical activity to cardiac electrical activity, electrocardiographic characterization prediction information is obtained, which may specifically be a heartbeat waveform.

[0065] Specifically, the original input features are input into the spatiotemporal feature extraction basic network through the spatiotemporal feature extraction basic network, and the time, space, and channel features are extracted through three networks respectively, so that various abstract features in the original signal can be more accurately extracted, and the heartbeat information can be accurately extracted to improve the accuracy of the prediction results; the spatial features are compressed through the spatial feature compression network to avoid the surge in calculation amount due to the increase in dimension; the compressed spatial features are restored through the spatial feature restoration network to prevent the damage to the features by the ordinary compression step and avoid the loss of important features; wherein, in the above feature compression process, the spatial features and the temporal features are isolated and processed separately, and targeted compression processing based on the respective feature types of multi-dimensional features is realized, and different compression strategies are adopted for each, so that the feature accuracy is guaranteed on the basis of reducing the complexity of feature processing. The temporal features are compressed through the temporal feature compression network to reduce the storage and calculation amount; the temporal features and spatial features are fused through the spatiotemporal feature fusion network, and the heartbeat waveform is obtained through feature mapping network mapping.

[0066] According to the embodiments of the present invention, based on the various networks in the heart rhythm prediction model of the present invention, the heart rhythm prediction model as a whole realizes the perception of mechanical activity of human chest heart micro-movements based on millimeter wave radar, and realizes the spatiotemporal relationship mapping of reflected signals and heartbeats. At the same time, the spatial distribution of the entire chest surface and the possible differences in signals at different positions are taken into account, and the mapping of various indicators of the mechanical activity of the human chest heart to electrical activity is completed, and non-contact fine-grained heart rate variability prediction is realized, as well as the automatic mapping processing of artificial intelligence. The prediction results are provided for user reference, providing a new information dimension and improving the user experience.

[0067] According to an embodiment of the present invention, the above method uses fine-grained three-dimensional spatial signal information for the first time to obtain and monitor heart rate variability (HRV). Compared with the traditional single-point signal processing method and the previous one-dimensional (distance) or two-dimensional (distance and angle) neural network input, this method takes into account more and more accurate information near the chest cavity, and realizes non-contact fine-grained heart rate variability prediction.

[0068] On the other hand, contactless detection based on millimeter-wave radar signals can avoid the discomfort caused by wearing electrodes and sensor devices, and can avoid cross-infection. Compared with traditional contact measurement methods, it greatly improves user experience and scope of application (for example, traditional measurement methods cannot be implemented due to poor physical health and cannot directly touch body parts, but this method is not limited by this). BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0070] Figure 1 A flowchart of a method for predicting heart rate variability based on millimeter-wave radar signals according to an embodiment of the present invention is schematically shown;

[0071] Figure 2 The appearance of a millimeter wave radar according to an embodiment of the present invention is schematically shown;

[0072] Figure 3 The flowchart of the method for obtaining a reference signal according to an embodiment of the present invention is schematically shown;

[0073] Figure 4 A schematic diagram of a network structure of a heartbeat rhythm prediction model according to an embodiment of the present invention is shown;

[0074] Figure 5 A schematic diagram of a scenario in which a subject is tested using the heart rate variability prediction method according to an embodiment of the present invention is schematically shown;

[0075] Figure 6 The appearance of an electrocardiogram monitoring device according to an embodiment of the present invention is schematically shown;

[0076] Figure 7 A diagram schematically shows a comparison of heartbeat waveforms of an embodiment of the present invention and a real electrocardiogram monitoring device;

[0077] Figure 8 A diagram schematically shows a comparison of IBI values ​​between an embodiment of the present invention and a real electrocardiogram monitoring device;

[0078] Fig. 9 The structure block diagram of the heart rate variability prediction device according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0079] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0080] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0081] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0082] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0083] Radio frequency signal monitoring is a non-contact monitoring technology that captures and analyzes tiny electromagnetic wave changes generated by cardiac activity wirelessly, thereby indirectly measuring heart rate and heart rate variability. However, in actual applications, the surface of the human chest is not a single point, but an area with spatial distribution. This means that the signal generated by cardiac activity will show different characteristics at different locations on the chest surface. Existing technologies only analyze chest motion information at a specific point (single point) without considering the spatial distribution of the entire chest surface. This method ignores the possible differences in signals at different locations. In addition, since only single-point information is analyzed, the acquired signal may contain more noise, resulting in an unstable signal-to-noise ratio, affecting the accuracy and reliability of HRV monitoring. Therefore, in order to accurately monitor HRV, it is necessary to consider the reflected signal of the entire body surface.

[0084] Radar sensors detect human motion by emitting radio waves and receiving reflected wave signals. However, the entire human torso surface (such as the chest and abdomen) reflects radar signals, making it difficult for radar sensors to separate useful heartbeat signals from only the surface motion caused by heart activity. That is, in addition to the surface motion caused by the heartbeat, the motion of other body parts will also generate reflected signals, which may cause confusion in the monitoring results. In addition, radio signals are interfered by many factors during propagation, such as environmental noise, signal interference from other electronic devices, etc. These noises and interferences affect the quality of the signal, making it difficult to accurately extract heartbeat information from radar signals.

[0085] Based on this, the present invention provides a heart rate variability prediction method based on millimeter wave radar signals performed by an electronic device, which obtains more comprehensive physiological information by considering the reflection signals from various parts of the body. Furthermore, a spatiotemporal relationship model between body reflection signals and heartbeats is established through a deep learning network model to perform non-contact HRV prediction.

[0086] Figure 1The flowchart of the heart rate variability prediction method based on millimeter wave radar signals according to an embodiment of the present invention is schematically shown. Figure 2 The appearance of the millimeter wave radar according to the embodiment of the present invention is schematically shown. Figure 1 and Figure 2 Specific explanation is given.

[0087] like Figure 1 As shown, the heart rate variability prediction method includes operations S110 to S120.

[0088] In operation S110, multiple reference signals related to the heartbeat movement of the heart part of the object under test are read from the data storage area, wherein the reference signals include three-dimensional spatial features and time features, the multiple reference signals correspond to multiple spatial position points, the reference signal of each spatial position point corresponds to multiple time windows, and the multiple reference signals are obtained by processing multiple original signals, and the multiple original signals are: multiple millimeter-wave radar echo signals reflected by the irradiated area of ​​the object under test.

[0089] According to an embodiment of the present invention, Figure 2 As shown, the millimeter wave radar can be a multi-transmitter and multi-receiver device, including N transmitting antennas and M receiving antennas. For example, it includes 3 transmitting antennas and 4 receiving antennas. The specific signal collection method can be, for example, a human body lying flat on the bed, using Figure 2 The millimeter-wave radar shown transmits millimeter-wave signals directly at the human chest and receives echo signals.

[0090] According to an embodiment of the present invention, a millimeter wave radar is used to transmit a millimeter wave signal to the object under test. After the object under test receives the millimeter wave signal, the radiation area of ​​the object under test can reflect the received millimeter wave signal to obtain multiple original signals. Among them, the radiation area can be the radiation range of the millimeter wave radar. For example, it includes but is not limited to the chest area of ​​the object under test. Furthermore, in addition to the target signal required for detection, that is, the millimeter wave radar echo signal generated by the heartbeat movement, the multiple original signals are also mixed with a variety of interference signals, such as the millimeter wave radar echo signal generated by breathing, the millimeter wave radar echo signal generated by other radiation areas except the heart part, etc. Therefore, it is necessary to process the multiple original signals to obtain multiple reference signals for characterizing the law of the heartbeat movement of the heart part of the object under test.

[0091] According to an embodiment of the present invention, the processed multiple reference signals are stored in a data storage area, and when the heart rate variability prediction method is applied, the relevant reference signals are directly read from the data storage area. Each reference signal includes three-dimensional spatial features and time features, the multiple reference signals correspond to multiple spatial position points, and the reference signal of each spatial position point corresponds to multiple time windows.

[0092] In operation S120, a plurality of reference signals are predicted using a trained heart rhythm prediction model, and a target prediction result for characterizing the heart rhythm of the subject is output, wherein the heart rhythm prediction model includes a spatiotemporal feature extraction basic network, a spatial feature compression network, a spatial feature restoration network, a temporal feature compression network, a spatiotemporal feature fusion network, and a feature mapping network.

[0093] According to an embodiment of the present invention, the target prediction result for characterizing the heart rhythm can reflect the heart rate of the subject, that is, the number of heartbeats per minute. The heart rate of the subject and its regularity indicate whether the heart beats within the normal rate range and whether the heart beats uniformly.

[0094] According to an embodiment of the present invention, multiple reference signals stored in a data storage area are read, and a spatiotemporal relationship between body reflection signals and heartbeats is established based on a pre-trained heart rhythm prediction model. By extracting cardiac mechanical activity signals and using the heart rhythm prediction model to map cardiac mechanical activity to cardiac electrical activity, electrocardiographic characterization prediction information is obtained, which may specifically be a heartbeat waveform.

[0095] Specifically, the original input features are input into the spatiotemporal feature extraction basic network through the spatiotemporal feature extraction basic network, and the time, space, and channel features are extracted through three networks respectively, so that various abstract features in the original signal can be more accurately extracted, and the heartbeat information can be accurately extracted to improve the accuracy of the prediction results; the spatial features are compressed through the spatial feature compression network to avoid the surge in calculation amount due to the increase in dimension; the compressed spatial features are restored through the spatial feature restoration network to prevent the damage to the features by the ordinary compression step and avoid the loss of important features; wherein, in the above feature compression process, the spatial features and the temporal features are isolated and processed separately, and targeted compression processing based on the respective feature types of multi-dimensional features is realized, and different compression strategies are adopted for each, so that the feature accuracy is guaranteed on the basis of reducing the complexity of feature processing. The temporal features are compressed through the temporal feature compression network to reduce the storage and calculation amount; the temporal features and spatial features are fused through the spatiotemporal feature fusion network, and the heartbeat waveform is obtained through feature mapping network mapping.

[0096] According to the embodiments of the present invention, based on the various networks in the heart rhythm prediction model of the present invention, the heart rhythm prediction model as a whole realizes the perception of mechanical activity of human chest heart micro-movements based on millimeter wave radar, and realizes the spatiotemporal relationship mapping of reflected signals and heartbeats. At the same time, the spatial distribution of the entire chest surface and the possible differences in signals at different positions are taken into account, and the mapping of various indicators of the mechanical activity of the human chest heart to electrical activity is completed, and non-contact fine-grained heart rate variability prediction is realized, as well as the automatic mapping processing of artificial intelligence. The prediction results are provided for user reference, providing a new information dimension and improving the user experience.

[0097] According to an embodiment of the present invention, the above method uses fine-grained three-dimensional spatial signal information for the first time to obtain and monitor heart rate variability (HRV). Compared with the traditional single-point signal processing method and the previous one-dimensional (distance) or two-dimensional (distance and angle) neural network input, this method takes into account more and more accurate information near the chest cavity, and realizes non-contact fine-grained heart rate variability prediction.

[0098] On the other hand, contactless detection based on millimeter-wave radar signals can avoid the discomfort caused by wearing electrodes and sensor devices, and can avoid cross-infection. Compared with traditional contact measurement methods, it greatly improves user experience and scope of application (for example, traditional measurement methods cannot be implemented due to poor physical health and cannot directly touch body parts, but this method is not limited by this).

[0099] It should be noted that the heart rhythm prediction results output by the heart rhythm prediction model have similar information representation functions as the traditional heart rate variability detection results. The prediction results are only used as reference information for medical staff to make diagnoses. They are used to obtain intermediate results of processing information parameters and are not used as direct diagnosis results. In addition, all steps of the above method are information processing methods implemented by computers and other devices. The whole process does not require the participation of medical personnel. The direct purpose is not to obtain diagnosis results or health status. The information obtained itself cannot directly derive the diagnosis results of the disease or health status.

[0100] Figure 3 The flowchart of the method for acquiring a reference signal according to an embodiment of the present invention is schematically shown.

[0101] like Figure 3 As shown, obtaining a plurality of reference signals for characterizing the heartbeat movement regularity of the subject's heart includes operations S310 to S350.

[0102] In operation S310, a millimeter wave radar transmission signal is transmitted to an object under measurement, and a plurality of millimeter wave radar echo signals reflected by an irradiated area of ​​the object under measurement are received.

[0103] According to the embodiments of the present invention, a virtual antenna array can be constructed based on the physical arrangement of the millimeter wave radar antenna, and a phase shift vector can be constructed according to the antenna spacing and signal bandwidth in the virtual antenna array, and spatial beamforming can be calculated to complete the spatial filtering of the millimeter wave radar echo signal. By using the array antenna of the radio frequency equipment and the frequency modulated continuous wave system, the propagation distance of the reflected signal at different positions of the chest can be mapped to the phase change, and finally the reflected signal at all positions in the entire chest can be restored by performing coherent accumulation operations on the received signals in the virtual channels composed of different transmitting and receiving antennas.

[0104] In operation S320, first signal processing is performed on the plurality of millimeter-wave radar echo signals to obtain a plurality of first-stage signals.

[0105] According to an embodiment of the present invention, after receiving the echo of the radar transmission signal, preliminary coarse-grained spatial domain filtering is performed through the first signal processing, and multiple first-stage signals for characterizing the mechanical vibration law of the surface of the irradiated area of ​​the measured object can be obtained to remove or suppress signals from non-target areas, thereby improving the quality of the signal and making subsequent signal analysis more accurate. Among them, the first signal processing includes: coherent accumulation operations are performed on multiple millimeter-wave radar echo signals defined by multiple channels composed of N transmitting antennas and M receiving antennas, and the first-stage signals contain three-dimensional spatial features and time features.

[0106] According to an embodiment of the present invention, a coherent accumulation operation is performed on a plurality of millimeter-wave radar echo signals defined by a plurality of channels as shown in formula (1):

[0107] (1)

[0108] in: The i-th signal corresponding to the spatial position (x, y, z) and the signal acquisition period t among the multiple first-stage signals obtained after the coherent accumulation operation;

[0109] N is the number of receiving antennas;

[0110] M is the number of transmitting antennas;

[0111] T is the total number of sampling points in each frame period of the radar;

[0112] is the sampling point within the cycle;

[0113] is a plurality of millimeter wave radar echo signals, corresponding to the The channel defined by the mth receiving antenna and the mth transmitting antenna is in the acquisition period signal;

[0114] j is the representation of imaginary number in mathematics;

[0115] is the sampling period;

[0116] is the rate of change of the transmission frequency of the millimeter wave radar transmission signal;

[0117] For the transmitting antenna to the target location And return The round trip distance of each receiving antenna;

[0118] is the speed of light;

[0119] is the wavelength of the signal transmitted by the millimeter-wave radar.

[0120] In operation S330, a second signal processing is performed on the plurality of first-stage signals to generate a second-stage signal for characterizing the center position of the human body of the measured object. The second-stage signal contains three-dimensional spatial features, specifically including steps 11 to 13.

[0121] In step 11, the signal values ​​corresponding to each spatial position point in the plurality of first-stage signals are feature accumulated based on a plurality of time windows to generate a plurality of time-accumulated signals corresponding to the plurality of spatial position points one by one, wherein the time-accumulated signals include three-dimensional spatial features. Here, by performing feature accumulation based on a plurality of time windows, the generated second-stage signal only includes three-dimensional spatial features and does not include time features.

[0122] In step 12, a signal difference value between each time-accumulated signal and a noise signal group is calculated, wherein the noise signal group is composed of a plurality of target signals within a predetermined three-dimensional neighborhood of the time-accumulated signal, and the plurality of target signals are not directly adjacent to the time-accumulated signal. Signals that are not directly adjacent to the time-accumulated signal are treated as noise signals, so that the noise signals can be reasonably distinguished, thereby facilitating improved calculation accuracy.

[0123] In step 13, a second stage signal for characterizing the body center position of the measured object is determined from the multiple time-accumulated signals according to the signal difference value corresponding to each time-accumulated signal.

[0124] According to an embodiment of the present invention, a three-dimensional unit average-constant false alarm rate (CA-CFAR) algorithm is used to adaptively detect the exact position of the human chest. First, based on multiple time windows, the signal amplitude in the time dimension is added to enhance the signal and reduce the influence of noise, thereby improving the accuracy of subsequent signal extraction; then the noise level of each point is estimated by the difference between the signal amplitude in the three-dimensional space and the noise signal in the surrounding neighborhood. Among them, the predetermined three-dimensional neighborhood range can be set based on experience, and the larger the signal difference value between the time accumulated signal and the noise signal group, the higher the possibility that it may belong to the human body position. The detection threshold is adjusted according to the estimated noise level to determine when the signal amplitude exceeds the background noise, so as to determine that the position where the object is most likely to be is the rough position of the human body.

[0125] In operation S340, the predetermined neighborhood range of the spatial position point of the second-stage signal is used as the target three-dimensional spatial area, and the multiple millimeter-wave radar echo signals within the target three-dimensional spatial area are subjected to third signal processing to obtain multiple third-stage signals related to the mechanical movement of the heart part of the measured object, wherein the third-stage signal contains three-dimensional spatial features and time features. The third signal processing is used to find the exact position (x, y, z) of the human body based on the second-stage signal. Specifically, it includes processing the original signal using the method of formula (1) to obtain the spatial information of precise positioning. In the above operation, the multiple time accumulation signals obtained by feature accumulation based on multiple time windows are used to determine the approximate range of the position (x, y, z) of the human body. The above-mentioned method for processing the signal is: firstly process the original signal with the rough spatial position x, y, z values ​​to obtain the rough positioning data. Then, the time dimension of the amplitude value of the rough positioning data is summed, and only the superposition information on the rough positioning data space is retained. Then, according to the superposition information on the rough positioning data space, the accurate x, y, z values ​​of the human body are found, and the original signal is processed using formula (1) to obtain the spatial information of precise positioning.

[0126] The predetermined neighborhood range may be a spatial range set based on experience for characterizing the heart position, for example, the predetermined neighborhood range is 1.5×1.5×1.5m (for rough positioning), and the further predetermined neighborhood range is 0.5m×0.5m×0.5m (for precise positioning). The third stage signal contains both three-dimensional spatial features and time features.

[0127] According to an embodiment of the present invention, the position of the human body is determined within a spatial range by performing coarse-grained filtering on the first-stage signal. Then, the signal is further fine-grained spatially positioned within a predetermined neighborhood range of the spatial position point of the second-stage signal to more accurately obtain the signal conditions of the area around the human body. It can be seen that the input signal is spatially filtered twice, firstly performing coarse-grained spatial positioning to determine the position of the human body, and then performing fine-grained spatial positioning, so that the signal conditions of the area around the human body can be more accurately obtained, thereby improving the signal-to-noise ratio of the input signal.

[0128] In operation S350, a fourth signal processing is performed on the plurality of third stage signals to eliminate the influence of mechanical motion caused by the breathing of the subject, and obtain a plurality of reference signals related to the heartbeat motion of the heart of the subject.

[0129] According to an embodiment of the present invention, the non-cardiac part in the irradiated area also has body surface vibration, so it is necessary to perform fourth signal processing on multiple third stage signals to eliminate the influence of the body surface vibration of the non-cardiac part in the irradiated area.

[0130] According to an embodiment of the present invention, performing fourth signal processing on a plurality of third-stage signals includes: performing second-order difference processing on the plurality of third-stage signals to eliminate the influence of mechanical movement caused by breathing of the measured object, wherein the second-order difference processing is performed on the plurality of third-stage signals, specifically referring to formula (2):

[0131] (2)

[0132] in, It is the third stage signal;

[0133] For the i-th The i-th reference signal obtained by performing second-order difference processing; h is the period of each second-stage signal;

[0134] for The third stage signal of the nth sample before, n=1, 2, 3;

[0135] for Then the third stage signal of the nth sampling;

[0136] is the value of the time series that is i samples away.

[0137] According to an embodiment of the present invention, by eliminating the influence of body surface vibration caused by breathing and body surface vibration of non-heart parts in the millimeter wave radar echo signal, the multiple reference signals finally obtained can more accurately characterize the law of heartbeat movement of the heart part of the measured object.

[0138] According to an embodiment of the present invention, by adopting fine-grained three-dimensional spatial signal information to acquire and monitor heart rate variability (HRV), the prediction method of the embodiment of the present invention contains more and more accurate information near the chest cavity compared to the traditional single-point signal processing method and one-dimensional (distance) or two-dimensional (distance and angle) neural network input.

[0139] Furthermore, after obtaining a plurality of reference signals, they can be stored in a data storage area for subsequent use, or directly input into a heart rhythm prediction model to obtain a target prediction result for characterizing the heart rhythm of the subject.

[0140] Figure 4 The network structure diagram of the heartbeat rhythm prediction model according to an embodiment of the present invention is schematically shown.

[0141] like Figure 4 As shown in the figure, the heart rhythm prediction model includes a basic network for spatiotemporal feature extraction (Basic Blocks network), a spatial feature compression network (compress1 network), a spatial feature restoration network (transformer encoder1), a temporal feature compression network (compress2 network), a spatiotemporal feature fusion network (transformer encoder2), and a feature mapping network (MLP).

[0142] According to an embodiment of the present invention, predicting multiple reference signals using a heartbeat rhythm prediction model includes performing multiple prediction operations, in which the heartbeat rhythm prediction model outputs a set of local prediction results based on the original input feature signal, and the multiple sets of local prediction results corresponding to the multiple prediction operations are collectively used as the target prediction results. Exemplarily, a set of original input feature signals with a duration of 10s is input and a set of local prediction results with a duration of 10s is output. When the number of prediction operations is 5, the 5 local prediction results with a duration of 10s are spliced ​​to obtain the target prediction result.

[0143] According to an embodiment of the present invention, the number of prediction operations may be set based on experience, such as Figure 4As shown in the figure, for example, if the number of prediction operations is set to 3, then the original input features need to be input 3 times to obtain 3 sets of local prediction results, and the 3 sets of local prediction results corresponding to the 3 prediction operations are taken as the target prediction results. The feature dimensions of the original input features include three-dimensional spatial feature dimensions (for example, represented by H, W, L), time feature dimensions (for example, represented by T), and channel feature dimensions (for example, Figure 3 value "2" in the .

[0144] Specifically, the reference signal is input into the heart rhythm prediction model, wherein the application process of the prediction model is explained by taking the Kth prediction operation among multiple prediction operations as an example, which specifically includes step 11 and step 12.

[0145] In step 11, some reference signals are selected from multiple reference signals as K-th selection signals, and the K-th selection signals and the K-1-th local prediction results output by the heartbeat rhythm prediction model are feature concatenated to generate the K-th original input features.

[0146] According to an embodiment of the present invention, K is a positive integer greater than or equal to 1. Some reference signals are selected from multiple reference signals as the Kth selection signal. Exemplarily, a reference signal with a duration of 0 to 10 seconds is obtained, 6 seconds of which are selected as the selection signal and input into the heartbeat rhythm prediction model.

[0147] Exemplarily, when K is equal to 1, that is, the selection signal is input into the heart rhythm prediction model for the first time, the K-1th local prediction result cannot be obtained, that is, the K-1th local prediction result is set to 0, and the first original input feature is generated.

[0148] When K is equal to 2, that is, the selection signal is input into the heart rhythm prediction model for the second time, the local prediction result of the first (last time), that is, the original input feature of the first time, can be obtained, and the feature splicing with the selection signal to be input for the second time is performed as the second input to generate the second original input feature.

[0149] According to an embodiment of the present invention, in step 11, feature splicing of the Kth selection signal and the K-1th local prediction result output by the heartbeat rhythm prediction model includes:

[0150] The K-th selection signal is feature concatenated based on the three-dimensional spatial feature dimension and the time feature dimension, and the K-1-th local prediction result is feature concatenated based on the channel feature dimension to generate the K-th original input feature.

[0151] In step 12, the K-th original input feature is input into the heart rhythm prediction model, and the K-th local prediction result is output, including steps 121 to 126.

[0152] In step 121, the Kth original input feature is input into the spatiotemporal feature extraction basic network for preliminary extraction of spatiotemporal features to generate initial spatiotemporal features.

[0153] like Figure 4 As shown, the basic network for spatiotemporal feature extraction is a Basic Blocks network, and the basic network for spatiotemporal feature extraction (Basic Blocks network) includes a spatial feature extraction unit, a channel feature extraction unit, and a temporal feature extraction unit. The spatial feature extraction unit is used to extract three-dimensional spatial features from the original input features, such as a 3D CNN model; the channel feature extraction unit is used to extract channel features from the original input features, such as using a bottleneck-shaped two-layer full connection to extract channel information; the temporal feature extraction unit is used to extract temporal features from the original input features, such as using a bottleneck-shaped two-layer full connection to extract temporal information. Furthermore, through the basic network for spatiotemporal feature extraction, the three-dimensional spatial features, channel features, and temporal features in the selected signal can be extracted respectively to obtain the initial spatiotemporal features.

[0154] According to an embodiment of the present invention, in step 121, by separating the spatial features, time features and channel features, and using three networks to extract the time, space and channel features respectively, and by processing the feature separation, various abstract features in the original signal can be more accurately extracted, so that it can adapt to various complex scenarios, such as processing data with large distribution differences, accurately extracting heartbeat information, and improving the accuracy of prediction results.

[0155] In step 122, the time features in the initial space-time features are isolated, and the space features in the initial space-time features are input into a space feature compression network for feature compression to generate compressed space features.

[0156] like Figure 3 As shown, the spatial feature compression network is the compress1 network, which puts the time features in the initial spatiotemporal features into batch processing for isolation. At the same time, only the spatial features are input into the spatial feature compression network (compress1 network) for feature compression to generate compressed spatial features.

[0157] In step 123, the compressed spatial features are input into the spatial feature restoration network for feature extraction and restoration to generate restored spatial features. Figure 3 As shown, the spatial feature restoration network is Transformer encoder 1. For example, a 6-layer transformer encoder 1 is used to extract compressed spatial information.

[0158] According to an embodiment of the present invention, in steps 122 to 123, by compressing the spatial features, a surge in calculation amount due to an increase in dimension is avoided, and the compressed spatial features are restored, thereby preventing damage to the features by ordinary compression steps and avoiding the loss of important features.

[0159] In step 124, the time features in the initial spatiotemporal features are input into the time feature compression network for feature compression to generate compressed time features. Figure 3 As shown in Figure 1, the temporal feature compression network is a compress2 network. The temporal dimension can be compressed to a size of 32 so that the subsequent spatiotemporal mixing module can extract spatiotemporal features.

[0160] In step 125, the restored spatial features and compressed temporal features are input into the spatiotemporal feature fusion network for feature fusion to generate fused spatiotemporal features. Figure 3 As shown, the spatiotemporal feature fusion network is a Transformer encoder 2. For example, a 6-layer transformer encoder 2 is used to perform feature fusion on the restored spatial features and compressed temporal features. The spatiotemporal feature fusion network includes a common spatiotemporal feature extraction unit and a global spatiotemporal feature extraction unit. The common spatiotemporal feature extraction unit contains the spatiotemporal information extracted from the spatiotemporal feature extraction basic network. The global spatiotemporal feature extraction unit is used to compress and extract features related to the HRV monitoring task from all common spatiotemporal feature extraction units.

[0161] In step 126, the fused spatiotemporal features are input into the feature mapping network for feature mapping, and the Kth local prediction result is output. Figure 3 As shown in the figure, the feature mapping network is a multilayer perceptron (MLP). Two layers of multilayer perceptrons are used to map the extracted spatiotemporal features to cardiac activity. In addition, 2 seconds of historical data will be spliced ​​to the signal output after zero padding and repetition to form the input signal of the network.

[0162] Furthermore, the local prediction results obtained multiple times are concatenated and taken as the target prediction result.

[0163] According to an embodiment of the present invention, based on a deep learning architecture that has learned the spatiotemporal relationship mapping between radar signals and cardiac activities, the extracted chest and abdomen radar cardiac micromotion data are input to output the predicted results of human heart rhythm activity, so as to realize heart rhythm prediction through the heart rhythm prediction model of the present invention.

[0164] According to an embodiment of the present invention, in the above-mentioned feature compression process, spatial features and temporal features are isolated and processed separately (temporal features are isolated in the spatial feature compression process, and spatial features are isolated in the temporal feature compression process), thereby achieving targeted compression processing based on the feature types of each multi-dimensional feature. Different compression strategies are adopted for each feature, thereby ensuring feature accuracy while reducing the complexity of feature processing.

[0165] The present invention is further described below by embodiments and related test experiments. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present invention. However, it is obvious that one or more embodiments may also be implemented without these specific details. And, in the absence of conflict, the details in the following embodiments may be combined into other feasible embodiments at will.

[0166] Figure 5 The schematic diagram schematically shows a scenario of testing a subject using the heart rate variability prediction method according to an embodiment of the present invention.

[0167] like Figure 5 As shown, the subject lies flat on the bed, and the millimeter wave radar transmits and collects signals 0.4-0.5m above the human chest. In the experiment, the radar is in a 3-transmit 4-receive working state, and the starting frequency of the signal is set to 77GHz, the bandwidth is 4GHz, and the frame rate is 100Hz. The signal is collected for 30s, and the heart rate variability prediction method of the embodiment of the present invention is used to output the ECG representation prediction information.

[0168] Figure 6 The external view of the electrocardiogram monitoring device according to an embodiment of the present invention is schematically shown. Figure 6 The device shown collects heart rate changes, specifically, by Figure 6 The schematic diagram of the scene of the ECG monitoring device performing ECG detection is the same as the prior art and is not shown in the figure. Figure 6 The ECG monitoring device shown collects the ECG of the subject, and the output of the ECG device is 500HZ. In the experiment, the subject lies flat on the bed and collects data for 30s.

[0169] Figure 7 The schematic diagram shows the comparison of the heartbeat waveform of the embodiment of the present invention and the real ECG monitoring device. The heartbeat waveform generated by the heart rate variability prediction method in the embodiment of the present invention is as follows: Figure 7 As shown by the blue line in the middle. Figure 6 The heartbeat waveform generated by the conventional ECG monitoring device shown is as follows Figure 7 The orange line in the middle and the dashed black line (R wave) are used to represent the contraction of the heart.

[0170] Figure 8The diagram schematically shows the IBI value comparison between the embodiment of the present invention and a real ECG monitoring device. The inter-beat interval (IBI) refers to the time interval between two consecutive heartbeats. By calculating the time difference between consecutive heartbeats, the value of IBI can be obtained. The value of IBI is calculated by using the heartbeat waveform generated by the heart rate variability prediction method in the embodiment of the present invention. Figure 8 As shown by the blue line in the middle. Figure 6 The value of IBI is calculated by the heartbeat waveform generated by the conventional ECG monitoring device shown in FIG. Figure 8 Shown in black lines.

[0171] like Figure 7 and Figure 8 As shown in FIG. 1 , it can be seen that the prediction result obtained by using the heart rate variability prediction method in the embodiment of the present invention is different from that obtained by using Figure 6 The results detected by the conventional ECG monitoring equipment shown are relatively close, that is, the heart rate variability prediction method of the present invention is close to the actual detection results, and its prediction results have reference value.

[0172] According to an embodiment of the present invention, the overall performance of the model in the embodiment of the present invention is measured by the ratio between the absolute heartbeat interval error and the true value, and the following three indicators commonly used in heart rate variability analysis are used: the square root of the sum of squares of consecutive heartbeat interval differences (Root Mean Square of the Successive Differences, RMSSD), the standard net difference of consecutive normal RR intervals (Standard Deviation of the Differences Between Adjacent Normal R-R Intervals, SDSD), and the logarithmic proportion of consecutive normal heartbeat interval differences exceeding 50 milliseconds (Proportion of Pairs of Successive NN Intervals that Differ by More than 50ms, pNN50). The above three indicators are used to measure the monitoring ability of the model in the embodiment of the present invention on heart rate variability.

[0173] Specifically, as shown in Table 1, the medians and means of the four indicators IBI, RMSSD, SDSD, and pNN50 were obtained by calculation.

[0174] Table 1

[0175]

[0176] According to the embodiments of the present invention, the experimental results are as follows: Figure 8As shown in Table 1. From the above results, it can be seen that this embodiment has achieved relatively accurate monitoring performance in monitoring heart rate variability in a variety of test cases such as sinus arrhythmia, sinus tachycardia, sinus bradycardia, ST segment abnormality, sinus rhythm, and completely normal heart rate. The smaller the error value, the higher the consistency between the disclosed embodiment and the conventional ECG method. Figure 8 As shown, with 800ms as the standard IBI interval, the relative errors of various diseases measured according to the embodiments of the present disclosure are all within 4% or less, indicating that the results measured by the embodiments of the present disclosure are highly consistent with the standard implementation, further proving the accuracy of the heart rate variability prediction method in the embodiments of the present disclosure.

[0177] Fig. 9 The structure block diagram of the heart rate variability prediction device according to an embodiment of the present invention is schematically shown.

[0178] like Fig. 9 As shown, the heart rate variability prediction device 900 of this embodiment includes a reading module 910 and a prediction module 920 .

[0179] The reading module 910 is used to read multiple reference signals related to the heartbeat movement of the heart part of the measured object from the data storage area, wherein the reference signal includes three-dimensional spatial features and time features, multiple reference signals correspond to multiple spatial position points, and the reference signal of each spatial position point corresponds to multiple time windows. The multiple reference signals are obtained by processing multiple original signals, and the multiple original signals are: multiple millimeter wave radar echo signals reflected by the irradiated area of ​​the measured object. In one embodiment, the reading module 910 can be used to perform the operation S110 described above, which will not be repeated here.

[0180] The prediction module 920 is used to predict multiple reference signals using a trained heart rhythm prediction model, and output a target prediction result for characterizing the heart rhythm of the subject, wherein the heart rhythm prediction model includes a spatiotemporal feature extraction basic network, a spatial feature compression network, a spatial feature restoration network, a temporal feature compression network, a spatiotemporal feature fusion network, and a feature mapping network. In one embodiment, the prediction module 920 can be used to perform the operation S120 described above, which will not be repeated here.

[0181] According to an embodiment of the present invention, the heart rhythm prediction model establishes a spatiotemporal relationship model between body reflection signals and heartbeats through a deep learning network model. During the training process, an end-to-end deep learning network is constructed using the extracted cardiac mechanical activity signals, i.e., the above-mentioned heart rhythm prediction model is constructed. The heart rhythm prediction model is used to complete the mapping of cardiac mechanical activity to cardiac electrical activity, and obtain electrocardiogram representation prediction information, which can specifically be a heartbeat waveform.

[0182] According to an embodiment of the present invention, through the above-mentioned device, signal processing technology is used in combination with an end-to-end deep learning network to realize the perception of mechanical activity of human chest heart micro-movements based on millimeter wave radar, and to realize the spatiotemporal relationship mapping of reflected signals and heartbeats. The spatial distribution of the entire chest surface and the possible differences in signals at different positions are taken into account, and the mapping of various indicators of human chest heart mechanical activity to electrical activity is completed, and non-contact fine-grained heart rate variability prediction is realized, as well as artificial intelligence automated mapping processing. The prediction results are provided for user reference, providing a new information dimension and improving user experience.

[0183] On the other hand, contactless detection based on millimeter-wave radar signals can avoid the discomfort caused by wearing electrodes and sensor devices, and can avoid cross-infection. Compared with traditional contact measurement methods, it greatly improves user experience and scope of application (for example, traditional measurement methods cannot be implemented due to poor physical health and cannot directly touch body parts, but this method is not limited by this).

[0184] According to an embodiment of the present invention, in each prediction operation, the heart rhythm prediction model outputs a set of local prediction results based on the original input features, and multiple sets of local prediction results corresponding to multiple prediction operations are jointly used as target prediction results, and the feature dimensions of the original input features include three-dimensional spatial feature dimensions, time feature dimensions, and channel feature dimensions; wherein the prediction module 920 includes a selection submodule and an output submodule.

[0185] A selection submodule is used to select some reference signals from multiple reference signals as K-th selection signals, and perform feature splicing on the K-th selection signal and the K-1-th local prediction result output by the heartbeat rhythm prediction model to generate the K-th original input feature;

[0186] The output submodule is used to input the K-th original input features into the heartbeat rhythm prediction model and output the K-th local prediction results.

[0187] According to an embodiment of the present invention, the selection submodule includes a feature splicing unit.

[0188] The feature concatenation unit is used to concatenate the K-th selection signal based on the three-dimensional spatial feature dimension and the time feature dimension, and to concatenate the K-1-th local prediction result based on the channel feature dimension, so as to generate the K-th original input feature.

[0189] According to an embodiment of the present invention, the output submodule includes an input unit, an isolation unit, a feature restoration unit, a feature compression unit, a feature fusion unit and a feature mapping unit.

[0190] An input unit, used for inputting the Kth original input feature into the spatiotemporal feature extraction basic network to perform preliminary extraction of spatiotemporal features and generate initial spatiotemporal features;

[0191] An isolation unit is used to isolate the time features in the initial space-time features, and input the space features in the initial space-time features into a space feature compression network for feature compression to generate compressed space features;

[0192] A feature restoration unit, used for inputting the compressed spatial features into the spatial feature restoration network for feature extraction and restoration, and generating restored spatial features;

[0193] A feature compression unit, used for inputting the time feature in the initial spatiotemporal feature into the time feature compression network for feature compression to generate a compressed time feature;

[0194] A feature fusion unit is used to input the restored spatial features and the compressed temporal features into the spatiotemporal feature fusion network for feature fusion to generate fused spatiotemporal features;

[0195] The feature mapping unit is used to input the fused spatiotemporal features into the feature mapping network for feature mapping and output the Kth local prediction result.

[0196] According to an embodiment of the present invention, the spatiotemporal feature extraction basic network includes:

[0197] A spatial feature extraction unit, used for extracting three-dimensional spatial features from original input features;

[0198] A channel feature extraction unit, used to extract channel features from original input features;

[0199] The time feature extraction unit is used to extract the time features from the original input features.

[0200] According to an embodiment of the present invention, the heart rate variability prediction device further includes a transmitting module, a first processing module, a second processing module, a third processing module and a fourth processing module.

[0201] A transmitting module, used to transmit a millimeter-wave radar transmission signal to the object under test, and receive a plurality of millimeter-wave radar echo signals reflected by the irradiated area of ​​the object under test;

[0202] A first processing module is used to perform first signal processing on multiple millimeter-wave radar echo signals to obtain multiple first-stage signals, wherein the first signal processing includes: performing coherent accumulation operations on multiple millimeter-wave radar echo signals defined by multiple channels composed of N transmitting antennas and M receiving antennas, and the first-stage signals include three-dimensional spatial features and time features;

[0203] A second processing module is used to perform second signal processing on the plurality of first-stage signals to generate a second-stage signal for characterizing the center position of the human body of the measured object, wherein the second-stage signal includes a three-dimensional spatial feature;

[0204] A third processing module is used to use a predetermined neighborhood range of the spatial position point of the second-stage signal as a target three-dimensional spatial area, and perform third signal processing on multiple millimeter-wave radar echo signals within the target three-dimensional spatial area to obtain multiple third-stage signals related to the mechanical movement of the heart part of the measured object, wherein the third-stage signals include three-dimensional spatial features and time features;

[0205] The fourth processing module is used to perform fourth signal processing on the plurality of third stage signals to eliminate the influence of mechanical movement caused by the breathing of the measured object and obtain a plurality of reference signals related to the heartbeat movement of the heart part of the measured object.

[0206] According to an embodiment of the present invention, the second processing module includes a feature accumulation submodule, a calculation submodule and a determination submodule.

[0207] A feature accumulation submodule, used for performing feature accumulation of the signal values ​​corresponding to each spatial position point in the multiple first-stage signals based on multiple time windows, to generate multiple time accumulation signals corresponding to the multiple spatial position points one by one, wherein the time accumulation signals contain three-dimensional spatial features;

[0208] A calculation submodule, used for calculating a signal difference value between each time-accumulated signal and a noise signal group, wherein the noise signal group is composed of a plurality of target signals within a predetermined three-dimensional neighborhood range of the time-accumulated signal, and the plurality of target signals are not directly adjacent to the time-accumulated signal;

[0209] The determination submodule is used to determine the second stage signal for characterizing the body center position of the measured object from the multiple time accumulation signals according to the signal difference value corresponding to each time accumulation signal.

[0210] According to an embodiment of the present invention, any multiple modules in the reading module 910 and the prediction module 920 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the reading module 910 and the prediction module 920 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the reading module 910 and the prediction module 920 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0211] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination. The scope of the present invention is defined by the appended claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A heart rate variability prediction method based on millimeter wave radar signals performed by an electronic device, comprising: Reading multiple reference signals related to the heartbeat movement of the heart of the object under test from the data storage area, wherein the reference signals include three-dimensional spatial features and time features, the multiple reference signals correspond to multiple spatial position points, the reference signal of each spatial position point corresponds to multiple time windows, the multiple reference signals are obtained by processing multiple original signals, and the multiple original signals are: multiple millimeter wave radar echo signals reflected by the irradiated area of ​​the object under test; Using the trained heartbeat rhythm prediction model to predict the multiple reference signals, outputting a target prediction result for characterizing the heartbeat rhythm of the subject, wherein the heartbeat rhythm prediction model includes a spatiotemporal feature extraction basic network, a spatial feature compression network, a spatial feature restoration network, a temporal feature compression network, a spatiotemporal feature fusion network, and a feature mapping network; Wherein: predicting the multiple reference signals using the heartbeat rhythm prediction model includes performing multiple prediction operations, in each prediction operation, the heartbeat rhythm prediction model outputs a set of local prediction results based on the original input features, and the multiple sets of local prediction results corresponding to the multiple prediction operations are collectively used as the target prediction results, and the feature dimensions of the original input features include three-dimensional spatial feature dimensions, time feature dimensions, and channel feature dimensions; The Kth prediction operation in the multiple prediction operations includes: selecting a part of the reference signals from the multiple reference signals as the Kth selection signal, and performing feature splicing on the Kth selection signal and the K-1th local prediction result output by the heartbeat rhythm prediction model to generate the Kth original input feature; inputting the Kth original input feature into the heartbeat rhythm prediction model to output the Kth local prediction result; Among them, feature splicing the Kth selection signal and the K-1th local prediction result output by the heartbeat rhythm prediction model includes: feature splicing the Kth selection signal based on the three-dimensional spatial feature dimension and the time feature dimension, and feature splicing the K-1th local prediction result based on the channel feature dimension to generate the Kth original input feature; Among them, inputting the K-th original input feature into the heartbeat rhythm prediction model, and outputting the K-th local prediction result includes: inputting the K-th original input feature into the spatiotemporal feature extraction basic network to perform preliminary extraction of spatiotemporal features to generate initial spatiotemporal features; isolating the time features in the initial spatiotemporal features, and inputting the spatial features in the initial spatiotemporal features into the spatial feature compression network to perform feature compression to generate compressed spatial features; inputting the compressed spatial features into the spatial feature restoration network to perform feature extraction and restoration to generate restored spatial features; inputting the time features in the initial spatiotemporal features into the temporal feature compression network to perform feature compression to generate compressed time features; inputting the restored spatial features and the compressed time features into the spatiotemporal feature fusion network to perform feature fusion to generate fused spatiotemporal features; inputting the fused spatiotemporal features into the feature mapping network to perform feature mapping, and outputting the K-th local prediction result; Wherein, the spatiotemporal feature extraction basic network includes: A spatial feature extraction unit, used for extracting three-dimensional spatial features from original input features; A channel feature extraction unit, used to extract channel features from original input features; The time feature extraction unit is used to extract the time features from the original input features.

2. The method according to claim 1, further comprising: Transmitting a millimeter-wave radar transmission signal to the object under test, and receiving a plurality of millimeter-wave radar echo signals reflected by the irradiated area of ​​the object under test; Performing first signal processing on the multiple millimeter-wave radar echo signals to obtain multiple first-stage signals, wherein the first signal processing includes: performing coherent accumulation operations on the multiple millimeter-wave radar echo signals defined by multiple channels composed of N transmitting antennas and M receiving antennas, and the first-stage signals include three-dimensional spatial features and time features; Performing second signal processing on the plurality of first-stage signals to generate a second-stage signal for characterizing the body center position of the measured object, wherein the second-stage signal includes a three-dimensional spatial feature; Taking a predetermined neighborhood range of the spatial position point of the second-stage signal as a target three-dimensional spatial region, performing third signal processing on a plurality of millimeter-wave radar echo signals within the target three-dimensional spatial region to obtain a plurality of third-stage signals related to the mechanical movement of the heart of the measured object, wherein the third-stage signals include three-dimensional spatial features and time features; The fourth signal processing is performed on the plurality of third stage signals to eliminate the influence of mechanical movement caused by the breathing of the measured object, and the plurality of reference signals related to the heartbeat movement of the heart part of the measured object are obtained.

3. The method according to claim 2, wherein: Performing second signal processing on the plurality of first-stage signals to generate a second-stage signal for characterizing the center position of the human body of the measured object includes: Accumulating the signal values ​​corresponding to each spatial position point in the multiple first-stage signals based on the multiple time windows to generate multiple time accumulation signals corresponding to the multiple spatial position points one by one, wherein the time accumulation signals include three-dimensional spatial features; Calculating a signal difference value between each of the time-accumulated signals and a noise signal group, wherein the noise signal group is composed of a plurality of target signals within a predetermined three-dimensional neighborhood of the time-accumulated signal, and the plurality of target signals are not directly adjacent to the time-accumulated signal; According to the signal difference value corresponding to each of the time-accumulated signals, a second-stage signal for characterizing the body center position of the measured object is determined from the multiple time-accumulated signals.

4. The method according to claim 2, wherein: The coherent accumulation operation on the multiple millimeter-wave radar echo signals defined by the multiple channels includes adopting the following method: in: Among the multiple first-stage signals obtained after the coherent accumulation operation, the corresponding spatial position and the ith signal in signal acquisition period t; N is the number of receiving antennas; M is the number of transmitting antennas; T is the total number of sampling points in each frame period of the radar; is the sampling point within the cycle; is a channel defined by the nth receiving antenna and the mth transmitting antenna in the acquisition period corresponding to the multiple millimeter wave radar echo signals signal; is the sampling period; is the rate of change of the transmission frequency of the millimeter wave radar transmission signal; From the mth transmitting antenna to the target position And returns the round trip distance of the nth receiving antenna; is the speed of light; is the wavelength of the signal transmitted by the millimeter-wave radar.

5. The method according to claim 2, wherein: Performing fourth signal processing on the plurality of third-stage signals includes: Performing second-order differential processing on the plurality of third-stage signals to eliminate the influence of mechanical movement caused by breathing of the measured object, wherein performing second-order differential processing on the plurality of third-stage signals comprises: in: It is the third stage signal; For the i-th The i-th reference signal obtained by performing second-order difference processing; h is the period of each second-stage signal; for The third stage signal of the nth sample before, n=1, 2, 3; for Then the third stage signal of the nth sampling; is the value of the time series that is i samples away.

6. A heart rate variability prediction device based on millimeter wave radar signals, comprising: A reading module, used for reading from a data storage area a plurality of reference signals related to the heartbeat movement of the heart of the object under test, wherein the reference signals include three-dimensional spatial features and time features, the plurality of reference signals correspond to a plurality of spatial position points, the reference signal of each spatial position point corresponds to a plurality of time windows, the plurality of reference signals are obtained by processing a plurality of original signals, and the plurality of original signals are: a plurality of millimeter wave radar echo signals reflected by the irradiated area of ​​the object under test; A prediction module, used to predict the multiple reference signals using a trained heartbeat rhythm prediction model, and output a target prediction result for characterizing the heartbeat rhythm of the subject, wherein the heartbeat rhythm prediction model includes a spatiotemporal feature extraction basic network, a spatial feature compression network, a spatial feature restoration network, a temporal feature compression network, a spatiotemporal feature fusion network, and a feature mapping network; Wherein: predicting the multiple reference signals using the heartbeat rhythm prediction model includes performing multiple prediction operations, in each prediction operation, the heartbeat rhythm prediction model outputs a set of local prediction results based on the original input features, and the multiple sets of local prediction results corresponding to the multiple prediction operations are collectively used as the target prediction results, and the feature dimensions of the original input features include three-dimensional spatial feature dimensions, time feature dimensions, and channel feature dimensions; The Kth prediction operation in the multiple prediction operations includes: Selecting some reference signals from the multiple reference signals as K-th selected signals, and performing feature splicing on the K-th selected signals and the K-1-th local prediction results output by the heartbeat rhythm prediction model to generate K-th original input features; inputting the K-th original input features into the heartbeat rhythm prediction model, and outputting the K-th local prediction results; Among them, feature splicing the Kth selection signal and the K-1th local prediction result output by the heartbeat rhythm prediction model includes: feature splicing the Kth selection signal based on the three-dimensional spatial feature dimension and the time feature dimension, and feature splicing the K-1th local prediction result based on the channel feature dimension to generate the Kth original input feature; Among them, inputting the K-th original input feature into the heartbeat rhythm prediction model, and outputting the K-th local prediction result includes: inputting the K-th original input feature into the spatiotemporal feature extraction basic network to perform preliminary extraction of spatiotemporal features to generate initial spatiotemporal features; isolating the time features in the initial spatiotemporal features, and inputting the spatial features in the initial spatiotemporal features into the spatial feature compression network to perform feature compression to generate compressed spatial features; inputting the compressed spatial features into the spatial feature restoration network to perform feature extraction and restoration to generate restored spatial features; inputting the time features in the initial spatiotemporal features into the temporal feature compression network to perform feature compression to generate compressed time features; inputting the restored spatial features and the compressed time features into the spatiotemporal feature fusion network to perform feature fusion to generate fused spatiotemporal features; inputting the fused spatiotemporal features into the feature mapping network to perform feature mapping, and outputting the K-th local prediction result; Wherein, the spatiotemporal feature extraction basic network includes: A spatial feature extraction unit, used for extracting three-dimensional spatial features from original input features; A channel feature extraction unit, used to extract channel features from original input features; The time feature extraction unit is used to extract the time features from the original input features.