Radar-based vital sign estimation

Abstract

The present disclosure relates to radar-based vital sign estimation. In one embodiment, a method includes: receiving a radar signal using a millimeter-wave radar; generating range data based on the received radar signal; detecting a target based on the range data; performing ellipse fitting on in-phase (I) and quadrature (Q) signals associated with the detected target to generate compensated I and Q signals associated with the detected target; classifying the compensated I and Q signals; when the classification of the compensated I and Q signals corresponds to a first category, determining a displacement signal based on the compensated I and Q signals, and determining a vital sign based on the displacement signal; and when the classification of the compensated I and Q signals corresponds to a second category, discarding the compensated I and Q signals.

Description

Technical Field

[0001] The present invention relates generally to electronic systems and methods, and in a particular embodiment, to radar-based vital sign estimation. Background Art

[0002] Millimeter-wave frequency domain applications have attracted significant attention over the past few years due to the rapid advancement of low-cost semiconductor technologies such as silicon-germanium (SiGe) and fine-geometry complementary metal-oxide-semiconductor (CMOS) processes. The availability of high-speed bipolar and metal-oxide-semiconductor (MOS) transistors has led to a growing demand for integrated circuits (ICs) for millimeter-wave applications (e.g., 24 GHz, 60 GHz, 77 GHz, 80 GHz, and above 100 GHz). These applications include, for example, automotive radar systems and multi-gigabit communication systems.

[0003] In some radar systems, the distance between the radar and the target is determined by transmitting a frequency modulated signal, receiving a reflected wave of the frequency modulated signal (also called an echo), and based on the time delay and / or frequency difference between the transmission and reception of the frequency modulated signal. Therefore, some radar systems include a transmitting antenna for transmitting radio frequency (RF) signals and a receiving antenna for receiving reflected RF signals, as well as associated RF circuits for generating the transmitted and received RF signals. In some cases, multiple antennas can be used to implement directional beams using phased array technology. Multiple-input multiple-output (MIMO) configurations with multiple chipsets can also be used to perform coherent and non-coherent signal processing. Summary of the Invention

[0004] According to one embodiment, a method includes: receiving a radar signal using a millimeter wave radar; generating range data based on the received radar signal; detecting a target based on the range data; performing ellipse fitting on in-phase (I) and quadrature (Q) signals associated with the detected target to generate compensated I and Q signals associated with the detected target; classifying the compensated I and Q signals; when the classification of the compensated I and Q signals corresponds to a first category, determining a displacement signal based on the compensated I and Q signals, and determining a vital sign based on the displacement signal; and when the classification of the compensated I and Q signals corresponds to a second category, discarding the compensated I and Q signals.

[0005] According to one embodiment, a device includes a millimeter wave radar configured to transmit a chirp signal chirp and receive a reflected chirp signal, and a processor, the processor being configured to: generate range data based on the received reflected chirp signal; detect a target based on the range data; perform ellipse fitting on in-phase (I) and quadrature (Q) signals associated with the detected target to generate compensated I and Q signals associated with the detected target; classify the compensated I and Q signals; when the classification of the compensated I and Q signals corresponds to a first category, determine a displacement signal based on the compensated I and Q signals, and determine a vital sign based on the displacement signal; and when the classification of the compensated I and Q signals corresponds to a second category, discard the compensated I and Q signals.

[0006] According to one embodiment, a method for generating a time-domain displacement signal using a millimeter-wave radar includes: receiving a reflected chirp signal using a receiver circuit of the millimeter-wave radar; performing a range FFT on the received reflected chirp signal to generate range data; detecting a target based on the range data; detecting movement of a detected target by calculating a standard deviation of the range data within a range of the detected target; generating compensated I and Q signals associated with the detected target by performing ellipse fitting on in-phase (I) and quadrature (Q) signals associated with the detected target when the standard deviation is lower than a first threshold; classifying the compensated I and Q signals; determining a time-domain displacement signal based on the compensated I and Q signals when the classification of the compensated I and Q signals corresponds to a first category; and discarding the compensated I and Q signals when the classification of the compensated I and Q signals corresponds to a second category. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] For a more complete understanding of the present invention and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which:

[0008] Figure 1 A radar system according to an embodiment of the present invention is shown;

[0009] Figure 2 A flow chart illustrating an embodiment method for generating a time domain displacement signal from a reflected radar signal according to an embodiment of the present invention is shown;

[0010] Figure 3 and Figure 4 Exemplary in-phase (I) and quadrature (Q) plots are shown for low and high quality, respectively;

[0011] Figure 5 shows a filtered time-domain displacement signal of a heartbeat of a human target according to an embodiment of the present invention;

[0012] Figure 6 The method for executing the embodiment of the present invention is shown. Figure 2 Flowchart of an embodiment method of an ellipse fitting step;

[0013] Figure 7 and Figure 8 The method for executing the Figure 2 A flowchart of an embodiment method of the classification step;

[0014] Figure 9 A block diagram illustrating an embodiment for classifying IQ data using a deep neural network according to an embodiment of the present invention; and

[0015] Figure 10 and Figure 11 The present invention is respectively shown in FIG. Figure 9 The training of deep neural networks and Figure 9 Block diagram of a trained deep neural network.

[0016] Corresponding numerals and symbols in the different figures generally refer to corresponding parts unless otherwise indicated.The figures are drawn to clearly illustrate the relevant aspects of the preferred embodiments and are not necessarily drawn to scale. DETAILED DESCRIPTION

[0017] The making and using of the disclosed embodiments are discussed in detail below. However, it should be appreciated that the present invention provides many applicable inventive concepts that can be embodied in a variety of specific contexts. The specific embodiments discussed are merely illustrative of specific ways to make and use the invention and do not limit the scope of the invention.

[0018] The following description shows various specific details that provide an in-depth understanding of several example embodiments according to the description. The embodiments can be obtained without the use of one or more specific details, or by other methods, components, materials, etc. In other cases, known structures, materials, or operations are not shown or described in detail to avoid obscuring different aspects of the embodiments. References to "one embodiment" in this specification indicate that the specific configuration, structure, or feature described in connection with the embodiment is included in at least one embodiment. Therefore, phrases such as "in one embodiment" appearing in different places in this specification do not necessarily refer to the same embodiment. In addition, specific configurations, structures, or features may be combined in any appropriate manner in one or more embodiments.

[0019] Embodiments of the present invention will be described in the specific context of a radar-based system and method for estimating vital signs (such as respiratory rate and heart rate) in a human target. Embodiments of the present invention may be used for non-human targets (such as animals) and other types of vital signs (such as cardiac signal shape monitoring).

[0020] Embodiments of the present invention may be used in a variety of applications. For example, some embodiments may be used for patient monitoring in hospitals, sleep apnea detection, and some embodiments may allow for continuous monitoring of vital signs, which may facilitate better diagnosis through early detection and prevention of critical health conditions.

[0021] Some embodiments may be used in applications such as presence sensing in homes and offices, driver monitoring in automobiles, and physiological monitoring for surveillance and earthquake rescue operations. For example, monitoring the vital signs of human subjects has widespread application in areas such as consumer electronics, medical, surveillance, driver assistance, and industrial applications. Some embodiments may be used in other applications.

[0022] In an embodiment of the present invention, a millimeter wave radar is used to generate a time domain displacement signal based on the in-phase (I) and quadrature (Q) signals associated with a human target, wherein the time domain displacement signal is used, for example, to estimate vital signs (e.g., heart rate or respiratory rate). Before generating the time domain displacement signal, random limb motion (RBM) and / or intermodulation products (IMP) that may affect the IQ signal are compensated. For example, the compensated IQ signal is classified as high quality (e.g., data without RBM or IMP) or low quality (e.g., data with RBM or IMP). Low quality data is discarded and not used to assess vital signs. In some embodiments, a neural network is used to classify the compensated IQ signal.

[0023] Radar, such as millimeter-wave radar, can be used to detect and track humans. Once a human target is identified, the radar can be used to monitor vital signs, such as the identified human target's heart rate and / or respiratory rate. Thus, in some embodiments, radar, such as millimeter-wave radar, can enable a non-contact, non-invasive method for life sensing, which can advantageously improve the comfort of the human target during vital sign monitoring.

[0024] Traditionally, radar-based life sensing has utilized discrete Fourier transforms and determined respiratory and heart rates by detecting the maximum peak in the spectrum. This approach can be susceptible to artifacts such as multiple reflections, multipath effects, motion artifacts, RBMs, and IMPs. For example, f h +f r IMP at a frequency may occur at the heart rate (e.g., a frequency f between 0.7 Hz and 3 Hz). h ) and the respiratory rate (e.g., a frequency f between 0.2 Hz and 0.5 Hz) r )between.

[0025] Figure 1 A radar system 100 according to one embodiment of the present invention is shown. The radar system 100 includes a millimeter wave radar 102 and a processor 104. In some embodiments, the millimeter wave radar 102 includes the processor 104.

[0026] During normal operation, mmWave radar 102 transmits a plurality of radiation pulses 106 (such as a chirp signal) toward scene 108 using transmitter (TX) circuitry 120. In some embodiments, the chirp signal is a linear chirp signal (ie, the instantaneous frequency of the chirp signal varies linearly with time).

[0027] The emitted radiation pulse 106 is reflected by an object in the scene 108. The reflected radiation pulse ( Figure 1 The return signal (not shown in FIG. 1 ) is received by the millimeter wave radar 102 using a receiver (RX) circuit 122 and processed by the processor 104 to, for example, detect and track targets such as people.

[0028] Objects in scene 108 may include stationary people (such as lying person 110), people exhibiting low and infrequent motion (such as standing person 112), and moving people (such as running or walking people 114 and 116). Objects in scene 108 may also include static objects (not shown), such as furniture, and periodically moving devices. Other objects may also be present in scene 108.

[0029] Processor 104 uses signal processing techniques to analyze the echo data to determine the person's location. For example, in some embodiments, a range discrete Fourier transform (DFT), such as a range fast Fourier transform (FFT), is used to estimate the range component of the detected person's location (e.g., relative to the millimeter-wave radar's position). An angle estimation technique can be used to determine the azimuth component of the detected person's location.

[0030] The processor 104 may be implemented as a general-purpose processor, a controller, or a digital signal processor (DSP), including, for example, a combination of circuits coupled to a memory. In some embodiments, for example, the processor 104 may be implemented using an ARM architecture. In some embodiments, the processor 104 may be implemented as a customized application-specific integrated circuit (ASIC). Some embodiments may be implemented as a combination of hardware accelerators and software running on a DSP or a general-purpose microcontroller. Other implementations are also possible.

[0031] Millimeter wave radar 102 operates as a frequency modulated continuous wave (FMCW) radar, which includes millimeter wave radar sensor circuitry and one or more antennas. Millimeter wave radar 102 transmits (using TX 120) and receives (using RX 122) signals in the range of 20 GHz to 122 GHz via one or more antennas (not shown). Some embodiments may use frequencies outside this range, such as frequencies between 1 GHz and 20 GHz or frequencies between 122 GHz and 300 GHz.

[0032] In some embodiments, a bandpass filter (BPF), a low-pass filter (LPF), a mixer, a low-noise amplifier (LNA), and an intermediate frequency (IF) amplifier are used to filter and amplify the return signal received by the millimeter-wave radar 102 in a manner known in the art. The return signal is then digitized using one or more analog-to-digital converters (ADCs) for further processing. Other implementations are also possible.

[0033] In general, using radar-based systems to monitor vital signs, such as a human subject's heart rate or respiration rate, is a complex task. For example, the amplitude of a heartbeat signal is typically smaller than the amplitude of a human subject's respiration signal. The amplitude of a heartbeat signal is also typically smaller than the amplitude caused by a human subject's motion (e.g., walking) and random limb movements (e.g., raising an arm, twisting the torso, etc.). Furthermore, the signal shape of a single heartbeat can depend on the subject, the selected measurement point, and the distance from the antenna.

[0034] In one embodiment of the present invention, an ellipse fitting of an IQ signal associated with a detected target is used to generate a compensated IQ signal. For example, a classifier is used to classify the compensated IQ signal as high quality (e.g., without RBM and IMP) or low quality (e.g., with RBM or IMP). The IQ signal classified as low quality is discarded. The IQ signal classified as high quality (e.g., high quality data) is used to generate a time domain displacement signal. The time domain displacement signal is then filtered, and the filtered time domain displacement signal is used to estimate vital signs (e.g., heart rate or respiratory rate). In some embodiments, a Kalman filter is used to track estimated vital signs over time, for example, as described in co-pending U.S. patent application No. 16 / 794,904, entitled “Radar VitalSignal Tracking Using a Kalman Filter,” filed on February 19, 2020, which is incorporated herein by reference. In this embodiment, the discarded IQ signal is not used to update the Kalman filter.

[0035] In some embodiments, discarding data affected by RBM or IMP advantageously allows for preventing jumps in estimated vital signs, such as in estimated heart rate or respiratory rate.

[0036] Figure 2 A flow chart of an exemplary method 200 for generating a displacement signal from a reflected radar signal according to an embodiment of the present invention is shown. For example, the method 200 may be executed by the processor 104.

[0037] During step 202, the millimeter wave radar 102 uses the transmitter (TX) circuit 120 to transmit, for example, linear chirp signals organized in frames. The time between chirp signals in a frame is generally referred to as the pulse repetition time (PRT). In some embodiments, the time interval between the end of the last chirp signal of a frame and the beginning of the first chirp signal of the next frame is the same as the PRT, so that all chirp signals are transmitted (and received) equidistantly.

[0038] In some embodiments, the chirp signal has a bandwidth of 2 GHz within the 60 GHz UWB band, the frame time has a duration of 1.28 s, and the PRT is 5 ms (corresponding to an effective sampling rate of 200 Hz).

[0039] After reflecting from the object, the receiver (RX) circuit 122 receives the reflected chirp signal during step 204. During step 204, raw data is generated based on the reflected chirp signal received by the millimeter wave radar 102. For example, in some embodiments, during step 204, the transmitted and received radar signals are mixed to generate an IF signal. The IF signal may be referred to as a beat frequency signal. The IF signal is then filtered (e.g., using a low-pass and / or band-pass filter) and digitized using an ADC to generate raw data.

[0040] During step 206, a range FFT is performed on the raw data to generate range data. For example, in some embodiments, the raw data is zero-padded and a Fast Fourier Transform (FFT) is applied to generate range data that includes range information for all targets. In some embodiments, the maximum deterministic range for the range FFT is based on the PRT, the number of samples per chirp signal, the chirp signal time, and the sampling rate of the analog-to-digital converter (ADC). In some embodiments, the ADC has a 12-bit resolution. ADCs with different resolutions, such as 10-bit, 14-bit, or 16-bit, may also be used.

[0041] In some embodiments, a range FFT is applied to all samples in the observation window. The observation window can be implemented as a continuous window or a sliding window and can have a length of one or more frames. For example, in some embodiments, the observation window is implemented as a sliding window, where the length of the observation window corresponds to the number of time steps evaluated during each time step. For example, in an embodiment where the time step is equal to 1 frame and the observation window is a sliding window with 8 frames, then for each frame, the last 8 frames are used as the observation window. In one embodiment, an observation window with a duration of 8 frames has a duration of approximately 10 seconds.

[0042] In some embodiments, range data such as a range image, such as a range Doppler image or a range-cross-range image, is generated during step 206 .

[0043] During step 208, detection of potential targets is performed. For example, in some embodiments, an order statistics (OS) constant false alarm rate (CFAR) (OS-CFAR) detector is performed during step 208. The CFAR detector generates target detection data (also referred to as target data) in which, for example, a "1" represents a target and a "0" represents a non-target based on the power level of the range data. For example, in some embodiments, the CFAR detector performs a peak search by comparing the power level of the range image to a threshold. Points above the threshold are marked as targets, while points below the threshold are marked as non-targets. Although targets can be represented by 1 and non-targets can be represented by 0, it will be appreciated that other values ​​can be used to indicate targets and non-targets.

[0044] In some embodiments, during step 210, targets present in the target data are clustered to generate clustered targets (e.g., because a human target can occupy more than one range bin (also called a "range bin"). Clustering is used to "fuse" target point clouds belonging to one target into a single target, and thereby determine the average range of such a single target. For example, in one embodiment, during step 210, a density-based spatial clustering with noise (DBSCAN) algorithm is used to associate targets with clusters. The output of DBSCAN is a grouping of detected points into specific targets. DBSCAN is a popular unsupervised algorithm that clusters targets using minimum point and minimum distance criteria and can be implemented in any manner known in the art. Other clustering algorithms can also be used. Clustered targets are used to identify ranges of interest (target distances) associated with detected targets.

[0045] During step 212, motion detection of the clustered targets is performed. For example, in some embodiments, the standard deviation of the complex range data at the target distance is calculated for motion detection. For example, in some embodiments, the complex FFT output is stored in a sliding window. The amplitude of each range bin is then summed along the complete sliding window. The peak in the complete sliding window within the selected minimum and maximum ranges is the target range bin for the current frame, which is used for target detection (step 208). The standard deviation of the range bins of the detected targets indicates the amount of motion of the target detected at the target range bin.

[0046] In some embodiments, data is discarded if the standard deviation in the target range bins along the sliding window is above a first predetermined threshold (motion detection). For example, in some embodiments, if the human target is determined to be moving during step 214 (if the standard deviation in the target range bins along the sliding window is above a first predetermined threshold), data from the current frame may be discarded. It should be understood that when the target is determined not to be moving during step 214 (when the standard deviation is below the first predetermined threshold), the target may exhibit some movement, such as movement of the target's hand outside the radar field of view or any other movement that causes the standard deviation to be below the first predetermined threshold. This may be the case, for example, if the human target is sitting or standing.

[0047] In some embodiments, data is also discarded during step 214 if the standard deviation in the target range bin along the sliding window is below a second predetermined threshold (eg, discarding static objects such as chairs).

[0048] In some embodiments, during step 214 , data is further processed only if the standard deviation in the target range bins along the sliding window is determined to be above a low standard deviation threshold and below a high standard deviation threshold.

[0049] Data that is not discarded during step 214 is further processed during step 216. During step 216, an ellipse fitting algorithm (also known as an ellipse correction algorithm) is applied to the IQ trace (of the negative range data) associated with the detected target to compensate for offset, amplitude, and gain errors. In some embodiments, the compensated data (compensated IQ signal) is an IQ signal corresponding to a best-fit ellipse associated with the uncompensated IQ signal. Some embodiments may avoid the use of an ellipse fitting algorithm and, for example, may use an offset compensation algorithm.

[0050] During step 218, the compensated data (compensated IQ signal) is provided to a classifier. During step 218, the classifier estimates the quality of the compensated data and classifies the compensated data as "high quality" (first class) or "low quality" (second class). Low-quality compensated data is discarded during step 220, while high-quality data is further processed, for example, in step 222. Figure 3 and Figure 4 Exemplary IQ curve graphs of low quality (low quality data) and high quality (high quality data) are shown in FIG.

[0051] In some embodiments, the RBM and IMP associated with the target result in low-quality data (e.g., Figure 3 shown).

[0052] In some embodiments, the classifier used during step 218 is a deep learning-based classifier that, for example, uses a neural network to estimate the quality of the compensation data. In other embodiments, the quality of the compensation data is estimated based on, for example, the amplitude and phase imbalance values ​​that may be generated during step 216. In other embodiments, the classifier used during step 218 is implemented using traditional machine learning methods, such as using a support vector machine (SVM), random forest, etc.

[0053] During step 222, the angle of the compensated target data is calculated by demodulating the arctangent of the IQ signal (the IQ signal associated with the detected target) from the range bin selected during step 208. During step 224, the resulting phase value in the [-π, +π] range is unwrapped between two consecutive data points. For example, during step 224, the phase is unwrapped by adding or subtracting 2π for phase jumps greater than -π or +π, respectively.

[0054] In some embodiments, steps 222 and 224 may be performed by calculating the displacement signal as follows:

[0055]

[0056] where D represents the time domain displacement signal, λ is the wavelength of the carrier frequency, λ / 2 represents the deterministic (phase) range, and I and Q are the in-phase and quadrature-phase components of the carrier associated with the detected target, respectively.

[0057] During step 226, the displacement signal is filtered and vital sign estimation is performed during step 228. For example, in some embodiments, a bandpass FIR filter with a passband frequency from 0.7 Hz to 3 Hz is used, and heart rate estimation is performed during step 228. In some embodiments, a bandpass FIR filter with a passband frequency from 0.2 Hz to 0.5 Hz is used, and respiratory rate estimation is performed during step 228. In some embodiments, for example, for the purpose of heart rate estimation, a fourth-order Butterworth bandpass digital filter with a passband from 0.75 Hz to 3.33 Hz is applied to the displacement signal. In some embodiments, for example, for the purpose of respiratory rate estimation, a fourth-order Butterworth bandpass digital filter with a passband from 0.2 Hz to 0.33 Hz is applied to the displacement signal. Filters of different orders and / or different types and / or different frequencies may also be applied. In some embodiments, the passband of the bandpass filter is adaptive, for example, as described in co-pending U.S. patent application Ser. No. 16 / 794,904, filed on February 19, 2020, entitled “Radar VitalSignal Tracking Using a Kalman Filter,” which is incorporated herein by reference.

[0058] In some embodiments, multiple bandpass filters are applied to the time-domain displacement signal, for example, simultaneously to estimate corresponding multiple vital signs. For example, some embodiments include a first bandpass filter for heart rate estimation and a second bandpass filter for respiratory rate estimation.

[0059] In some embodiments, during step 228, vital sign estimation is performed by counting the number of peaks exhibited by the filtered time-domain displacement signal over a period of time or by measuring the time between detected peaks of the filtered time-domain displacement signal. Figure 5 FIG. 4 shows a filtered time domain displacement signal of a heartbeat of a human target according to an embodiment of the present invention. Figure 5 In the example above, the heart rate is determined by counting the number of peaks over a period of time, in this case the heart rate is estimated to be 58 bpm.

[0060] Figure 6 A flow chart of an embodiment method 600 for performing the ellipse fitting step 216 is shown according to one embodiment of the present invention.

[0061] During step 602, IQ data corresponding to the cluster target is received (e.g., from step 214). The I and Q signals can be represented as:

[0062]

[0063]

[0064] Where d(t) is the time-varying displacement signal, B I and B Q represents the DC offset, and λ is the wavelength of the carrier frequency.

[0065] During step 604, the amplitude and phase imbalances are estimated. For example, the amplitude imbalance A e and phase imbalance φ e It can be given by the following formula:

[0066]

[0067] φ e =φ Q -φ I (5)

[0068] For example, as shown in the article “Data-Based Quadrature Imbalance Compensation for a CW Doppler Radar System” published by A. Singh et al. in IEEE Transactions on Microwave Theory and Techniques in April 2013 (Volume 61, Issue 4, Pages 1718-1724), the amplitude imbalance A e and phase imbalance φ e It can be calculated by the following formula:

[0069]

[0070]

[0071] The normalized equation of the ellipse (where I is represented on the horizontal axis and Q is represented on the vertical axis) can be given as follows:

[0072] I 2 +A×Q 2 +B×IQ+C×I+D×Q+E=o (8)

[0073] The optimal solutions for A, B, C, D, and E can be as follows:

[0074]

[0075] Where M and b can be given as follows:

[0076]

[0077]

[0078] where N is the number of (I,Q) samples.

[0079] Once Equation 9 is solved, the magnitude imbalance A e and phase imbalance φ e This can be calculated using Equations 6 and 7.

[0080] Once the amplitude imbalance A is determined e and phase imbalance φ e , then during step 604 compensated IQ data are generated, for example in a known manner using Gram-Schmidt (GS) or ellipse fitting methods.

[0081] Figure 7 A flow chart of an embodiment method 700 for performing classification steps 218 and 220 is shown, according to one embodiment of the present invention.

[0082] During step 702, an imbalance value is received (eg, determined during step 604). In some embodiments, for example, the imbalance value may be the magnitude imbalance A calculated during step 604. e and phase imbalance φ e .

[0083] During step 704, the imbalance value is compared to an imbalance threshold. If the imbalance value is above the imbalance threshold, the compensated IQ data (e.g., generated during step 604) is classified as low quality. If the imbalance value is below the imbalance threshold, the compensated IQ data is classified as high quality.

[0084] In some embodiments, if the amplitude imbalance A e and phase imbalance φ e If any of the above is higher than the corresponding threshold, the compensated IQ data is classified as low quality.

[0085] Figure 8 A flow chart of an embodiment method 800 for performing classification steps 218 and 220 is shown, according to one embodiment of the present invention.

[0086] During step 802, compensated IQ data is received (e.g., determined during step 604). The compensated IQ data is fed through a neural network during step 804. Based on the neural network model, the neural network classifies the compensated IQ data as low quality or high quality.

[0087] Figure 9 A block diagram of an embodiment 900 for classifying IQ data using a deep neural network 904 is shown, according to one embodiment of the present invention.

[0088] like Figure 9 As shown, after receiving N chirp signals (e.g., 2048 chirp signals) from one or more consecutive frames (e.g., 8 consecutive frames), a target bin (also referred to as a "target bin," e.g., in step 208) is identified. The compensation module 902 receives IQ data associated with the target bin (e.g., in step 602) and generates compensated IQ data (e.g., in step 604). The deep neural network classifier 904 receives the compensated IQ data (e.g., in step 802) and performs multi-level classification. For example, in some embodiments, the compensated IQ data is classified into 3 categories: RBM, IMP, and high-quality data. Only IQ data classified as "high-quality data" is classified as high quality (e.g., in step 220). IQ data classified as RBM or IMP is classified as low quality (e.g., in step 220) and discarded.

[0089] In some embodiments, the classifier 904 may be implemented using a one-dimensional (1D) convolutional neural network (CNN) layer. In some embodiments, the 1D CNN model advantageously learns directly from the IQ time series.

[0090] In some embodiments, a long short-term memory (LSTM) model may be used instead of a 1D CNN model.

[0091] Figure 10 A block diagram of a training deep neural network 1000 according to one embodiment of the present invention is shown. The deep neural network 904 can be trained as the training deep neural network 1000. The training deep neural network 1000 includes an input layer 1002, convolutional 1D layers 1004, 1010, 1018, 1026, 1032, 1038, and 1046, average pooling layers 1006, 1012, 1020, 1028, 1034, 1040, and 1048, batch normalization layers 1008, 1014, 1022, 1030, 1036, 1042, and 1050, dropout layers 1016, 1024, and 1044, and a fully connected layer 1052. Different arrangements (e.g., different numbers of layers and / or different types of layers) can also be used.

[0092] like Figure 10 As shown, the deep neural network 1000 receives batches of training data (e.g., 20 samples per batch) using an input layer 1002 and generates an M-element vector corresponding to the classification of the corresponding data using a fully connected layer 1052, where M is equal to or greater than 2. For example, in some embodiments, M is equal to 3, the output vector is in the form of ["RBM", "IMP", "high-quality data"], and the training data includes IQ data (e.g., IQ time series) pre-labeled as RBM, IMP, or high-quality data. For example, the output vector [1 0 0] represents the "RBM" classification of the corresponding data; the output vector [0 1 0] represents the "IMP" classification of the corresponding data; and the output vector [0 0 1] represents the "high-quality data" classification of the corresponding data.

[0093] In some embodiments, the output vector includes a confidence value (i.e., the probability that a particular label is correct). In such an embodiment, the output vector [0.8 0.15 0.05] can be interpreted as corresponding data having an 80% probability of corresponding to the "RBM" classification, a 15% probability of corresponding to the "IMP" classification, and a 5% probability of corresponding to the "high-quality data" classification. In this case, the corresponding data can be assigned to the classification with the highest confidence (in this non-limiting example, the classification is "RBM").

[0094] In some embodiments, when the amplitude imbalance A e and phase imbalance φe When the training data is pre-labeled as RBM when the difference between the estimated heart rate and the reference heart rate is similar to the corresponding respiratory rate of the human target, in some embodiments, the training data is pre-labeled as IMP. In some embodiments, when the criteria for RBM and IMP are not met, the training data is pre-labeled as high-quality data. In some embodiments, the training data can be pre-labeled in different ways.

[0095] During normal operation, the input layer 1002 receives training data in the form of, for example, a 256x1 one-dimensional signal (256 data points in slow time).

[0096] The convolutional 1D layers (1004, 1010, 1018, 1026, 1032, 1038, and 1046) convolve their respective inputs and pass their results to the next layer. Figure 10 As shown, convolution 1D layer 1004 uses 1024 filters to convolve 256 slow-time data; convolution 1D layer 1010 uses 512 filters to convolve 128 slow-time data; convolution 1D layer 1018 uses 512 filters to convolve 64 slow-time data; convolution 1D layer 1026 uses 128 filters to convolve 32 slow-time data; convolution 1D layer 1032 uses 128 filters to convolve 16 slow-time data; convolution 1D layer 1038 uses 128 filters to convolve 8 slow-time data; and convolution 1D layer 1046 uses 64 filters to convolve 4 slow-time data. Other embodiments are also possible.

[0097] In some embodiments, the type of convolutional filters used in the convolutional 1D layers (1004, 1010, 1018, 1026, 1032, 1038, and 1046) has a size of 3x3. Some embodiments may use other filter sizes.

[0098] Each convolutional 1D layer (1004, 1010, 1018, 1026, 1032, 1038, and 1046) uses a rectified linear unit (ReLU) as an activation function. Other activation functions can also be used.

[0099] The pooling layers (1006, 1012, 1020, 1028, 1034, 1040, and 1048) smooth the data, for example, by applying averaging. Figure 10As shown, pooling layer 1006 uses 1024 filters to filter 256 slow time data; pooling layer 1012 uses 512 filters to filter 64 slow time data; pooling layer 1020 uses 512 filters to filter 32 slow time data; pooling layer 1028 uses 128 filters to filter 16 slow time data; pooling layer 1034 uses 128 filters to filter 8 slow time data; pooling layer 1040 uses 128 filters to filter 4 slow time data; and pooling layer 1048 uses 64 filters to filter 2 slow time data. Other embodiments are also possible.

[0100] In some embodiments, each pooling layer has a size of 2x2 and a stride of 2 (corresponding to the application of the next adjacent pool, the filter window is shifted 2 units to the right and downwards). Some embodiments may use other filter sizes and different strides.

[0101] like Figure 10 As shown, each pooling layer (1006, 1012, 1020, 1028, 1034, 1040 and 1048) applies average pooling (using an average function). In some embodiments, a "max" function may also be used.

[0102] The batch normalization layers (1008, 1014, 1022, 1030, 1036, 1042, and 1050) subtract the mean of the corresponding batch data being processed from the batch data and divide the result by the standard deviation of the batch data, so that the batch data is normalized (so that the resulting mean is 0 and the resulting standard deviation is 1).

[0103] The dropout layers (1016, 1024, and 1044) help create redundancy in the neural network by randomly removing nodes in the neural network (e.g., randomly zeroing the weights on the previous convolutional layer) and removing corresponding edges to / from the removed nodes of the neural network. In some embodiments, 20% of the nodes are removed by each dropout layer. Figure 10 In the illustrated embodiment, dropout layer 1016 has a spatial domain of 64 dimensions and 512 filters; dropout layer 1024 has a spatial domain of 32 dimensions and 512 filters; and dropout layer 1044 has a spatial domain of 4 dimensions and 128 filters. Other implementations are possible.

[0104] A fully connected layer 1052 using, for example, softmax as an activation function is used to generate an M-element vector corresponding to the classification of the batch data.

[0105] During training, the generated output vectors are compared with the pre-labeled data batches, and the weights of the neural network are adjusted so that the classification of the data batches corresponds to the pre-labeled data. The model (neural network 1000) is optimized by running multiple training data batches (e.g., thousands of training data batches). In some embodiments, adaptive moment estimation (Adam) is used to optimize the deep neural network 1000.

[0106] Once trained, the trained deep neural network may be used to classify the compensated IQ data during steps 218 and 220 . Figure 11 FIG. 1 shows a deep neural network 1100 after training according to one embodiment of the present invention. Figure 10 As described, deep neural network 1100 may have been trained. Deep neural network 904 may be implemented as deep neural network 1100.

[0107] In some embodiments, the training model (e.g., Figure 10 of trained deep neural networks 1000) and inference models (e.g., Figure 11 The difference between the trained deep neural network 1100 is batch normalization. For example, in the training model, batch normalization (1008, 1014, 1022, 1030, 1036, 1042, and 1050) is calculated across the mini-batches used to train the network (i.e., the mean and variance across the batches are calculated). In the inference model, the learned global mean and variance are used for (unlearned) normalization ( Figure 11 Normalization is not shown).

[0108] During normal operation, the input layer 1002 receives the compensated IQ data (e.g., generated during step 216). The training convolutional 1D layer and the pooling layer process the received IQ data, and the fully connected layer generates, for example, a 3-element vector that classifies the received IQ data as, for example, RBM, IMP, or high-quality data. IQ data classified as RBM or IMP is classified as low quality (in step 220) and discarded. IQ data classified as high-quality data is further processed in steps 222, 224, 226, and 228.

[0109] Advantages of some embodiments include that by avoiding estimating vital signs when the quality of the IQ signal is low (eg, due to RBM or IMP), jumps in the estimated vital signs (eg, jumps in heart rate or respiratory rate) may be prevented.

[0110] In some embodiments, time-domain based radar-based displacement estimation of vital signs advantageously allows for removal of IMP and RBM, taking into account phase sensitivity in the IF signal caused by minute displacements of human targets, when compared to vital sign estimation methods that rely on spectrum analysis and are easily corrupted by RBM and IMP.

[0111] This document summarizes example embodiments of the present invention. Other embodiments may be understood from the overall description and claims submitted herein.

[0112] Example 1. A method comprising: receiving a radar signal using a millimeter wave radar; generating range data based on the received radar signal; detecting a target based on the range data; performing ellipse fitting on in-phase (I) and quadrature (Q) signals associated with the detected target to generate compensated I and Q signals associated with the detected target; classifying the compensated I and Q signals; when the classification of the compensated I and Q signals corresponds to a first category, determining a displacement signal based on the compensated I and Q signals, and determining a vital sign based on the displacement signal; and when the classification of the compensated I and Q signals corresponds to a second category, discarding the compensated I and Q signals.

[0113] Example 2. The method of Example 1, wherein classifying the compensated I and Q signals comprises using a neural network.

[0114] Example 3. The method of one of Examples 1 or 2, wherein the neural network comprises a one-dimensional (1D) convolutional neural network (CNN) layer.

[0115] Example 4. The method of any one of Examples 1 to 3, wherein the neural network includes a fully connected layer.

[0116] Example 5. The method of any one of Examples 1 to 4, further comprising training a neural network.

[0117] Example 6. The method according to one of Examples 1 to 5 further includes: generating an amplitude imbalance value and a phase imbalance value associated with the compensated I and Q signals of the detected target; and classifying the compensated I and Q signals into a first category when the amplitude imbalance value is lower than a first threshold and when the phase imbalance value is lower than a second threshold.

[0118] Example 7. The method according to one of Examples 1 to 6 further includes: generating an amplitude imbalance value and a phase imbalance value associated with the compensated I and Q signals of the detected target; and classifying the compensated I and Q signals into a second category when the amplitude imbalance value is higher than a first threshold or when the phase imbalance value is higher than a second threshold.

[0119] Example 8. The method of any one of Examples 1 to 7, wherein determining the displacement signal comprises determining an angle of arrival based on the compensated I and Q signals.

[0120] Example 9. The method of one of Examples 1 to 8, wherein determining the vital sign comprises filtering the displacement signal and estimating the heart rate or the respiratory rate based on the filtered displacement signal.

[0121] Example 10. The method of any one of Examples 1 to 9, wherein determining the vital sign comprises using a Kalman filter to track changes in the vital sign over time.

[0122] Example 11. The method of any one of Examples 1 to 10, wherein discarding the compensated I and Q signals comprises refraining from updating the Kalman filter with the compensated I and Q signals.

[0123] Example 12. The method of one of Examples 1 to 11, wherein detecting the target comprises performing a peak search based on the range data.

[0124] Example 13. The method of one of Examples 1 to 12, wherein generating the range data comprises performing a range FFT on the received radar signal.

[0125] Example 14. The method of one of Examples 1 to 13, further comprising transmitting a radar signal, wherein the received radar signal is based on the transmitted radar signal.

[0126] Example 15. A device comprising a millimeter wave radar configured to transmit a chirp signal and receive a reflected chirp signal, and a processor configured to: generate range data based on the received reflected chirp signal; detect a target based on the range data; perform ellipse fitting on in-phase (I) and quadrature (Q) signals associated with the detected target to generate compensated I and Q signals associated with the detected target; classify the compensated I and Q signals; when the classification of the compensated I and Q signals corresponds to a first category, determine a displacement signal based on the compensated I and Q signals, and determine a vital sign based on the displacement signal; and when the classification of the compensated I and Q signals corresponds to a second category, discard the compensated I and Q signals.

[0127] Example 16. The apparatus of Example 15, wherein the processor is configured to classify the compensated I and Q signals using a neural network comprising a one-dimensional (1D) convolutional neural network (CNN) layer.

[0128] Example 17. The apparatus of one of Examples 15 or 16, wherein the processor is configured to determine the displacement signal as where λ is the wavelength of the carrier frequency of the transmitted chirp signal, and I and Q are the in-phase and quadrature-phase components, respectively, associated with the detected target.

[0129] Example 18. A method for generating a time domain displacement signal using a millimeter wave radar, the method comprising: receiving a reflected chirp signal using a receiver circuit of the millimeter wave radar; performing a range FFT on the received reflected chirp signal to generate range data; detecting a target based on the range data; detecting the movement of the detected target by calculating the standard deviation of the range data within the range of the detected target; when the standard deviation is lower than a first threshold, generating compensated I and Q signals associated with the detected target by performing ellipse fitting on the in-phase (I) and quadrature (Q) signals associated with the detected target; classifying the compensated I and Q signals; when the classification of the compensated I and Q signals corresponds to a first category, determining a time domain displacement signal based on the compensated I and Q signals; and when the classification of the compensated I and Q signals corresponds to a second category, discarding the compensated I and Q signals.

[0130] Example 19. The method of Example 18, wherein generating the compensated I and Q signals comprises using a Gram-Schmidt (GS) or ellipticity correction method.

[0131] Example 20. The method of one of Examples 18 or 19, wherein classifying the compensated I and Q signals comprises using a neural network comprising one-dimensional (1D) convolutional neural network (CNN) layers.

[0132] Although the present invention has been described with reference to exemplary embodiments, this description is not intended to be construed in a limiting sense. Various modifications and combinations of the illustrative embodiments, as well as other embodiments of the present invention, will become apparent to those skilled in the art upon reference to the description. Accordingly, the appended claims encompass any such modifications or embodiments.

Claims

1. A method for estimating vital signs, comprising: Receive radar signals using millimeter-wave radar; generating range data based on the received radar signal; detecting a target based on the range data; performing ellipse fitting on in-phase I and quadrature Q signals associated with a detected object to generate compensated I and Q signals associated with the detected object; generating an amplitude imbalance value and a phase imbalance value based on the in-phase I and quadrature Q signals associated with the detected object; classifying the compensated I and Q signals into a first category when the amplitude imbalance value is below a first threshold and when the phase imbalance value is below a second threshold; classifying the compensated I and Q signals into a second category when the amplitude imbalance value is higher than a first threshold or when the phase imbalance value is higher than a second threshold; determining a displacement signal based on the compensated I and Q signals when the classification of the compensated I and Q signals corresponds to the first category, and determining a vital sign based on the displacement signal; as well as When the classification of the compensated I and Q signals corresponds to the second category, the compensated I and Q signals are discarded.

2. The method of claim 1 , wherein classifying the compensated I and Q signals comprises using a neural network.

3. The method of claim 2, wherein the neural network comprises a one-dimensional (1D) convolutional neural network (CNN) layer.

4. The method of claim 2, wherein the neural network comprises fully connected layers. The method of claim 2 , further comprising training the neural network. The method of claim 1 , wherein determining the displacement signal comprises determining an angle of arrival based on the compensated I and Q signals.

7. The method of claim 1 , wherein determining the vital sign comprises: The displacement signal is filtered, and the heart rate or respiratory rate is estimated based on the filtered displacement signal.

8. The method of claim 1 , wherein determining the vital sign comprises: A Kalman filter is used to track changes in the vital signs over time.

9. The method of claim 8, wherein discarding the compensated I and Q signals comprises: Updating the Kalman filter with the compensated I and Q signals is avoided.

10. The method of claim 1 , wherein detecting the target comprises: A peak search is performed based on the range data.

11. The method of claim 1 , wherein generating the range data comprises: A range FFT is performed on the received radar signal.

12. The method of claim 1, further comprising transmitting a radar signal, wherein the received radar signal is based on the transmitted radar signal.

13. A vital sign estimation device comprising: a millimeter wave radar configured to transmit a chirp signal and receive a reflected chirp signal; as well as The processor is configured to: generating range data based on the received reflected chirp signal; detecting a target based on the range data; performing ellipse fitting on in-phase I and quadrature Q signals associated with a detected object to generate compensated I and Q signals associated with the detected object; generating an amplitude imbalance value and a phase imbalance value based on the in-phase I and quadrature Q signals associated with the detected object; classifying the compensated I and Q signals into a first category when the amplitude imbalance value is below a first threshold and when the phase imbalance value is below a second threshold; classifying the compensated I and Q signals into a second category when the amplitude imbalance value is higher than a first threshold or when the phase imbalance value is higher than a second threshold; determining a displacement signal based on the compensated I and Q signals when the classification of the compensated I and Q signals corresponds to the first category, and determining a vital sign based on the displacement signal; as well as When the classification of the compensated I and Q signals corresponds to the second category, the compensated I and Q signals are discarded.

14. The apparatus of claim 13, wherein the processor is configured to classify the compensated I and Q signals using a neural network comprising a one-dimensional (1D) convolutional neural network (CNN) layer.

15. A method for generating a time-domain displacement signal using a millimeter-wave radar, the method comprising: receiving the reflected chirp signal using a receiver circuit of the millimeter-wave radar; performing a range FFT on the received reflected chirp signal to generate range data; detecting a target based on the range data; detecting movement of the detected object by calculating a standard deviation of the range data within the range of the detected object; generating compensated I and Q signals associated with the detected object by performing ellipse fitting on in-phase I and quadrature Q signals associated with the detected object when the standard deviation is below a predetermined threshold; generating an amplitude imbalance value and a phase imbalance value based on the in-phase I and quadrature Q signals associated with the detected object; classifying the compensated I and Q signals into a first category when the amplitude imbalance value is below a first threshold and when the phase imbalance value is below a second threshold; classifying the compensated I and Q signals into a second category when the amplitude imbalance value is higher than a first threshold or when the phase imbalance value is higher than a second threshold; determining the time-domain shifted signal based on the compensated I and Q signals when the classification of the compensated I and Q signals corresponds to the first category; as well as When the classification of the compensated I and Q signals corresponds to the second category, the compensated I and Q signals are discarded.

16. The method of claim 15, wherein generating the compensated I and Q signals comprises using a Gram-Schmidt (GS) or ellipticity correction method.

17. The method of claim 15, wherein classifying the compensated I and Q signals comprises using a neural network comprising one-dimensional (1D) convolutional neural network (CNN) layers.

Images