Multi-sensor system for cardiovascular and respiratory tracking

Abstract

A multi-sensor system is used to measure cardiovascular or respiratory data of a subject. This system includes a millimeter-wave frequency-modulated continuous wave (FMCW) radar sensor, an inertial measurement unit (IMU) sensor, and one or more proximity sensors. The millimeter-wave FMCW radar sensor can be selected and its viewing angle can be adjusted based on positioning data about the subject obtained from one or more proximity sensors. Each of the millimeter-wave FMCW radar sensor and the IMU sensor can acquire cardiovascular or respiratory measurements of the subject, and these measurements can be fused to improve accuracy and performance.

Description

[Technical Field]

[0001] This application claims priority to U.S. Patent Application No. 17 / 302,444, filed May 3, 2021, entitled “MULTI-SENSOR SYSTEM FOR CARDIOVASCULAR AND RESPIRATORY TRACKING”, which is incorporated herein by reference.

[0002] This disclosure relates in general to tracking physiological data, and more specifically to using millimeter-wave radar sensors, such as millimeter-wave frequency-modulated continuous wave (FMCW) radar sensors, to track cardiovascular and respiratory data. [Previous Technology]

[0003] Monitoring human vital signs is crucial for saving lives. In many situations, continuous monitoring of individuals or the detection of an individual's presence through the measurement of vital signs is necessary. In many cases, such as monitoring patients in clinical settings, measuring health status, detecting driver fatigue, studying sleep, and searching for survivors after natural disasters, tracking physiological data such as heart rate and respiratory rate may be essential.

[0004] Typically, wires or cables are used to perform electronic sensing of heart rate and respiratory rate. In this case, the wires or cables are connected from a sensor attached to the human body. Alternatively, elastic bands can be used to perform electronic sensing of heart rate and respiratory rate. The electronic signals are sent to a processing system for analysis into heart rate, respiratory rate, pulse rate, etc. However, contact-based sensors can be difficult to implement, impractical, inconvenient, and abrupt.

[0005] Various methods for contactless sensing of heart rate and respiratory rate have been proposed. One method is to use optical interferometry, but this optical-based method is limited by the fact that the light signal may be blocked by clothing or other materials. Another method is to use ultrasound to detect motion. However, this method is limited by the low signal-to-noise ratio caused by low reflectivity. [Summary of the Invention]

[0006] The apparatus, system and method of this disclosure each have several aspects, and no single aspect is solely responsible for the desired properties disclosed herein.

[0007] One aspect of the subject matter of this disclosure can be implemented in a multi-sensor system. The multi-sensor system includes a millimeter-wave FMCW radar sensor located inside a vehicle, wherein the millimeter-wave FMCW radar sensor is configured to acquire a first cardiovascular or respiratory measurement of a subject. The multi-sensor system also includes: a first inertial measurement unit (IMU) sensor located inside the vehicle, wherein the first IMU sensor is configured to acquire a second cardiovascular or respiratory measurement of the subject; and a second IMU sensor located inside the vehicle, wherein the second IMU sensor is configured to acquire motion data associated with the vehicle. The multi-sensor system also includes one or more proximity sensors configured to identify the position of the subject's chest within the vehicle.

[0008] In some implementations, the millimeter-wave FMCW radar sensor is configured to send a signal toward the location of the subject's chest. In some implementations, the first IMU sensor is positioned in the seat, on the outer surface of the seat, or under the seat of the vehicle. In some implementations, the second IMU sensor is positioned on the steering wheel or dashboard of the vehicle. In some implementations, one or more proximity sensors include an array of capacitive proximity sensors. In some implementations, the multi-sensor system also includes a control system communicatively connected to the millimeter-wave FMCW radar sensor, the first IMU sensor, the second IMU sensor, and one or more proximity sensors, wherein the control system is configured to: use one or more proximity sensors to determine the position of the subject's chest relative to the millimeter-wave FMCW radar sensor, and adjust the viewing angle of the millimeter-wave FMCW radar sensor to point toward the location of the subject's chest. In some implementations, the control system is also configured to: merge a first cardiovascular or respiratory measurement, a second cardiovascular or respiratory measurement, and motion data associated with the vehicle to obtain an optimal cardiovascular or respiratory measurement. In some implementations, the multi-sensor system also includes: multiple millimeter-wave FMCW radar sensors for obtaining multiple cardiovascular or respiratory measurements from multiple subjects inside the vehicle, each of which is calculated to determine the subject's stress level, drowsiness level, or health; and a control system communicatively connected to the multiple millimeter-wave FMCW radar sensors, wherein the control system is configured to perform actions in the vehicle in response to the subject's stress level, drowsiness level, or health.

[0009] Another innovative aspect of the subject matter described in this disclosure can be implemented in a multi-sensor system. The multi-sensor system includes one or more millimeter-wave FMCW radar sensors in a room, wherein the millimeter-wave FMCW radar sensors are configured to acquire a first cardiovascular or respiratory measurement of a subject in the room. The multi-sensor system also includes: an IMU sensor in the room, wherein the IMU sensor is configured to acquire a second cardiovascular or respiratory measurement of the subject; and one or more proximity sensors configured to identify the body orientation of the subject within the room.

[0010] In some implementations, at least one of one or more millimeter-wave FMCW radar sensors is configured to be selected to obtain a first cardiovascular or respiratory measurement of the subject based on the subject's body orientation. In some implementations, at least one selected millimeter-wave FMCW radar sensor is configured to send a signal toward the location of the subject's chest. In some implementations, one or more proximity sensors comprise an array of capacitive proximity sensors positioned on a bed in a room. In some implementations, the multi-sensor system further includes a control system communicatively connected to the millimeter-wave FMCW radar sensor, the IMU sensor, and one or more proximity sensors, wherein the control system is configured to: determine the subject's body orientation using one or more proximity sensors, select at least one of one or more millimeter-wave FMCW radar sensors based on the body orientation, and adjust the viewing angle of at least one selected millimeter-wave FMCW radar sensor to point toward the location of the subject's chest. In some implementations, the control system is also configured to: merge the first cardiovascular or respiratory measurement and the second cardiovascular or respiratory measurement to obtain an optimal cardiovascular or respiratory measurement.

[0011] Another innovative aspect of the subject matter described in this disclosure can be implemented in a method for calculating and combining cardiovascular or respiratory measurements using a multi-sensor system. The method includes: identifying the presence of a subject inside a vehicle or room using one or more proximity sensors; determining the position of the subject's chest and / or the subject's body orientation using one or more proximity sensors; adjusting the viewing angle of a millimeter-wave FMCW radar sensor positioned inside the vehicle or room based on the position of the subject's chest and / or the subject's body orientation; obtaining a first cardiovascular or respiratory measurement from the millimeter-wave FMCW radar sensor; obtaining a second cardiovascular or respiratory measurement from an IMU sensor; and combining the first and second cardiovascular or respiratory measurements to obtain an optimal cardiovascular or respiratory measurement.

[0012] In some implementations, one or more proximity sensors include an array of capacitive proximity sensors positioned on a stationary object inside a vehicle or room. In some implementations, the method further includes obtaining multiple cardiovascular or respiratory measurements for multiple subjects from multiple millimeter-wave FMCW radar sensors in the vehicle or room, each of which is calculated to determine the subject's stress level, drowsiness level, or health status. In some implementations, the method further includes selecting a millimeter-wave FMCW radar sensor from the multiple millimeter-wave FMCW radar sensors based on the subject's body orientation.

Implementation Method

[0025] To describe various aspects of this disclosure, the following description is directed to certain specific embodiments. However, those skilled in the art will readily recognize that the teachings herein can be applied in many different ways. Various embodiments will be described in detail with reference to illustrations. References to specific examples and embodiments are for illustrative purposes and are not intended to limit the scope of the claims.

[0026] Contact or wired devices used for monitoring vital signs may restrict mobility and potentially spread contact infections between patients and hospital staff. In space-constrained environments, contact connections may also increase complexity and implementation difficulties. Furthermore, contact connections may pose challenges for individuals who intentionally or unintentionally disconnect from contact devices. Additionally, contact or wired connections may be burdensome or impractical for some individuals, such as burn patients. Contactless options may be valuable for monitoring heart and respiratory rates and can overcome the inconvenience and discomfort associated with wearing multiple wired electrodes and / or breathing chest straps.

[0027] Radar sensors offer another option for non-invasive monitoring of human vital signs. Traditionally, radar sensors have been used for speed measurement, gesture recognition, position estimation, people counting, imaging, and tumor detection, among other applications. Due to the nature of its electromagnetic backscattering, radar sensors can be used to wirelessly detect minute displacements caused by human breathing and heartbeat.

[0028] Millimeter-wave radar sensing is a non-contact sensing technology that operates in the frequency domain between 30 GHz and 300 GHz and transmits signals with wavelengths in the millimeter range. Objects in the path reflect the signal back, and the millimeter-wave radar sensor can determine the distance, speed, and angle of (multiple) objects. Millimeter-wave radar sensors can provide sub-millimeter accuracy to detect minute displacements, making them ideal for sensing heartbeats and breathing. Millimeter-wave radar signals can penetrate materials such as plastic, drywall, and clothing, and they are unaffected by environmental conditions such as rain, fog, dust, and snow. Millimeter-wave sensors are characterized by their small size, light weight, and high spatial resolution.

[0029] Various types of radar sensors can be used for non-contact vital sign monitoring. Continuous wave (CW) radar sensors transmit a continuous wave radio frequency carrier and mix the transmitted carrier with the echo. This means that the echo is received simultaneously with the transmission. Doppler frequency shift provides a method to separate the transmitted and received signals. Ultra-wideband (UWB) radar sensors transmit a series of pulses to the target and the received signal can be visualized in the frequency domain. Vital signs can be obtained in the time domain by analyzing the time delay changes of the transmitted pulses. Frequency-modulated continuous wave (FMCW) radar sensors transmit continuously modulated linear frequency modulation. FMCW radar sensors are similar to CW radar sensors by transmitting a continuous wave signal instead of periodically transmitting short pulses. However, in the case of FMCW radar sensors, the frequency of the transmitted signal increases linearly over a fixed period of time. This type of signal is called "chirp". As the frequency of the transmitted signal changes, the returned signal has a different frequency. The target's distance information can be determined by the frequency difference between the returned and transmitted signals. The distance to the target is proportional to the frequency difference.

[0030] If an object moves toward or away from the FMCW radar sensor, the frequency and phase of the reflected linear frequency modulation will change. When the wavelength is on the order of millimeters, small changes result in large phase changes, making small movements easily detectable. For example, millimeter-wave FMCW radar sensors operating at 60 GHz or 77 GHz, corresponding to wavelengths in the range of approximately 4 mm, will be able to detect movements as short as fractions of a millimeter. Therefore, micro-movements such as chest wall movements caused by breathing or heartbeats can be detected wirelessly.

[0031] Millimeter-wave FMCW radar sensing has many advantages over UWB radar sensing. Millimeter-wave FMCW radar sensing has higher resolution, can operate effectively at greater distances, and provides better noise robustness. Although the electromagnetic signals in millimeter-wave FMCW radar sensors cannot penetrate the human body, it is very sensitive to minute displacements of the human body, including chest movements.

[0032] Figure 1A shows a graph illustrating the sensing of a stationary object by an FMCW radar using transmitted and received linear frequency modulated (LFM) signals. The FMCW radar transmits LFM, a continuous wave whose frequency is linearly modulated. When represented on a frequency-to-time graph, LFM, which increases linearly with time, is represented by a straight line with a certain slope. LFM has a start frequency and an end frequency, and the difference between its start and end frequencies can be called its bandwidth B. The bandwidth B and the time period tc determine the slope S of the LFM, where the slope S defines the rate of LFM ramp-up. The LFM is reflected by the object and received by the antenna. Assuming there is only one object in front of the radar, the received LFM is a delayed copy of the transmitted LFM. The round-trip delay τ is proportional to the distance to the target object. When a subsequent LFM is transmitted and the reflected LFM results in the same round-trip delay τ, the object in Figure 1A is stationary.

[0033] Figure 1B shows a graph illustrating how an FMCW radar senses a non-stationary object using transmitted and received linear frequency modulated (LFM) signals. The FMCW radar sensor transmits continuous LFM signals. Each LFM has round-trip delays τ1, τ2, τ3, and τ4. The round-trip delay τ1 can differ from the round-trip delay τ2, and the round-trip delay τ2 can differ from the round-trip delay τ3, and the round-trip delay τ3 can differ from the round-trip delay τ4. Furthermore, each LFM can differ in phase. The changes in round-trip delay and phase indicate that the target object is changing position (i.e., moving) relative to the FMCW radar sensor. Vital signs can be obtained by analyzing the phase changes associated with the target's position. To measure vibrations such as heartbeats or breathing, the displacement of the object may be a fraction of the wavelength (e.g., one millimeter or less).

[0034] Figure 1C shows a graph illustrating the measurement of heart rate using displacement sensed by FMCW radar as a function of time. Because the round-trip delay τ can be correlated with the displacement Δd, the displacement Δd can be mapped onto a graph showing the displacement Δd as a function of time. The displacement Δd can indicate heartbeat or respiration. The displacement Δd can appear as oscillating motion.

[0035] Obtaining accurate cardiovascular and respiratory measurements using FMCW radar sensing technology presents numerous challenges. Typically, highly directional FMCW radar sensors are better able to distinguish target objects from surrounding clutter. However, for optimal performance, high directionality requires the FMCW radar sensor's viewing angle to be pointed towards the person's chest. Furthermore, the subject's position and body orientation can enhance or impair the performance of the FMCW radar sensor. For example, if the subject's right side or back is facing the FMCW radar sensor, the measurement error may increase slightly. The subject's setup and body movement can interfere with the FMCW radar sensor's ability to obtain accurate cardiovascular and respiratory measurements. Because FMCW radar sensors can be highly sensitive to motion artifacts, signals caused by various movements may be misinterpreted as heartbeats or respiration. In one example, bumps or turbulence in a vehicle can cause bodily vibrations, interfering with cardiovascular or respiratory measurements. In another example, a person in bed may change position and / or orientation, which could interfere with cardiovascular or respiratory measurements.

[0036] This disclosure relates to a multi-sensor system for monitoring cardiovascular and respiratory data. The multi-sensor system includes a millimeter-wave FMCW radar sensor and one or more proximity sensors. The millimeter-wave FMCW radar sensor acquires cardiovascular measurements (such as heart rate) or respiratory measurements (such as respiratory rate). The one or more proximity sensors can detect the presence of a target subject. In some implementations, the one or more proximity sensors can detect the position of the target subject's chest, thereby allowing the viewing angle of the millimeter-wave FMCW radar sensor to be adjusted to point towards the target subject's chest. In some implementations, the one or more proximity sensors can detect the target subject's body position and / or orientation. The multi-sensor system may also include one or more inertial measurement unit (IMU) sensors. At least one of the IMU sensors can be used to acquire cardiovascular or respiratory measurements. For example, at least one IMU sensor can be used to acquire a projected heart rate measurement. In some implementations, at least another IMU sensor can be used to detect motion associated with the target subject's environment to filter out noise in the target subject's cardiovascular or respiratory measurements.

[0037] Figure 2 illustrates a schematic diagram of a millimeter-wave FMCW radar sensor and one or more IMU sensors positioned inside a vehicle, according to some implementations. Thus, a multi-sensor system 200 for monitoring cardiovascular or respiratory data can be implemented in a vehicle such as a car, bus, truck, train, airplane, spacecraft, or ship. The multi-sensor system 200 includes a millimeter-wave FMCW radar sensor 210. The millimeter-wave FMCW radar sensor 210 can be positioned anywhere inside the vehicle, such that the millimeter-wave FMCW radar sensor 210 is facing the subject 205. In some implementations, the millimeter-wave FMCW radar sensor 210 can be positioned on a steering wheel, dashboard, center console, overhead console, seat, ceiling, rearview mirror, or windshield. In one example, the millimeter-wave FMCW radar sensor 210 is positioned on a steering wheel 215. In another example, the millimeter-wave FMCW radar sensor 210 is positioned on a rearview mirror 225.

[0038] The multi-sensor system 200 also includes one or more proximity sensors 220. The one or more proximity sensors 220 can be located anywhere within the vehicle interior, enabling them to detect the presence of the subject 205. In some implementations, the one or more proximity sensors 220 can be integrated into the driver's or passenger's seat 235. Specifically, the one or more proximity sensors 220 can be located inside the seat 235, on the outer surface of the seat 235, or below the seat 235. In some implementations, the one or more proximity sensors 220 can include a capacitive proximity sensor array. It should be understood that the one or more proximity sensors 220 can include other types of proximity sensors known in the art.

[0039] The multi-sensor system 200 also includes one or more IMU sensors 230a, 230b. The first IMU sensor 230a can be located anywhere within the vehicle to acquire motion data associated with the subject 205. The first IMU sensor 230a can be a motion sensor, such as an accelerometer, positioned to acquire cardiovascular or respiratory data associated with the subject 205. In some cases, the first IMU sensor 230a can acquire a projected heart rate map of the subject 205. In some implementations, the first IMU sensor 230a can be located within the seat 235, on the outer surface of the seat 235, or below the seat 235. The second IMU sensor 230b can be located anywhere within the vehicle to acquire motion data associated with the vehicle. Such motion data can be related to internal motion activities occurring within the vehicle. In particular, sudden jolting, swaying, turning, abrupt changes, stopping, acceleration, deceleration, and other movements of the vehicle can cause movement or vibration of components within the vehicle. This could cause movement or vibration of the subject 205 inside the vehicle, which could interfere with the monitoring of the subject 205's cardiovascular or respiratory data. The second IMU sensor 230b can record such movements occurring within the vehicle to account for "noise" or "motion artifacts" in the cardiovascular or respiratory data acquired from the first IMU sensor 230a or the millimeter-wave FMCW radar sensor 210. In some implementations, the second IMU sensor 230b can be positioned on the vehicle's steering wheel, dashboard, center console, overhead console, seat, ceiling, rearview mirror, or windshield. For example, the second IMU sensor 230b is positioned on the steering wheel 215.

[0040] In some implementations, one or more proximity sensors 220 may be configured to estimate the position of the subject's chest. As used herein, the position of the subject's chest may also refer to the height of the subject's chest. The position of the subject's chest may be determined relative to the position of the millimeter-wave FMCW radar sensor 210. In some implementations, one or more proximity sensors 220 may acquire an image, shape, or outline of the subject's body in a vehicle. Specifically, one or more proximity sensors 220 may acquire an image, shape, or outline of at least the subject's torso. By using the outline of the subject's body or torso, one or more proximity sensors 220 may calculate the height of the subject's chest.

[0041] Once the position of the subject's chest is determined relative to the millimeter-wave FMCW radar sensor 210, the viewing angle of the millimeter-wave FMCW radar sensor 210 can be adjusted to point towards the subject's chest. In this way, the transmitted signal from the millimeter-wave FMCW radar sensor 210 can be better focused towards the region of interest. This more effectively separates the reflected signals corresponding to heartbeat and respiration from other motion / vibration. The millimeter-wave FMCW radar sensor 210 can be pointed at the subject's chest so that the subject's chest is within the line of sight of the millimeter-wave FMCW radar sensor 210. Because the millimeter-wave FMCW radar sensor 210 can emit highly directional signals, the strongest signal is emitted within the radar sensor's viewing angle, while a weaker signal is emitted outside the viewing angle. The viewing angle represents the radiation direction pattern of the radio frequency signal. Typically, the millimeter-wave FMCW radar sensor 210 generates a signal with a certain observation area, so that motion / vibration is effectively separated within the observation area. For example, the observation area can be within approximately 10 degrees, 20 degrees, 30 degrees, 40 degrees, or 45 degrees of the radar sensor's field of view. Many millimeter-wave FMCW radar sensors capture a cone at 20 to 30 degrees relative to the antenna tip. The strongest signal is given at exactly the midpoint (i.e., 0 degrees) with the optimal signal-to-noise ratio. Some signal attenuation may occur when deviating from the midpoint. Nevertheless, even if the viewing angle is not perfectly aligned with the subject's chest, the subject's chest is likely within the observation area of ​​the millimeter-wave FMCW radar sensor 210. Pointing the millimeter-wave FMCW radar sensor 210 towards the subject's chest allows the measurement to be optimally correlated with heart rate or respiration.

[0042] The millimeter-wave FMCW radar sensor 210 can acquire cardiovascular measurements such as heart rate or respiratory measurements such as respiratory rate. Cardiovascular or respiratory measurements can be acquired after the viewing angle is adjusted to point towards the subject's chest. An exemplary cardiovascular measurement acquired from the millimeter-wave FMCW radar sensor 210 is shown in Figure 4A. The first IMU sensor 230a can also acquire cardiovascular measurements such as heart rate or respiratory measurements such as respiratory rate. For example, the first IMU sensor 230a can acquire a projected heart rate chart recording a heartbeat pattern. Figure 4B shows an exemplary projected heart rate chart. Data from the millimeter-wave FMCW radar sensor 210 and the first IMU sensor 230a can be fused to improve accuracy.

[0043] The first IMU sensor 230a can be used as a projected heart rate monitoring (BCG) system, which can be integrated into a chair, bed, and other everyday objects to record the cardiac activity of the subject 205. In Figure 2, the first IMU sensor 230a can be integrated into a chair 235 and records the cardiac activity of the subject 205 when the subject 205 is seated. The first IMU sensor 230a is sensitive enough to detect body vibrations caused by the mechanical activity of the heart.

[0044] Data from the millimeter-wave FMCW radar sensor 210 can be represented as a first cardiovascular or respiratory measurement, and data from the first IMU sensor 230a can be represented as a second cardiovascular or respiratory measurement. The first and second cardiovascular or respiratory measurements can be combined. The intensity at which signals are filtered out in the cardiovascular or respiratory measurements can depend on a calculated confidence factor associated with the measurement technique. In some implementations, the associated confidence factor can be based on various factors affecting the accuracy of the measurement technique, including but not limited to: noise in the cardiovascular or respiratory measurements, relative differences between the cardiovascular or respiratory measurements, the position or body orientation of the subject 205, the position of the millimeter-wave FMCW radar sensor 210 relative to the subject's chest, the position of the first IMU sensor 230a relative to the subject 205, the periodicity of the signals in the cardiovascular or respiratory measurements, and the sensitivity of the millimeter-wave FMCW radar sensor 210 and the first IMU sensor 230a, as well as other factors. The confidence factor can be estimated through a probabilistic model of the received signals or measurements. The matched filter output can be transformed using the Karhunen-Loève transform to provide a target indication of signal strength directly related to the likelihood function used for statistical hypothesis testing, thus providing a confidence factor associated with the cardiovascular or respiratory measurement. The confidence factor 205 is then further estimated based on whether the cardiovascular or respiratory measurement captures the subject's actual heartbeat or respiration. This can be accomplished using another probabilistic modeling approach employing time-statistical hypotheses. Data fusion of the first and second cardiovascular or respiratory measurements can be performed based on the estimated confidence factors. In some implementations, the measurement with the higher confidence factor is selected. Specifically, the first cardiovascular or respiratory measurement can be selected if its confidence factor is greater than that of the second, or vice versa. In some implementations, the first and second cardiovascular or respiratory measurements are combined in a Bayesian manner, which models measurement indeterminacy by calculating a posterior distribution, the opposite of accepting or rejecting a single-point estimate. The confidence factor of the measurement technique can vary over time as the context of the environment changes. As the relative accuracy of measurement techniques varies over time, the confidence factor associated with different measurement techniques can also vary over time. This can occur when settings change and / or the subject moves around.

[0045] The merging of the first and second cardiovascular or respiratory measurements combines the data streams to provide the optimal cardiovascular or respiratory measurement. The optimal cardiovascular or respiratory measurement may be a heartbeat pattern or respiratory rate pattern represented as a single data stream to improve accuracy and reliability.

[0046] In some implementations, motion data associated with the vehicle acquired from the second IMU sensor 230b can be combined with a first cardiovascular or respiratory measurement and / or a second cardiovascular or respiratory measurement. Various motion artifacts caused by the vehicle may interfere with accurate cardiovascular or respiratory measurements. Such motion artifacts caused by the vehicle may manifest as heartbeats or breathing in cardiovascular or respiratory measurements. Motion artifacts may manifest as spikes or anomalies in the data stream. Figure 4C shows an exemplary data stream used to detect motion artifacts caused by the vehicle.

[0047] The second IMU sensor 230b can record such motion artifacts, thereby allowing any motion artifacts caused by the vehicle to be subtracted from the first cardiovascular or respiratory measurement and / or the second cardiovascular or respiratory measurement. The noise-free first cardiovascular or respiratory measurement can be obtained by calculating the relative difference between the first cardiovascular or respiratory measurement and the motion data associated with the vehicle. Alternatively, the noise-free first cardiovascular or respiratory measurement can be obtained by calculating the relative difference between the second cardiovascular or respiratory measurement and the motion data associated with the vehicle. Or, the noise-free optimal cardiovascular or respiratory measurement can be obtained by calculating the relative difference between the optimal cardiovascular or respiratory measurement and the motion data associated with the vehicle. Eliminating body motion artifacts from the optimal (fused) cardiovascular or respiratory measurement improves accuracy and reliability.

[0048] As shown in Figure 2, one or more proximity sensors 220 can be used to detect the presence of a driver / passenger in a vehicle and determine the height or position of the subject's chest. This information can be used to optimize the viewing angle of the millimeter-wave FMCW radar sensor 210 to improve the accuracy and performance of cardiovascular or respiratory measurements. Furthermore, a first IMU sensor 230a and a second IMU sensor 230b supplement, enhance, adjust, or improve the data collected by the millimeter-wave FMCW radar sensor 210. The first IMU sensor 230a can acquire motion data associated with the subject's body to combine with cardiovascular or respiratory measurements acquired from the millimeter-wave FMCW radar sensor 210. Such motion data can be correlated with heart rate or respiratory rate. Alternatively, such motion data can be used to remove noise from body motion artifacts. The second IMU sensor 230b can acquire motion data associated with the vehicle. Such motion data can be used to remove noise from motion artifacts that may appear in cardiovascular or respiratory measurements.

[0049] Optimal, noise-reduced cardiovascular or respiratory measurements can be analyzed to determine information about subject 205. In some implementations, the optimal, noise-reduced cardiovascular or respiratory measurements can be used to determine different metrics, such as subject 205's stress level, drowsiness level, or health level. As an example, after measuring heart rate or respiratory rate, the heart rate or respiratory rate can be used to determine the drowsiness level of the driver or passenger in the vehicle. If the drowsiness level meets a certain threshold, a predefined action can be performed. Such predefined actions may include, but are not limited to: instant audio alarm, instant light alarm, instant telephone alarm, instant video alarm, engine / brake control, contacting a remote operator, etc. These actions are intended to attempt to wake the driver or passenger. In some implementations, multiple setups of millimeter-wave FMCW radar sensor 210, proximity sensor 220, first IMU sensor 230a, and second IMU sensor 230b can be arranged in the vehicle to obtain cardiovascular or respiratory measurements of multiple occupants. The measured heart rate or respiratory rate can be used to determine each passenger's stress level, drowsiness level, or health level. For example, the volume of the vehicle's audio system can be adjusted based on the passengers' level of drowsiness if one or more passengers fall asleep. Alternatively, the temperature of the vehicle's air conditioning system can be adjusted based on the passengers' level of drowsiness.

[0050] Figure 3 illustrates a schematic diagram of one or more millimeter-wave FMCW radar sensors and IMU sensors positioned in a room according to some implementations. A multi-sensor system 300 for monitoring cardiovascular or respiratory data can be implemented in a room with a subject 305. The room can be any space in which the subject 305 can reside for a relatively long period, thereby enabling continuous tracking of cardiovascular or respiratory data. In some implementations, the room can be a hospital room, bedroom, library, office space, or other space where the subject 305 may lie, sit, or remain relatively still for extended periods. For example, the room can be a hospital room.

[0051] A multi-sensor system 300 for monitoring cardiovascular or respiratory data may include one or more millimeter-wave FMCW radar sensors 310. In some implementations, the multiple millimeter-wave FMCW radar sensors 310 may be positioned at different locations in a room. Each millimeter-wave FMCW radar sensor 310 may be positioned in the room facing the subject 305. Alternatively, each millimeter-wave FMCW radar sensor 310 may be positioned in the room facing at least a stationary object, such as a bed or chair, on which the subject 305 will be placed. In some implementations, the millimeter-wave FMCW radar sensors 310 are positioned on walls, ceilings, or stationary objects surrounding the room.

[0052] The multi-sensor system 300 also includes one or more proximity sensors 320. The one or more proximity sensors 320 can be positioned anywhere in the room to detect the presence of the subject 305. Furthermore, the one or more proximity sensors 320 can be positioned anywhere in the room to determine the body orientation and / or position of the subject 305 within the room. In some implementations, the one or more proximity sensors 320 are integrated into the bed 335 of the room. For example, the one or more proximity sensors 320 can be positioned inside the bed 335, on the outer surface of the bed 335, or under the bed 335. It should be understood that the one or more proximity sensors 320 can be integrated into any stationary object on or near which the subject 305 may be located. Other stationary objects may include, but are not limited to, chairs, sofas, pillows, cushions, tables, etc. In some implementations, the one or more proximity sensors 320 may include an array of capacitive proximity sensors. In some implementations, the one or more proximity sensors 320 may include a time-of-flight sensor. In some implementations, one or more proximity sensors 320 may include an optical sensor (camera). In some implementations, one or more proximity sensors 320 may include one or more pressure transducers. It should be understood that one or more proximity sensors may include other types of proximity sensors known in the art.

[0053] The multi-sensor system 300 also includes an IMU sensor 330. The IMU sensor 330 can be positioned anywhere in a room to acquire motion data associated with the subject 305. The IMU sensor 330 may be a motion sensor positioned to acquire cardiovascular or respiratory data associated with the subject 330. In some cases, the IMU sensor 330 may acquire a projected heart rate chart of the subject 305. In some implementations, the IMU sensor 330 may be positioned inside the bed 335, on the outer surface of the bed 3335, or under the bed 335. In some implementations, the IMU sensor 330 and one or more proximity sensors 320 may be positioned in the same location, for example, under the bed 335. Strong signals recorded by the IMU sensor 330 may indicate body movements independent of heart rate or respiratory patterns and may be used to denoise cardiovascular or respiratory measurements acquired from one or more millimeter-wave FMCW radar sensors 310. Prior knowledge or information about how much acceleration the human body typically generates due to blood flow or respiration can establish a threshold acceleration value, and acceleration values ​​much higher than the threshold acceleration value (e.g., at least twice as high) can indicate body movement independent of heartbeat or breathing patterns. Weak signals recorded by IMU sensor 330 can indicate heartbeat or breathing patterns and can be combined with cardiovascular or respiratory measurements acquired from one or more millimeter-wave FMCW radar sensors 310.

[0054] In some implementations, one or more proximity sensors 320 may be configured to estimate the body orientation of the subject 305. The body orientation of the subject 305 may refer to the direction in which the frontal plane (front) of the body faces in three-dimensional (3-D) space. The body orientation may be relative to one of the objects in the room, such as the bed 335 or the millimeter-wave FMCW radar sensor 310. Thus, one or more proximity sensors 320 may determine whether the subject 305 is facing forward, backward, left, right, prone, supine, left lateral, or right lateral. In some implementations, one or more proximity sensors 320 may acquire a map, shape, or outline of the subject's body or at least the subject's torso. In some implementations, one or more proximity sensors 320 may acquire the position (e.g., height) of the subject's chest relative to the bed 335.

[0055] Once the subject's (305's) body orientation relative to the room is determined, at least one of one or more millimeter-wave FMCW radar sensors (310) can be selected to obtain cardiovascular or respiratory measurements of the subject (305). When multiple millimeter-wave FMCW radar sensors (310) are positioned at different locations in the room, only some of the millimeter-wave FMCW radar sensors (310) are positioned to face the subject's forehead. At least one millimeter-wave FMCW radar sensor (310) is selected such that the subject's forehead is within the line of sight of at least one millimeter-wave FMCW radar sensor (310). Utilizing millimeter-wave FMCW radar sensors (310) facing the subject's forehead improves the accuracy of cardiovascular or respiratory measurements. Frontal measurements provide the most accurate measurements, left-side measurements provide relatively high accuracy but slightly lower than frontal measurements, back-side measurements provide slightly lower accuracy than left-side measurements, and right-side measurements provide the worst accuracy. At least one millimeter-wave FMCW radar sensor (310) facing the subject's body orientation can be selected to provide the most accurate measurements.

[0056] In some implementations, the multi-sensor system 300 may be limited to a single millimeter-wave FMCW radar sensor 310 in a room. The single millimeter-wave FMCW radar sensor 310 may not be pointing towards the subject's forehead. In this case, a confidence factor (e.g., error rate, success rate) can be calculated to determine the accuracy of cardiovascular or respiratory measurements relative to the single millimeter-wave FMCW radar sensor 310. The confidence factor depends on the body orientation of the subject 305 relative to the single millimeter-wave FMCW radar sensor 310. The confidence factor can account for errors that may occur when obtaining cardiovascular or respiratory measurements when the subject's forehead is not within the line of sight of the millimeter-wave FMCW radar sensor 310. Alternatively, in some cases where there are multiple millimeter-wave FMCW radar sensors 310 in a room, none of the millimeter-wave FMCW radar sensors 310 may be pointing towards the subject's forehead. In this case, a confidence factor can be calculated for each millimeter-wave FMCW radar sensor 310, and the millimeter-wave FMCW radar sensor 310 with the highest confidence factor can be selected.

[0057] In some implementations, the viewing angle of at least one millimeter-wave FMCW radar sensor 310 is adjusted to point towards the subject's chest. This can occur after the position of the subject's chest has been obtained using one or more proximity sensors 320. The millimeter-wave FMCW radar sensor 310 can be pointed towards the subject's chest such that the subject's chest is within the line of sight of at least one millimeter-wave FMCW radar sensor 310. Adjusting the viewing angle of at least one millimeter-wave FMCW radar sensor to point towards the subject's chest can optimize the accuracy of cardiovascular or respiratory measurements obtained by at least one millimeter-wave FMCW radar sensor 310. At least one millimeter-wave FMCW radar sensor 310 can emit a signal with a defined observation area, such that motion / vibration is effectively separated within the observation area. For example, the observation area can be within 10 degrees, approximately 20 degrees, approximately 30 degrees, approximately 40 degrees, or approximately 45 degrees of the radar sensor's viewing angle. Therefore, the subject's chest may be within the observation area of ​​the millimeter-wave FMCW radar sensor 310, even if the viewing angle is not perfectly aligned with the subject's chest.

[0058] At least one millimeter-wave FMCW radar sensor 310 can acquire cardiovascular measurements such as heart rate or respiratory measurements such as respiratory rate. Cardiovascular or respiratory measurements can be acquired after the viewing angle is adjusted to point towards the subject's chest. IMU sensor 330 can also acquire cardiovascular measurements such as heart rate or respiratory measurements such as respiratory rate. For example, IMU sensor 330 can acquire a projected heart rate chart recording a heartbeat pattern. In Figure 3, IMU sensor 330 can be integrated into bed 335 and record cardiac activity while the subject 305 is on bed 335. IMU sensor 330 is sensitive enough to detect body vibrations caused by the mechanical activity of the heart. Data from at least one millimeter-wave FMCW radar sensor 310 and IMU sensor 330 can be fused to improve accuracy.

[0059] Data from at least one millimeter-wave FMCW radar sensor 310 can be represented as a first cardiovascular or respiratory measurement, and data from the IMU sensor 330 can be represented as a second cardiovascular or respiratory measurement. The first and second cardiovascular or respiratory measurements can be combined. The merging technique for the first and second cardiovascular or respiratory measurements is described in Figure 2 above. The first cardiovascular or respiratory measurement may have an associated confidence factor, and the second cardiovascular or respiratory measurement may have an associated confidence factor. Each confidence factor is based at least in part on the person's body orientation and / or the difference between the first and second cardiovascular or respiratory measurements. Therefore, the confidence factor of the measurement technique may depend on whether the subject 305 is sitting, prone, supine, lying on the left side, or lying on the right side. For example, a lower confidence factor may be estimated based on how the subject 305 faces the millimeter-wave FMCW radar sensor 310 or the proximity of the subject 305 to the IMU sensor 330. The first cardiovascular or respiratory measurement and the second cardiovascular or respiratory measurement are combined according to an algorithm based on a first association confidence factor and a second association confidence factor to obtain the optimal cardiovascular or respiratory measurement.

[0060] The merging of the first and second cardiovascular or respiratory measurements combines the data streams to provide the optimal cardiovascular or respiratory measurement. The merging of the first and second cardiovascular or respiratory measurements may undergo an adaptive filtering process. The filter parameters change based on the inputs to the IMU sensor 330 and / or the millimeter-wave FMCW radar sensor 310. In adaptive filtering, the heartbeat or respiratory cycle typically evolves smoothly over time. Smoothing constraints on the heartbeat or respiratory state variables can be applied during the estimation process. The adaptive filtering process only allows cardiovascular or respiratory measurements that satisfy the smoothness constraints to be considered as heartbeats or respiration. Otherwise, the cardiovascular or respiratory measurements are canceled or removed. This is a time-varying adaptive process, depending on the current heart rate or respiratory rate state.

[0061] The second cardiovascular or respiratory measurement provided by the IMU sensor 330 can help fuse or filter data. As described above, a weak vibration signal observed by the IMU sensor 330 may be suitable as a cardiovascular or respiratory measurement, while a strong motion signal observed by the IMU sensor 330 may be suitable as a motion artifact for noise removal. Compared to the weak vibration signal, the strong motion signal may exhibit a spike with an amplitude at least twice that of the weak vibration signal. The strong motion signal recorded by the IMU sensor 330 can be subtracted from the first cardiovascular or respiratory measurement provided by the millimeter-wave FMCW radar sensor 310. The weak vibration signal recorded by the IMU sensor 330 can be added to the first cardiovascular or respiratory measurement provided by the millimeter-wave FMCW radar sensor 310, wherein the millimeter-wave FMCW radar sensor 310 does not record heartbeat or respiratory patterns. Alternatively, a weak vibration signal recorded by the IMU sensor 330 can be added to the first cardiovascular or respiratory measurement provided by the millimeter-wave FMCW radar sensor 310, where the confidence factor associated with the millimeter-wave FMCW radar sensor 310 is low.

[0062] As shown in Figure 3, one or more proximity sensors 320 can be used to detect the presence of a subject in a room and determine the body orientation of the subject 305. In some implementations, the one or more proximity sensors 320 further determine the height or position of the subject's chest. This information can be used to optimize the viewing angle of one or more millimeter-wave FMCW radar sensors 310 to improve the accuracy and performance of obtaining cardiovascular or respiratory measurements. Furthermore, the IMU sensor 330 can acquire motion data associated with the subject's body. The motion data may be related to heart rate or respiratory rate over certain time periods, or the motion data may be related to body motion artifacts over other time periods. The motion data associated with the subject's body can be combined with cardiovascular or respiratory measurements acquired from one or more millimeter-wave FMCW radar sensors 310. In some cases or over certain time periods, the motion data associated with the subject's body can be added to, subtracted from, averaged together with, or otherwise combined with the cardiovascular or respiratory measurements acquired from one or more millimeter-wave FMCW radar sensors 310. How measurements are combined can depend on factors such as confidence factors related to the measurement technique and the subject's 30° body orientation. The aforementioned combination provides the best cardiovascular or respiratory measurements after noise removal.

[0063] Optimal, noise-reduced cardiovascular or respiratory measurements can be analyzed to determine information about the subject 305. In some implementations, optimal, noise-reduced cardiovascular or respiratory measurements can be used to monitor the vital signs or other health measures of the subject 305. As an example, the heart rate or respiratory rate of a patient in a hospital bed can be monitored. If the heart rate or respiratory rate falls below a threshold level, an alarm can be triggered to a doctor or other healthcare professional. In another example, heart rate variability can be estimated using optimal, noise-reduced cardiovascular measurements, which can be used to correlate with many chronic diseases. It can also be used to correlate with stress levels or anxiety. In yet another example, respiratory measurements can be used to detect speech, which can be analyzed to correlate with the subject's neurological health.

[0064] Figure 4A shows a graph illustrating an exemplary cardiovascular measurement obtained from a millimeter-wave FMCW radar sensor. The cardiovascular measurement could be the subject's heart rate. The millimeter-wave FMCW radar sensor can detect minute changes in vibration or displacement over time. When transmitting linear frequency modulation, minute changes in displacement may be related to round-trip delay. These minute changes in displacement can be visually represented as oscillating motion on the graph shown in Figure 4A. Interferences 410a and 410b may appear on the graph, interfering with the oscillating motion. Interferences 410a and 410b may be caused by body motion artifacts unrelated to the subject's heartbeat or respiration.

[0065] Figure 4B shows an exemplary projected pacemaker chart capturing a heartbeat pattern obtained from an IMU sensor. The projected pacemaker chart can graphically represent repetitive bodily movements caused by a sudden ejection of blood from a subject's blood vessels. The projected pacemaker chart can be obtained from an IMU such as an accelerometer. Vibrations from the subject's body may be transferred to another object (e.g., a chair, bed, scale) and recorded by the accelerometer to record cardiac activity. Interferences 420a, 420b, and 420c can appear as "noise" on the projected pacemaker chart. In some cases, this noise may be the result of motion induced on the object to which the accelerometer is attached, and such motion is unrelated to the subject's heartbeat or breathing. One or more of the interferences 420a, 420b, and 420c in Figure 4B can be correlated with the interferences 410a and 410b in Figure 4A.

[0066] Figure 4C shows a graph illustrating the motion activity of a moving vehicle obtained from an IMU sensor. The IMU sensor may be an accelerometer attached to an object in the vehicle. Motion activity (e.g., vibration) caused by the object in the vehicle can generate signals 430a, 430b. These signals 430a, 430b correspond to motion unrelated to the subject's heartbeat or breathing. These signals 430a, 430b may be caused by, for example, sudden movements in the vehicle (such as bumps, dips, or turns). These signals 430a, 430b may be correlated with interferences 410a, 410b for noise reduction of the cardiovascular measurements in Figure 4A, or with interferences 420a, 420b, 420c for noise reduction of the projected heart rate graph in Figure 4B.

[0067] Figure 4D shows a graph illustrating optimized cardiovascular measurements obtained by merging data from multiple sensors according to some implementation methods. Optimized cardiovascular measurements can record heartbeat patterns using motion artifacts from one or both of the IMU sensor data in Figures 4B and 4C after noise removal. Optimized cardiovascular measurements can eliminate interferences 410a, 410b in Figure 4A and interferences 420a, 420b, 420c in Figure 4B, and fuse heartbeat data from Figures 4A and 4B to obtain more accurate and reliable heartbeat patterns. Optimized cardiovascular measurements can record heartbeats 440a, 440b, 440c, and 440d by merging measurements from Figures 4A to 4C.

[0068] Figure 5 shows a flowchart illustrating an exemplary process of using a multi-sensor system to calculate combined cardiovascular or respiratory measurements according to some implementations. Process 500 can be performed in different orders or with different, fewer, or additional operations. Blocks of process 500 can be executed by one or more processors of the control system. In some implementations, blocks of process 500 can be implemented at least in part according to software stored on one or more non-transitory computer-readable media.

[0069] In block 510 of process 500, one or more proximity sensors may optionally be used to identify a subject inside a vehicle or room. The one or more proximity sensors may include an array of capacitive proximity sensors. Alternatively, the one or more proximity sensors may include other types of sensors, such as optical sensors, photoelectric sensors, time-of-flight sensors, and pressure sensors. Such sensors may be used to determine whether a subject is inside a vehicle or room. The identified subject may be any mammalian subject, such as a human. In some implementations, the one or more proximity sensors may determine that the subject is stationary or substantially stationary at a location within the vehicle or room for a sufficiently long duration. A sufficiently long duration may be on the order of seconds or minutes, such as approximately 3 seconds, 5 seconds, 10 seconds, 30 seconds, 1 minute, 2 minutes, or 3 minutes. In this way, when it is determined that the subject has not entered or exited the vehicle or room, cardiovascular or respiratory measurements may be initiated. In some implementations, the vehicle may include a car, bus, truck, train, airplane, spacecraft, or ship. In some implementations, the room may include a hospital room, bedroom, library, office space, or other enclosed space in which the subject may remain relatively still for a long period of time.

[0070] The multi-sensor system disclosed herein may include one or more proximity sensors and a control system. The control system may include a processor and / or memory. The processor may be programmable with processor-executable instructions. The processor may be a programmable microprocessor, microcomputer, or multiprocessor chip(s) that can be configured by software instructions to perform various functions associated with the multi-sensor system. The memory may store processor-executable instructions and data obtained from one or more proximity sensors and other sensors. In some implementations, the memory may be volatile memory, non-volatile memory (e.g., flash memory), or a combination thereof. The control system may be in direct or indirect electrical communication with one or more proximity sensors and other sensors. The control system may receive data (e.g., raw data) from one or more proximity sensors and other sensors. The control system may include dedicated hardware specifically adapted to perform various functions of the multi-sensor system.

[0071] In block 520 of process 500, one or more proximity sensors are used to determine the position of the subject's chest and / or body orientation. As an additional or alternative to detecting the subject's presence, one or more proximity sensors may detect how the subject is positioned relative to a vehicle or room. Determining how the subject is positioned relative to a vehicle or room will help other sensors obtain accurate cardiovascular or respiratory measurements. In some implementations, one or more proximity sensors may acquire a map, shape, or outline of the subject's body. Using this information, one or more proximity sensors may determine the position of the subject's chest relative to other sensors in the vehicle or room. In some implementations, one or more proximity sensors may determine the subject's body orientation relative to other sensors in the vehicle or room. Possible body orientations may include facing forward, facing backward, facing left, facing right, prone, supine, left lateral, or right lateral. Other sensors in the room may include one or more millimeter-wave FMCW radar sensors or one or more IMU sensors.

[0072] The control system may receive positioning data from one or more proximity sensors, wherein the control system may use the positioning data to identify the presence of the subject in a vehicle or room, and the control system may use the positioning data to determine the position of the subject's chest and / or the subject's body orientation. In some implementations, the control system may use the positioning data to determine the position of the subject's chest relative to a millimeter-wave FMCW radar sensor. In some implementations, the control system may use the positioning data to determine the subject's body orientation relative to a millimeter-wave FMCW radar sensor.

[0073] In block 530 of process 500, the viewing angle of a millimeter-wave FMCW radar sensor positioned inside a vehicle or room is adjusted based on the position of the subject's chest and / or body orientation. The viewing angle can also be referred to as the "elevation angle." Pointing the millimeter-wave FMCW radar sensor at the subject's chest improves the performance and reliability of the millimeter-wave FMCW radar sensor when acquiring cardiovascular or respiratory measurements. For the sensor to be pointed at the subject's chest, the subject's chest must be within the observation area of ​​the viewing angle. The millimeter-wave FMCW radar sensor can obtain optimized cardiovascular or respiratory measurements for signals transmitted within the observation area. In some implementations, the subject's chest may be within 10 degrees of the radar sensor's viewing angle to be within the observation area. In some implementations, the subject's chest may be within 20 degrees of the radar sensor's viewing angle to be within the observation area.

[0074] The multi-sensor system also includes a millimeter-wave FMCW radar sensor. The control system can communicate electrically with the millimeter-wave FMCW radar sensor (directly or indirectly). After determining the position of the subject's chest and / or the subject's body orientation, the control system can adjust the viewing angle of the millimeter-wave FMCW radar sensor based on the above determination. The viewing angle of the millimeter-wave FMCW radar sensor can be pointed towards the subject's chest or within a few degrees of the subject's chest.

[0075] In some implementations, process 500 further includes selecting a millimeter-wave FMCW radar sensor from a plurality of millimeter-wave FMCW radar sensors located in a vehicle or room. The plurality of millimeter-wave FMCW radar sensors can be positioned at different locations around the vehicle or room. After determining the subject's body orientation, one of the millimeter-wave FMCW radar sensors can be selected. The selected millimeter-wave FMCW radar sensor can face the subject's forehead. By selecting a millimeter-wave FMCW radar sensor facing the subject's forehead, the accuracy of cardiovascular or respiratory measurements using the millimeter-wave FMCW radar sensor is improved. In some implementations, the selected millimeter-wave FMCW radar sensor can be closer to the subject than other millimeter-wave FMCW radar sensors. The proximity of the millimeter-wave FMCW radar sensor to the subject can be in the range of approximately 5 m or less, approximately 3 m or less, or approximately 1 m or less. Generally, the performance and reliability of the millimeter-wave FMCW radar sensor are improved when placed closer to the subject. When millimeter-wave FMCW radar sensors are placed too far away, not only may the signal be attenuated, but certain objects are also more likely to block the signal path, thus preventing the millimeter-wave FMCW radar sensors from functioning.

[0076] In block 540 of process 500, a first cardiovascular or respiratory measurement is obtained from a millimeter-wave FMCW radar sensor. The first cardiovascular measurement may be a heartbeat pattern used to determine heart rate, and the first respiratory measurement may be a breathing pattern used to determine respiratory rate. The first cardiovascular or respiratory measurement can be obtained after adjusting the viewing angle of the millimeter-wave FMCW radar sensor. The millimeter-wave FMCW radar sensor can transmit and receive signals with wavelengths in the millimeter range to detect minute movements and vibrations. These movements and vibrations may correspond to the heartbeat or breathing of the subject. Heartbeat and human breathing can be distinguished from other movements of the subject. The millimeter-wave FMCW radar sensor can be positioned at any suitable location inside a vehicle or room. In some implementations, the millimeter-wave FMCW radar sensor can be positioned on the steering wheel, dashboard, center console, overhead console, seat, ceiling, rearview mirror, or windshield of the vehicle. In some other specific implementations, the millimeter-wave FMCW radar sensor can be positioned on a wall, ceiling, or stationary object in a room.

[0077] In block 550 of process 500, a second cardiovascular or respiratory measurement can be obtained from an IMU sensor. The second cardiovascular measurement may be a heartbeat pattern used to determine heart rate, and the second respiratory measurement may be a respiratory pattern used to determine respiratory rate. The second cardiovascular or respiratory measurement can be obtained simultaneously with the first cardiovascular or respiratory measurement. The IMU sensor can be positioned in a vehicle or room to record motion activities associated with the subject. Motion activities can indicate the subject's heartbeat or respiration. In some implementations, the IMU sensor may be an accelerometer positioned to record vibrations transmitted from the subject's body to another object. Specifically, the IMU sensor can obtain a projected heart rate map for recording the subject's heartbeat pattern. The IMU sensor can be integrated into or positioned on a stationary object in a vehicle or room. The stationary object may be in contact with the subject. In some implementations, the IMU sensor can be positioned inside a seat, on the outer surface of a seat, or under a seat. In some implementations, the IMU sensor can be positioned inside a bed, on the outer surface of a bed, or under a bed.

[0078] The multi-sensor system also includes an IMU sensor. The IMU sensor may include an accelerometer. The control system may communicate electrically with the IMU sensor (directly or indirectly). Data from the IMU sensor and data from the millimeter-wave FMCW radar sensor may be received and processed by the control system. Data from both the IMU sensor and the millimeter-wave FMCW radar sensor may record the subject's motion activities. In some implementations, data from the IMU sensor or the millimeter-wave FMCW radar sensor may include body motion artifacts. Data associated with body motion artifacts may be identified by the control system and subsequently removed when calculating cardiovascular or respiratory measurements. The identification and removal of body motion artifacts may be performed using statistical methods known in the art.

[0079] In some implementations, process 500 further includes obtaining internal motion data associated with the vehicle from a second IMU sensor. The internal motion data associated with the vehicle may be limited to internal motion activities occurring within the vehicle. Such internal motion activities may be caused by bumps, dips, turns, sudden movements, stops, accelerations, decelerations, and other movements that may cause movement or vibration of objects within the vehicle. Internal motion activities may be identified as noise or motion artifacts to be subtracted from a first cardiovascular or respiratory measurement or from a second cardiovascular or respiratory measurement. The second IMU sensor may be located on the vehicle's steering wheel, dashboard, center console, overhead console, seats, ceiling, rearview mirror, or windshield. The multi-sensor system may also include a second IMU sensor, wherein the control system may communicate electrically (directly or indirectly) with the second IMU sensor.

[0080] In block 560 of process 500, the first cardiovascular or respiratory measurement and the second cardiovascular or respiratory measurement are combined to obtain the optimal cardiovascular or respiratory measurement. The combination may include noise reduction to remove unwanted noise or motion artifacts unrelated to heartbeat or respiration. The control system receives data from one or more proximity sensors, data from a millimeter-wave FMCW radar sensor including the first cardiovascular or respiratory measurement, and data from an IMU sensor including the second cardiovascular or respiratory measurement. The data from the millimeter-wave FMCW radar sensor may have a first associated confidence factor or filter parameter, and the data from the IMU sensor may have a second associated confidence factor or filter parameter. Each associated confidence factor may be based in part on the data itself, such as its signal-to-noise ratio. Each associated confidence factor may be based in part on the subject's body orientation. Depending on how the subject is positioned in a vehicle or room, or depending on whether the subject is in direct contact with the object on which the IMU sensor is placed, the millimeter-wave FMCW radar sensor may be more reliable, or the IMU sensor may be more reliable. Each associated confidence factor can be based in part on how the sensor is positioned relative to the subject within the vehicle or room. The reliability of the millimeter-wave FMCW radar sensor may depend on the presence of interfering objects obstructing the field of view of the sensor, or on the proximity of the millimeter-wave FMCW radar sensor or IMU sensor to the subject. Each associated confidence factor may change over time as the environment within the vehicle or room changes.

[0081] After calculating the first and second correlation confidence factors, the algorithm fuses the first and second cardiovascular or respiratory measurements. The strength of signal filtering from the cardiovascular or respiratory measurements depends on the correlation confidence factor. The first and second cardiovascular or respiratory measurements are merged in a manner that removes unwanted noise or motion artifacts. Furthermore, the first and second cardiovascular or respiratory measurements are merged in a manner that filters out less reliable measurements (i.e., those with lower correlation confidence factors) to a greater extent, while more reliable measurements (i.e., those with higher correlation confidence factors) are weighted or selected more by the algorithm. Thus, merging removes motion artifacts and fuses the data to obtain more reliable cardiovascular or respiratory measurements.

[0082] As used herein, the phrase “at least one” in the list of items refers to any combination of those items, including individual members. For example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc.

[0083] The various exemplary logics, logic blocks, modules, circuits, and algorithmic processes described in conjunction with the specific embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. The interchangeability of hardware and software has been generally described in terms of functionality and illustrated in the various exemplary components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0084] Hardware and data processing apparatuses for implementing the various exemplary logics, logic blocks, modules, and circuits described in conjunction with the aspects disclosed herein may be implemented or executed using general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may be a microprocessor, or any known processor, controller, microcontroller, or state machine. A processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, a combination of one or more microprocessors with a DSP core, or any other such configuration. In some implementations, specific processes and methods may be executed by circuitry specific to a given function.

[0085] In one or more aspects, the described functions can be implemented by hardware, digital electronic circuits, computer software, firmware, the structures disclosed in this specification and their equivalents, or any combination thereof. Specific implementations of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a computer storage medium for execution by a data processing device or to control the operation of a data processing device.

[0086] If implemented in software, the functionality can be stored as one or more instructions or codes on or transmitted through a computer-readable medium such as a non-transitory medium. The processes of the methods or algorithms disclosed herein can be implemented as processor-executable software modules that can reside on a computer-readable medium. Computer-readable media include computer storage media and communication media, the latter including any media capable of transferring computer programs from one place to another. Storage media can be any available media accessible by a computer. By way of example and not limitation, non-transitory media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other media that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Additionally, any connection can be properly referred to as computer-readable media. As used herein, magnetic disks and optical disks include optical discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, wherein magnetic disks typically magnetically copy data, while optical discs optically copy data using lasers. The above combinations should also be included within the scope of computer-readable media. Furthermore, the operation of a method or algorithm may reside as one or any combination or set of codes and instructions on machine-readable and computer-readable media, and may be incorporated into a computer program product.

[0087] Various modifications to the specific embodiments described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other specific embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the specific embodiments shown herein, but should be given the broadest scope consistent with the claims, the principles disclosed herein, and the novel features.

[0088] Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed to be so, in some cases, one or more features of the claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0089] Similarly, although operations are depicted in a specific order in the figures, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential order, or that all shown operations must be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, other embodiments fall within the scope of the appended claims. In some cases, the actions described in the claims can be performed in a different order and the desired result can still be achieved.

[0090] It should be understood that unless features in any particular embodiment are explicitly marked as incompatible with each other or the surrounding context suggests that they are mutually exclusive and not readily combinable in a complementary and / or supporting sense, the entirety of this disclosure is capable of conceiving and envisioning that specific features of these complementary embodiments can be selectively combined to provide one or more comprehensive but slightly different technical solutions. Therefore, it should also be understood that the above description is given by way of example only and may be modified in detail within the scope of this disclosure. [Simplified Explanation of the Diagram]

[0013] Details of one or more specific embodiments of the subject matter described in this specification are set forth in the drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Note that the relative dimensions in the following drawings may not be drawn to scale.

[0014] Similar symbolic labels and names in various diagrams indicate the same elements.

[0015] Figure 1A shows a graph illustrating how the FMCW radar senses stationary objects using transmitted and received linear frequency modulation (chirp) signals.

[0016] Figure 1B shows a graph illustrating how the FMCW radar senses non-stationary objects using transmitted and received linear frequency modulated signals.

[0017] Figure 1C shows a graph illustrating the measurement of heart rate using displacement sensed by FMCW radar as a function of time.

[0018] Figure 2 shows a schematic diagram of a millimeter-wave FMCW radar sensor and one or more IMU sensors located inside a vehicle, according to some implementations.

[0019] Figure 3 shows a schematic diagram of one or more millimeter-wave FMCW radar sensors and IMU sensors located in a room according to some implementations.

[0020] Figure 4A shows a graph illustrating an exemplary cardiovascular measurement obtained from a millimeter-wave FMCW radar sensor.

[0021] Figure 4B shows an exemplary projected heart rate map capturing the heartbeat pattern obtained from the IMU sensor.

[0022] Figure 4C shows a graph illustrating the motion activity of a mobile vehicle obtained from an IMU sensor.

[0023] Figure 4D shows a graph illustrating the optimized cardiovascular measurements obtained by combining data from multiple sensors according to some implementation methods.

[0024] Figure 5 shows a flowchart illustrating an exemplary process of using a multi-sensor system to calculate combined cardiovascular or respiratory measurements according to some implementations.

Claims

1. A multi-sensor system, comprising: Multiple millimeter-wave frequency-modulated continuous wave (FMCW) radar sensors are used to acquire multiple cardiovascular or respiratory measurements from multiple subjects inside a vehicle, each of which is calculated to determine the subject's stress level, drowsiness level, or health; a first inertial measurement unit (IMU) sensor located inside the vehicle, wherein the first IMU sensor is configured to acquire a second cardiovascular or respiratory measurement of at least one of the multiple subjects; and a second IMU sensor located inside the vehicle, wherein the second IMU sensor is configured to acquire motion data associated with the vehicle. One or more proximity sensors are configured to identify the position of the chest of at least one of the plurality of subjects within the vehicle; And a control system communicatively connected to the plurality of millimeter-wave FMCW radar sensors, wherein the control system is configured to perform actions in the vehicle in response to the stress level, drowsiness level, or health of at least one of the plurality of subjects.

2. The multi-sensor system according to claim 1, wherein the millimeter-wave FMCW radar sensor is configured to send a signal to the location on the chest of the subject.

3. The multi-sensor system according to claim 1, wherein the first IMU sensor is located in the seat, on the outer surface of the seat, or below the seat of the vehicle.

4. The multi-sensor system according to claim 1, wherein the second IMU sensor is located on the steering wheel or dashboard of the vehicle.

5. The multi-sensor system according to claim 1, wherein the millimeter-wave FMCW radar sensor is positioned at the same or substantially the same location as the second IMU sensor.

6. The multi-sensor system according to claim 5, wherein the one or more proximity sensors comprise a capacitive proximity sensor array.

7. The multi-sensor system according to claim 6, wherein the capacitive proximity sensor array is configured to map at least a portion of the contour of a subject's body to identify the location of the subject's chest.

8. The multi-sensor system according to claim 1, further comprising: The control system is communicatively connected to the first IMU sensor, the second IMU sensor, and the one or more proximity sensors, wherein the control system is configured to: use the one or more proximity sensors to determine the position of the subject's chest relative to the millimeter-wave FMCW radar sensor; and adjust the viewing angle of the millimeter-wave FMCW radar sensor to point to the position of the subject's chest.

9. The multi-sensor system according to claim 8, wherein the control system is further configured to: combine the first cardiovascular or respiratory measurement, the second cardiovascular or respiratory measurement, and the motion data associated with the vehicle to obtain optimal cardiovascular or respiratory measurement.

10. The multi-sensor system of claim 9, wherein the first cardiovascular or respiratory measurement has a first associated confidence factor and the second cardiovascular or respiratory measurement has a second associated confidence factor, wherein the first cardiovascular or respiratory measurement and the second cardiovascular or respiratory measurement are combined according to an algorithm partially based on the first associated confidence factor and the second associated confidence factor, and the motion data associated with the vehicle is subtracted from the first cardiovascular or respiratory measurement or the second cardiovascular or respiratory measurement to obtain the optimal cardiovascular or respiratory measurement.

Images