System and method for remote measurement of a person's vital signs in a variable environment - Patents.com

Abstract

A remote photoplethysmography (RPPG) system for estimating a vital sign of a person is provided. The RPPG system is configured to receive a set of imaging photoplethysmography (iPPG) signals measured from various regions of the skin of the person. The RPPG system is further configured to determine frequency coefficients in frequency bins of a quantized frequency spectrum of the measured iPPG signals by minimizing a distance between the measured iPPG signals and corresponding iPPG signals reconstructed from the determined frequency coefficients, the determining of the frequency coefficients being performed while applying joint sparsity of the determined frequency coefficients under a sparsity level constraint such that the determined frequency coefficients of the various iPPG signals have non-zero values ​​in the same frequency bins, and the RPPG system is further configured to output one or a combination of the determined frequency coefficients, the iPPG signals reconstructed from the determined frequency coefficients, and a vital sign signal corresponding to the reconstructed iPPG signals.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation-in-part of U.S. patent application US20190350471A1, filed October 23, 2018, which claims the benefit of priority to Provisional Application No. 62 / 672,433, filed May 16, 2018, the contents of which are incorporated herein by reference in their entireties.

[0002] The present disclosure relates generally to remote monitoring of a person's vital signs, and more particularly to remote photoplethysmography (RPPG) measurement of a person's vital signs in a variable environment. [Background technology]

[0003] A person's vital signs, such as heart rate (HR), heart rate variability (HRV), respiration rate (RR), or blood oxygen saturation, serve as indicators of a person's current condition and potential predictors of serious medical events. For this reason, vital signs are widely monitored in inpatient and outpatient care settings, at home, and in other health, leisure, and fitness settings. One method of measuring vital signs is plethysmography. Plethysmography generally refers to the measurement of volume changes in an organ or body part, specifically the detection of volume changes resulting from the passage of cardiovascular pulse waves through a person's body with each heartbeat.

[0004] Photoplethysmography (PPG) is an optical measurement technique that assesses time-varying changes in light reflectance or transmittance over an area or volume of interest and can be used to detect blood volume changes in tissue microvascular beds. PPG is based on the principle that blood absorbs and reflects light differently from surrounding tissue, and therefore, fluctuations in blood volume with each heartbeat correspondingly affect light transmittance or reflectance. PPG is often used noninvasively to measure at the skin surface. The PPG waveform contains a pulsatile physiological waveform attributed to cardiac-synchronous changes in blood volume with each heartbeat, superimposed on a slowly varying baseline with various low-frequency components attributed to other factors such as respiration, sympathetic nervous system activity, and thermoregulation. Although the origins of the components of the PPG signal are not fully understood, it is generally accepted that they can provide valuable information about the cardiovascular system.

[0005] Conventional pulse oximeters for measuring a person's heart rate and (arterial) blood oxygen saturation are attached to the person's skin, such as the fingertip, earlobe, or forehead. Therefore, they are referred to as "contact" PPG devices. A typical pulse oximeter may include a combination of green, blue, red, and infrared LEDs as a light source and a photodiode for detecting light transmitted through the patient's tissue. Commercially available pulse oximeters measure the transmittance of the same area or volume of tissue at different wavelengths by rapidly switching between measurements at different wavelengths. This is called time-division multiplexing. The transmittance over time at each wavelength produces PPG signals for the different wavelengths. Although contact PPG is considered a primarily noninvasive technique, contact PPG measurements are often experienced as uncomfortable because the pulse oximeter is attached directly to the person, restricting freedom of movement through cables. Summary of the Invention [Problem to be solved by the invention]

[0006] Recently, non-contact remote photoplethysmography (RPPG) has been introduced for unobtrusive measurements. RPPG utilizes a light source, or generally a radiation source, located away from the subject. Similarly, a detector, such as a camera or photodetector, can also be located away from the subject. RPPG is often also referred to as imaging PPG (iPPG) due to the use of an imaging sensor such as a camera. We use the terms RPPG and iPPG interchangeably. Because they do not require direct contact with the person, remote photoplethysmography systems and devices are considered unobtrusive and, as such, are suitable for medical and non-medical routine applications.

[0007] One advantage of camera-based vital signs monitoring over on-body sensors is ease of use: simply pointing the camera at the person is sufficient; sensors do not need to be attached to the person. Another advantage of camera-based vital signs monitoring over on-body sensors is that cameras have higher spatial resolution than contact sensors, which in most cases include single-element detectors.

[0008] One of the challenges of RPPG technology is being able to provide accurate measurements in variable environments with inherent noise sources. For example, in a variable environment such as an in-vehicle environment, the lighting to the driver can change dramatically and suddenly while driving (e.g., while passing through the shadows of buildings, trees, etc.), making it difficult to distinguish the iPPG signal from other intensity fluctuations. Also, there is significant movement of the driver's head and face due to several factors, such as vehicle motion and the driver's looking around both inside and outside the car (e.g., looking into the rearview and side mirrors for oncoming traffic).

[0009] Several methods have been developed to enable robust camera-based vital sign measurements. For such measurements, multiple signals are typically captured based on image processing of the captured image sequence. For example, the multiple signals may originate from different color channels of the video sequence. In this case, a photoplethysmography signal is derived from these multiple signals. These photoplethysmography signals indicate a person's vital signs, which can be determined by further analysis of the signals. However, the quality of the photoplethysmography signal degrades to an extent determined by the signal-to-noise ratio (SNR) of the detected measurement. Low SNR due to illumination fluctuations and false peaks in the photoplethysmography signal due to movement can disrupt the PPG signal.

[0010] Therefore, a RPPG system that is robust to noise such as illumination variations and human motion is needed. [Means for solving the problem]

[0011] An object of some embodiments is to estimate a person's vital signs using remote photoplethysmography (RPPG). An object of some embodiments is to design an algorithm to enable robustness against motion noise. Furthermore, an object of some embodiments is to design a narrowband near-infrared (NIR) system to determine a wavelength range that reduces illumination variations. Additionally or alternatively, some embodiments are directed to denoising a noisy imaging photoplethysmography (iPPG) signal in a set of iPPG signals measured from different regions of a person's skin by projecting the noisy iPPG signal onto the orthogonal complement of the noise subspace.

[0012] Some embodiments are based on the recognition that the sensitivity of an RPPG signal to noise in measuring the intensity of a person's skin (e.g., pixel intensities in a camera image) is caused, at least in part, by independently estimating a photoplethysmographic signal from the intensities of a person's skin measured at different spatial locations. Some embodiments are based on the recognition that measured intensities at different locations, such as different regions of a person's skin, may be subject to different measurement noises. If the photoplethysmographic signal is estimated independently from the intensity at each location (e.g., a photoplethysmographic signal estimated from the intensity at one skin region is estimated independently from the intensity or estimated signal from other skin regions), the independence of the respective estimates may prevent the estimator from distinguishing between such noises.

[0013] Some embodiments are based on the recognition that the measured intensities at different regions of a person's skin may be subject to different, sometimes even unrelated, noises. Such noises include one or more of illumination variations, person movement, etc. In contrast, heartbeat is a common source of intensity variations present at different regions of the skin. Therefore, when independent estimates are replaced with joint estimates of photoplethysmographic signals measured from intensities at different regions of a person's skin, the impact of noise on the quality of vital sign estimates can be reduced. In this way, embodiments can extract photoplethysmographic signals that are common to many skin regions (including regions that may also contain significant noise), while ignoring noise signals that are not shared across many skin regions.

[0014] Some embodiments are based on the recognition that it would be beneficial to estimate the photoplethysmographic signals of different skin regions together, i.e., using a common metric. Some embodiments are based on the recognition that two types of noise act on skin intensity: external noise and internal noise. External noise affects skin intensity due to external factors such as lighting fluctuations, person movement, and the resolution of the sensor measuring the intensity. Internal noise affects skin intensity due to internal factors such as the different effects of cardiovascular blood flow on the appearance of different areas of a person's skin. For example, heartbeat may affect the intensity of a person's forehead and cheeks more than the intensity of the nose.

[0015] Some embodiments are based on the recognition that both types of noise can be addressed in the frequency domain of intensity measurements. Specifically, external noise is often aperiodic or has a periodic frequency that is different from the signal of interest (e.g., a pulsatile signal) and can therefore be detected in the frequency domain. On the other hand, internal noise causes intensity fluctuations or time shifts of intensity fluctuations in different areas of the skin, while preserving the periodicity of common causes of intensity fluctuations in the frequency domain.

[0016] To that end, some embodiments are based on the recognition that a common metric used to estimate photoplethysmographic signals of various skin regions should be applied in the frequency domain of intensity measurements, rather than in the time domain in which the intensity measurements were collected. Furthermore, the joint sparsity of the frequency coefficients forces various photoplethysmographic signals to be sparse overall within the same frequency bin and / or to have significant energy only within the same frequency bin of the quantized frequency spectrum of the measured iPPG signal. A frequency bin is a segment of the frequency axis that collects amplitude, magnitude, or energy from a small range of frequencies. Thus, the joint sparsity adequately reflects the concept of common causes of intensity fluctuations used by some embodiments.

[0017] Some embodiments are based on the recognition that some vital signs, such as heartbeat signals, are locally periodic and present in all regions, so this common metric should be applied in the frequency domain. However, intensity measurements can be affected by noise, which is also periodic. Therefore, if the frequency coefficients of the photoplethysmography signal were derived directly from intensity measurements at each location (e.g., each region of the skin), such direct estimation would not be well suited to applying a common metric in the frequency domain.

[0018] However, some embodiments are based on the alternative realization that rather than directly calculating frequency coefficients from the measured intensities, a direct estimation of the photoplethysmography signal, where the signal is derived directly from the intensity measurements, can be substituted for the optimization problem in order to reconstruct the frequency coefficients of the photoplethysmography signal to match the measured intensities. This backward direction in the estimation of the frequency coefficients allows the reconstruction to be performed under the constraint that a common metric, namely joint sparsity, can be applied to the frequency coefficients of different photoplethysmography signals in different regions.

[0019] To this end, some embodiments determine frequency coefficients of the photoplethysmography signal for intensity signals of various regions of a person's skin in a manner that minimizes the difference between the determined frequency coefficients and corresponding intensity signals estimated using the measured intensity signals while applying joint sparsity to the determined frequency coefficients. For example, some embodiments estimate the intensity signals using an inverse Fourier transform of the determined frequency coefficients. Such inverse reconstruction allows for reduced sensitivity of the RPPG estimation to measurement noise.

[0020] Some embodiments apply joint sparsity as a soft constraint to the optimization problem, such that the joint sparsity forces the estimated frequency coefficients to have non-zero values ​​only in a small number of frequency bins, so that non-zero frequency bins (i.e., frequency bins with non-zero values ​​for frequency coefficients) are the same frequency bins across all face regions. To this end, applying joint sparsity forces frequency bins to have frequency coefficients with non-zero or zero values. The non-zero values ​​for these frequency coefficients may be different. However, applying joint sparsity does not indicate how many frequency bins are allowed to have non-zero or zero-value frequency coefficients.

[0021] Some embodiments are based on the recognition that such information can be determined by determining a sparsity level constraint. This sparsity level constraint indicates the number of frequency bins with non-zero values ​​of the frequency coefficients. Specifically, the sparsity level constraint indicates that the sum of the frequency bin norms (or the row norms of the frequency matrix) should be bounded by the sparsity level constraint. The sparsity level constraint is adaptively determined based on a function of the strength of the measured iPPG signal as a limit on the minimum energy to which the joint sparse signal embedded in the measured iPPG signal applies. In one embodiment, the limit on the minimum energy of the joint sparse signal is determined iteratively by minimizing the energy deviation based on the gradient of the distance of the reconstructed iPPG signal to the measured iPPG signal. This gradient is calculated with respect to one of the frequency coefficients or the measurement noise. The sparsity level constraint imposes an upper bound on the energy level of the determined frequency coefficients of the reconstructed iPPG signal. The sparsity level constraint is applied by a regularization parameter that is iteratively determined in response to updated estimates of the reconstructed iPPG signal to ensure that the energy of the frequency coefficients of the reconstructed iPPG signal is equal to the sparsity level constraint.

[0022] To that end, some embodiments determine frequency coefficients in frequency bins of the quantized frequency spectrum of the measured iPPG signal by minimizing the distance between the measured iPPG signal and a corresponding iPPG signal reconstructed from the determined frequency coefficients, and determining the frequency coefficients is performed while applying joint sparsity of the determined frequency coefficients under a sparsity level constraint so that the determined frequency coefficients of various iPPG signals have non-zero values ​​in the same frequency bins.

[0023] Some embodiments determine frequency bin weights that indicate which frequency bins have frequency coefficients with non-zero values. In some embodiments, the frequency bin weights are determined based on a function of the phase difference between the measured iPPG signals. Some embodiments use these frequency bin weights to apply joint sparsity to the determined frequency coefficients under a sparsity level constraint. For example, in some implementations, applying such joint sparsity encourages the numerical values ​​of non-zero frequency coefficients indicated by the sparsity level constraint to be in locations indicated by frequency bins with smaller weights.

[0024] Some embodiments recognize that spatial averaging of pixel groups can reduce camera quantization noise. To do so, a set of iPPG signals measured from a video of a person is obtained by averaging pixel intensities across all pixels in each of a set of skin regions of the person at each time step (e.g., each video frame). In some embodiments, the skin regions are facial regions focused around the forehead, cheek, and chin areas of the person's face. The skin regions are also referred to as "average regions" because the iPPG signal obtained from each region is calculated as the average value of the intensities of the pixels within that region. To obtain the locations of the facial regions, some embodiments first detect multiple facial landmarks using a face registration (i.e., facial landmark detection) method, and then interpolate and extrapolate the detected landmarks to multiple interpolation locations that are used to subdivide the face into more regions.

[0025] For robustness against slight variations in the location of face regions over time, some embodiments use spatial medians to group the mean region into multiple larger regions to generate a clustering of iPPG signals. These larger regions are referred to as "median regions." The measured iPPG signal for each median region is obtained by calculating the median across the iPPG signal from the mean region that makes up the median region for each time step.

[0026] Various facial regions may be differently degraded by noise caused by ambient lighting changes, motion registration errors, and facial expressions, resulting in high-dimensional noise. Therefore, iPPG signals from such noisy regions are also noisy. However, blood flows through facial regions with roughly the same temporal profile during the cardiac cycle. As a result, the underlying iPPG signals present in the measured intensity fluctuations from all median regions correspond to a low-rank matrix when grouped into a matrix.

[0027] Some embodiments recognize that an orthogonal projection (OP) of a noisy iPPG signal can be used to suppress noise that is corrupting the iPPG signal. In other words, a noisy iPPG signal can be denoised by using an orthogonal projection (OP) of the noisy iPPG signal. To do so, some embodiments orthogonally project the noisy iPPG signal onto a noise subspace and subtract the projection from the noisy iPPG signal. This is equivalent to projecting the noisy iPPG signal onto the orthogonal complement of the noise subspace. In some embodiments, the noise subspace includes one or more of a vertical motion signal capturing vertical motion in the region generating the iPPG signal, a horizontal motion signal capturing horizontal motion in the region generating the iPPG signal, and a background illumination signal capturing light fluctuations in background regions outside the region generating the iPPG signal.

[0028] Some embodiments aim to accurately estimate vital signs even in variable environments where dramatic lighting variations exist. For example, in a variable environment such as a vehicle interior, some embodiments provide an RPPG system suitable for estimating the vital signs of a vehicle driver or passenger. However, while driving, lighting on a person's face can change dramatically. To address these challenges, in addition to or instead of the sparse reconstruction using joint sparsity described above, one embodiment uses active interior lighting in a narrow spectral band where the spectral energy of sunlight, street lights, and headlights and taillights is all minimal. For example, due to moisture in the atmosphere, sunlight reaching the Earth's surface has much less energy around the near-infrared (NIR) wavelength of 940 nm than other wavelengths. The light output by streetlights and vehicle lights is generally within the visible spectrum, with very little power in infrared frequencies. To this end, one embodiment uses an active narrowband illumination source at or near 940 nm and a camera filter at the same frequency to ensure that most lighting changes due to environmental ambient lighting are filtered out. Furthermore, because this narrow frequency band is beyond the visible range, humans do not perceive this light source and are therefore not distracted by its presence. Furthermore, as the bandwidth of the light source used for active illumination becomes narrower, the camera's bandpass filter can be narrower, further eliminating intensity variations due to ambient lighting.

[0029] Thus, one embodiment uses a narrow bandwidth near-infrared (NIR) light source to illuminate human skin in a narrow frequency band that includes near-infrared wavelengths of 940 nm, and an NIR camera with narrow band filters that overlap the wavelengths of the narrow band light source to measure the intensity of various areas of the skin in that narrow frequency band.

[0030] One embodiment discloses a remote photoplethysmography (RPPG) system for estimating a person's vital signs, the RPPG system comprising: at least one processor; and a memory having instructions stored thereon, which, when executed by the at least one processor, cause the RPPG system to: receive a set of imaging photoplethysmography (iPPG) signals measured from various regions of the person's skin; determine a sparsity level constraint indicating a number of frequency bins in a quantized frequency spectrum of the measured iPPG signals having non-zero values ​​for frequency coefficients; and determine frequency coefficients in the frequency bins of the quantized frequency spectrum of the measured iPPG signals by minimizing a distance between the measured iPPG signals and a corresponding iPPG signal reconstructed from the determined frequency coefficients, wherein determining the frequency coefficients is performed while applying joint sparsity of the determined frequency coefficients under the sparsity level constraint such that the determined frequency coefficients of various iPPG signals have non-zero values ​​in the same frequency bins. The instructions, when executed by the at least one processor, further cause the RPPG system to output one or a combination of the determined frequency coefficients, the iPPG signal reconstructed from the determined frequency coefficients, and a vital sign signal corresponding to the reconstructed iPPG signal.

[0031] Another embodiment discloses a remote photoplethysmography (RPPG) method for estimating a person's vital signs. The RPPG method includes receiving a set of imaging photoplethysmography (iPPG) signals measured from various regions of the person's skin, determining a sparsity level constraint indicating the number of frequency bins in a quantized frequency spectrum of the measured iPPG signals that have nonzero values ​​for the frequency coefficients, and determining frequency coefficients in the frequency bins of the quantized frequency spectrum of the measured iPPG signals by minimizing a distance between the measured iPPG signals and corresponding iPPG signals reconstructed from the determined frequency coefficients, wherein determining the frequency coefficients is performed while applying joint sparsity of the determined frequency coefficients under the sparsity level constraint such that the determined frequency coefficients of various iPPG signals have the nonzero values ​​in the same frequency bins. The RPPG method further includes outputting one or a combination of the determined frequency coefficients, the iPPG signals reconstructed from the determined frequency coefficients, and a vital sign signal corresponding to the reconstructed iPPG signals. [Brief explanation of the drawings]

[0032] [Figure 1A] FIG. 1 shows an overview illustrating some principles used by some embodiments to determine a person's vital signs using remote photoplethysmography (RPPG). [Figure 1B] FIG. 1 illustrates a summary of the joint sparsity of different photoplethysmography signals from different regions of a person's skin, according to some embodiments. [Figure 1C] FIG. 1 illustrates an overview of the principles used by some embodiments to apply joint sparsity in the frequency domain during joint estimation of photoplethysmography signals of different areas of a person's skin. [Figure 1D] FIG. 1 is a block diagram of an RPPG method according to one embodiment. [Figure 1E]FIG. 1 is a block diagram of an RPPG method utilizing sparsity level constraints, according to one embodiment. [Figure 2A] FIG. 1 illustrates an overview of the AutoSparsePPG algorithm for sparse spectral estimation, according to some embodiments. [Figure 2B] FIG. 1 illustrates an algorithm overview for the 2-1 norm regularization step of the AutoSparsePPG algorithm, according to some embodiments. [Figure 3A] FIG. 1 illustrates a schematic for determining frequency bin weights according to some embodiments. [Figure 3B] FIG. 1 illustrates an overview of an RPPG method that utilizes sparsity level constraints and weight estimators, according to some embodiments. [Figure 4] FIG. 1 illustrates the calculation of an imaging photoplethysmography (iPPG) signal from video intensity according to some embodiments. [Figure 5] FIG. 1 shows an overview of the power spectrum used to determine the signal-to-noise ratio (SNR) of an iPPG signal, which is used by some embodiments to assess the usefulness of various regions. [Figure 6A] FIG. 1 illustrates an overview of the extraction of motion noise and time-varying background illumination signals, according to some embodiments. [Figure 6B] FIG. 10 is a diagram illustrating an outline of orthogonal projection of a noisy iPPG signal for denoising the noisy iPPG signal, according to some embodiments. [Figure 7] FIG. 1 illustrates an overview of an AutoSparsePPG framework using a denoised iPPG signal, according to some embodiments. [Figure 8] FIG. 1 illustrates a plot of the spectrum of sunlight at the Earth's surface, as used by some embodiments. [Figure 9] FIG. 10 shows a plot for comparison of iPPG signal frequency spectrum in near infrared (IR) and iPPG signal frequency spectrum in RGB. [Figure 10] FIG. 1 is a block diagram of a remote photoplethysmography (RPPG) system according to some embodiments. [Figure 11] FIG. 1 illustrates an overview of a patient monitoring system using an RPPG system in a hospital scenario, according to some embodiments. [Figure 12] FIG. 1 illustrates an overview of a driver assistance system using an RPPG system, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0033] [Description of the embodiment] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, devices and methods are shown in block diagram form solely to avoid obscuring the present disclosure.

[0034] As used in this specification and claims, the words "for example," "for instance," and "such as," as well as the verbs "comprising," "having," "including," and other forms of these verbs, when used in conjunction with a list of one or more components or other items, should each be construed as open-ended, meaning that the list should not be considered to exclude other, additional components or items. The word "based on" means based at least in part on. Furthermore, it should be understood that the phraseology and terminology employed herein are for purposes of description and should not be regarded as limiting. Any headings used in this description are for convenience only and do not have any legal or restrictive effect.

[0035] FIG. 1A is a diagram illustrating an overview of some principles used by some embodiments to determine a person's vital signs using remote photoplethysmography (RPPG). FIG. 1B is a diagram illustrating joint sparsity of various photoplethysmography signals from various regions of a person's skin, according to some embodiments. FIGS. 1A and 1B are used in conjunction with each other. As blood flows through various skin regions of a person, the concentration of hemoglobin changes over time. As a result, the amount and color of light absorbed by these various skin regions changes. In other words, intensity variations exist due to blood flow. When an image of the skin region is captured, a camera can register intensity variations from the various regions of the person's skin. Such registered intensity variations are referred to as an imaging photoplethysmography (iPPG) signal or a set of iPPG signals. Alternatively, these iPPG signals may be referred to as photoplethysmography signals or RPPG signals.

[0036] Some embodiments are based on the recognition that the sensitivity of the RPPG signal to noise in measuring the intensities of a person's 101 skin (e.g., pixel intensities in a camera image) is caused, at least in part, by independently estimating 104 the photoplethysmography signal from intensities 102 and 103 of the person's skin measured at different spatial locations. Some embodiments are based on the recognition that measured intensities at different locations, such as different regions of a person's skin, may be subject to different measurement noises. If the photoplethysmography signal is estimated 104 independently at each location (e.g., if a photoplethysmography signal estimated from intensities at one skin region is estimated independently from intensities or estimated signals from other skin regions), the independence of each estimate may prevent the estimator from distinguishing such noise.

[0037] 1B is a diagram illustrating an overview of the joint sparsity of various photoplethysmography signals 108a, 108b, 108c from various regions of a person's skin, according to some embodiments. The frequency spectra of various photoplethysmography signals 108a, 108b, 108c are measured from various regions of the person's skin. For simplicity of explanation, only three photoplethysmography signals are considered herein.

[0038] Some embodiments recognize that photoplethysmography signals 108a, 108b, and 108c may be subject to various, sometimes even unrelated, noises. However, heartbeats are a common source of intensity fluctuations present in different regions of the skin. In other words, a joint sparse signal 108e is incorporated (108d) into each of the photoplethysmography signals 108a, 108b, and 108c. Thus, when independent estimates 104 of the signal from each skin region are replaced with a joint estimate 106 of the photoplethysmography signal measured from the intensity at different regions of the person's skin, the impact of noise on the quality of the vital sign estimate can be reduced. In this way, embodiments can extract photoplethysmography signals common to many skin regions (including regions that may also contain significant noise) while ignoring noise signals that are not shared across many skin regions.

[0039] Some embodiments are based on the recognition that it would be beneficial to estimate the photoplethysmographic signals of different skin regions together, i.e., using a joint sparse signal 108e (also referred to as a common metric). Some embodiments are based on the recognition that two types of noise act on skin intensity: external noise and internal noise. External noise affects skin intensity due to external factors such as lighting fluctuations, person movement, and the resolution of the sensor measuring the intensity. Internal noise affects skin intensity due to internal factors such as the different effects of cardiovascular blood flow on the appearance of different areas of a person's skin. For example, heartbeat may affect the intensity of a person's forehead and cheeks more than the intensity of their nose.

[0040] Some embodiments are based on the recognition that both types of noise can be addressed in the frequency domain of intensity measurements. Specifically, external noise is often aperiodic or has a periodic frequency that is different from the signal of interest (e.g., a pulsatile signal) and can therefore be detected in the frequency domain. On the other hand, internal noise causes intensity fluctuations or time shifts of intensity fluctuations in different areas of the skin, while preserving the periodicity of common causes of intensity fluctuations in the frequency domain.

[0041] To that end, some embodiments recognize that a common metric used to estimate photoplethysmographic signals of various skin regions should be applied in the frequency domain of intensity measurements, rather than in the time domain in which the intensity measurements were collected. Furthermore, the joint sparsity of frequency coefficients forces various photoplethysmographic signals to be sparse overall within the same frequency bin, e.g., bin 108f, and / or to have significant energy only within the same frequency bin of the quantized frequency spectrum of the measured iPPG signal. A frequency bin is a segment of the frequency axis that collects amplitude, magnitude, or energy from a small range of frequencies. Thus, the joint sparsity adequately reflects the concept of common causes of intensity fluctuations used by some embodiments.

[0042] 1C is a diagram illustrating an overview of the principles used by some embodiments to apply joint sparsity in the frequency domain for joint estimation of photoplethysmography signals from various regions of a person's skin. Some embodiments are based on the recognition that some vital signs, such as heartbeat signals, are locally periodic and present in all skin regions, so a common metric should be applied in the frequency domain. However, intensity measurements can be affected by noise, which is also periodic. Therefore, if the frequency coefficients of the photoplethysmography signal are derived directly from intensity measurements at each location (e.g., each region of the skin), such direct estimation is not well suited to applying a common metric in the frequency domain. However, some embodiments are based on the alternative recognition that, rather than directly calculating frequency coefficients from the measured intensity, the direct estimation of the photoplethysmography signal (109a), in which the signal is derived directly from intensity measurements, can be replaced with an optimization framework (109b) to reconstruct the frequency coefficients of the photoplethysmography waveform to match the measured intensity (109c). Such a backward direction in the estimation of the frequency coefficients allows the reconstruction to be performed under the constraint that a common metric, namely joint sparsity, can be applied to the frequency coefficients of different photoplethysmographic signals in different regions.

[0043] 1D is a block diagram of an RPPG estimation method according to one embodiment. Using an input interface 110, such as a video camera, a set of various skin regions 120 of a person are measured to generate a raw RPPG matrix 100. The video camera measures the intensity of light reflected from the skin as such light changes over time. While this illustration shows skin regions located on the face (facial regions), it is understood that various embodiments are not limited to using the face, and other embodiments use other areas of exposed skin, such as a person's neck or wrist. The raw RPPG matrix 100, which includes the measured intensities of the facial regions over time, is processed using a solver 150, which determines frequency coefficients corresponding to the person's vital signs through an iterative process.

[0044] In some implementations, this iterative process begins by setting estimated frequency coefficients 185 of all face regions to zero, and then generating estimated region intensities 175 by calculating the inverse Fourier transform 170 of the frequency coefficients 185. These estimated region intensities 175, which represent the system's estimate of the RPPG signal, are then subtracted from the raw RPPG matrix 100 to output a difference 191. The difference 191 between the raw RPPG matrix 100 and the estimated region intensities 175 is transformed using a Fourier transform 160 to generate temporary frequency coefficients 161. The temporary frequency coefficients 161 are added 192 to the estimated frequency coefficients 185 to generate updated frequency coefficients 162. The updated frequency coefficients 162 are modified to apply joint sparsity 180, and the resulting frequency coefficients are used as the new estimated frequency coefficients 185. The new estimated frequency coefficients 185, replacing the estimated frequency coefficients 185 of the previous iteration, are used in the next iteration of the solver process 150.

[0045] In some embodiments, applying joint sparsity 180 forces the estimated frequency coefficients 185 to have non-zero values ​​only in a small number of frequency bins, so that the non-zero frequency bins (i.e., frequency bins with non-zero values ​​for the frequency coefficients) are the same frequency bins across all facial regions. The iterative solving process is repeated until a convergence condition 186 is met, such as when the new estimated frequency coefficients 185 are essentially unchanged from the estimated frequency coefficients 185 of the previous iteration. After convergence 186, the estimated frequency coefficients 185 are output by the solver 150 and used to estimate the vital sign 140. For example, in one embodiment, the estimated vital sign 140 is the frequency 130 of the person's heartbeat over a period of time. To that end, applying joint sparsity forces the frequency coefficients 185 to have non-zero values ​​only in a small number of frequency bins, so that the non-zero frequency bins are the same frequency bins across all facial regions.

[0046] The application of joint sparsity forces frequency bins to have non-zero or zero-valued frequency coefficients. The non-zero values ​​of these frequency coefficients may be different. However, the application of joint sparsity does not indicate how many frequency bins are allowed to have non-zero or zero-valued frequency coefficients.

[0047] Some embodiments are based on the recognition that such information can be determined by determining a sparsity level constraint, which indicates the number of frequency bins having non-zero values ​​of frequency coefficients. Specifically, the sparsity level constraint indicates that the sum of frequency bin norms (or row norms of a frequency matrix) should be bounded by the sparsity level constraint. Additionally or alternatively, the sparsity level constraint indicates the number of frequency bins having zero values ​​of frequency coefficients.

[0048] FIG. 1E is a block diagram of an RPPG method utilizing a sparsity level constraint 181 according to one embodiment. The sparsity level constraint 181 is adaptively determined based on a function of the strength of the measured iPPG signal as a limit on the minimum energy to which the joint sparse signal embedded in the measured iPPG signal must conform. The determination of the sparsity level constraint is described in more detail below with reference to FIGS. 2A and 2B. The sparsity level constraint 181 imposes an upper bound on the energy level of the determined frequency coefficients of the reconstructed iPPG signal. Thus, the determined frequency coefficients 185 are subject to the sparsity level constraint 181.

[0049] To that end, solver 150 determines frequency coefficients in frequency bins of the quantized frequency spectrum of the measured iPPG signal by minimizing the distance between the measured iPPG signal and a corresponding iPPG signal reconstructed from the determined frequency coefficients, and determining the frequency coefficients is performed while applying (180) joint sparsity of the determined frequency coefficients under sparsity level constraint 181 such that the determined frequency coefficients of various iPPG signals have non-zero values ​​in the same frequency bins.

[0050] Limiting the energy level of the determined frequency coefficients of the joint sparse signal according to the sparsity level constraint ensures that only the joint sparse signal attributable to the heartbeat (e.g., pulsatile) signal is estimated by the determined frequency coefficients. If the energy level of the determined frequency coefficients exceeds the amount specified by the sparsity level constraint, periodic noise fluctuations present in the measured iPPG signal will also be included in the determined frequency coefficients, thereby degrading the estimated heart rate signal. Therefore, it is essential to limit the amount of energy of the determined frequency coefficients to the level specified by the sparsity level constraint. Similarly, the sparsity level constraint determines a sufficiently large energy level that the determined frequency coefficients of the joint sparse signal must meet to estimate the heart rate signal. If the energy level of the determined frequency coefficients is too small, the determined frequency coefficients will not be able to contain the entire signal corresponding to the heartbeat.

[0051] FIG. 2A illustrates an AutoSparsePPG algorithm for sparse spectrum estimation according to some embodiments. iPPG signals are quasi-periodic, meaning they have slowly varying frequencies. Over a short time window, the heartbeat signal is approximately periodic and consists of a dominant frequency along with its harmonics. As a result, the frequency spectrum of the heartbeat signal should be sparse. Therefore, some embodiments model the iPPG signal as sparse in the frequency domain. Furthermore, the same heartbeat signal will produce periodic behavior in the iPPG signal across the entire skin region. Therefore, skin regions containing iPPG signals should have the same sparse frequency spectrum and the same support for frequency coefficients corresponding to the underlying noise-free vital sign (heartbeat signal). According to some embodiments, the iPPG signal Z (which in some embodiments has already been denoised using orthogonal projection, as described in detail in the description of FIGS. 6A and 6B ) is modeled as the sum of two components: the desired iPPG signal Y, whose frequency matrix (or frequency spectrum) X has only a small fraction of frequency coefficients with non-zero values, and a noise matrix E. Thus, Z=Y+E=F -1 X+E, where F -1 is the inverse Fourier transform.

[0052] The first dimension of frequency matrix X corresponds to different regions of the person's skin, and the second dimension of frequency matrix X corresponds to the frequency bins of the frequency coefficients. The first dimension of noise matrix E corresponds to different regions of the person's skin, and the second dimension of noise matrix E corresponds to the time at which the measurements were collected. In some implementations, the first and second dimensions of frequency matrix X refer to the columns and rows of frequency matrix X, respectively, and the first and second dimensions of noise matrix E refer to the columns and rows of noise matrix E, respectively. In some other implementations, the first and second dimensions of frequency matrix X refer to the rows and columns of frequency matrix X, respectively, and the first and second dimensions of noise matrix E refer to the rows and columns of noise matrix E, respectively.

[0053] Because the frequency components in the frequency matrix X should be sparse and have the same support across the entire skin region, the columns (skin regions) of the frequency matrix X are jointly sparse, i.e., the entire rows (frequency bins) of the frequency matrix X are either entirely zero or non-zero. Furthermore, it is beneficial to ensure that the energy in the remaining skin regions is not large, because the iPPG signal is a weak signal and large amplitudes are likely to correspond to noise. Therefore, some embodiments define the following optimization problem to calculate the frequency matrix X and the noise matrix E from the denoised iPPG signal Z:

number

[0054]

number

[0055] Thus, 2-1 norm regularization is applied to opposing dimensions such that the 2-norm along the first dimension of the frequency matrix is ​​followed by the 1-norm along the second dimension of the frequency matrix, and the 2-norm along the second dimension of the noise matrix is ​​followed by the 1-norm along the first dimension of the noise matrix.

[0056] Some embodiments are based on the recognition that the sparsity level constraint can be applied through regularization parameters. The choice of regularization parameters λ and μ has a significant impact on the performance of heart rate (HR) estimation. According to one embodiment, changing any of the above regularization parameters can result in a difference in HR estimation accuracy of as much as 30%. Furthermore, the optimal parameter values ​​for different videos are very different.

[0057] Some embodiments are based on the recognition that the regularization parameter λ can be adaptively selected using the AutoSparsePPG algorithm shown in Figure 2A. According to some embodiments, to solve a sparse optimization problem with a least-squares constraint, (1) can be rewritten as follows:

number

number

[0058]

number

[0059] Optimization problem (2) can be solved by the AutoSparsePPG algorithm shown in Figure 2A. Thus, a bound on the minimum energy of the joint sparse signal is determined iteratively by minimizing the energy deviation based on the gradient of the distance of the reconstructed iPPG signal to the measured iPPG signal.

[0060] 2B illustrates an algorithm for the 2-1 norm regularization step of the AutoSparsePPG algorithm (i.e., step 7 of the AutoSparsePPG algorithm), according to some embodiments. The parameter λ is initialized to λ 0 , given by:

number

number

[0061] As a result, we modify λ using the following update rule to satisfy τ, the sparsity level constraint:

number

[0062] To combine the denoised iPPG signals from each skin region, some embodiments calculate the median across all skin regions of X at each frequency bin. If some skin regions are corrupted by noise, the median is more robust to outliers than the mean. The frequency component with the greatest power in the frequency spectrum is the heart rate output by the AutoSparsePPG algorithm for a given time window.

[0063] 3A illustrates an overview for determining frequency bin weights according to some embodiments. The frequency bin weights indicate which frequency bins have frequency coefficients with non-zero values. To determine the frequency bin weights, a pair of skin regions 302a and 302b are first considered. The weight estimator 300 is configured to obtain an iPPG signal 304 corresponding to the skin region 302a and an iPPG signal 306 corresponding to the skin region 302b. A fast Fourier transform (FFT) is then applied to the iPPG signal 304, resulting in a power spectrum 308 and a phase spectrum 310 of the iPPG signal 304. Similarly, an FFT is applied to the iPPG signal 306, resulting in a power spectrum 312 and a phase spectrum 314 of the iPPG signal 306.

[0064] A phase spectrum 310 of the iPPG signal 304 is compared with a phase spectrum 314 of the iPPG signal 306. Specifically, the weight estimator 300 is configured to calculate the difference between the phase spectrum 310 and the phase spectrum 314. The absolute value of this difference is then calculated at each frequency bin to obtain an absolute phase difference 316 between the two skin regions 302a and 302b. Similarly, the absolute phase difference across other pairs of skin regions is determined. In some embodiments, this absolute phase difference is calculated for each possible pair of regions, such that for N regions, there are N(N+1) / 2 pairs of regions. The weight estimator 300 then calculates a sum 318 of the absolute phase differences across the various pairs of regions to generate a signal 319. Some embodiments are based on the recognition that heartbeat signals have approximately the same phase across various skin regions (e.g., various regions of the face), while many noise signals (such as those due to changes in illumination as a result of movement) have different phases across various skin regions. Therefore, the frequency bin with the smallest sum of absolute phase differences is likely to contain the frequency of the heartbeat signal. To that end, some embodiments recognize that the frequency corresponding to the heart rate has the smallest sum of absolute phase differences. The signal 319 is normalized so that the weights have any value between 0 and 1, and the weights of the frequency bins are determined. In this manner, the weights of the frequency bins are determined based on a function of the phase difference between the measured iPPG signals. In these embodiments, the weights of each frequency bin may be different. In other embodiments, the weights of the frequency bins may be set to the same non-zero value (e.g., all weights are set equal to 1), which is equivalent to unweighted optimization.

[0065] 3B illustrates an overview of an RPPG method that utilizes a sparsity level constraint 320 and a weight estimator 300, according to one embodiment. Some embodiments incorporate weights into the AutoSparsePPG algorithm. As a result, instead of thresholding λ multiplied by 1, some embodiments threshold λ multiplied by the weight for each frequency bin. To that end, the optimization problem corresponds to weighted 2-1 norm regularization, which is applied to opposing dimensions of the frequency matrix X and the noise matrix E.

[0066] The weighted 2-1-norm regularization is applied to opposing dimensions, with the 2-norm along the first dimension of the frequency matrix X followed by the weighted 1-norm along the second dimension of the frequency matrix X, and the 2-norm along the second dimension of the noise matrix E followed by the weighted 1-norm along the first dimension of the noise matrix E. In one embodiment, the weights in the weighted 1-norm along the first dimension of the noise matrix are identical, and the weights in the weighted 1-norm along the first dimension of the noise matrix are identical. The weights in the weighted 1-norm along the second dimension of the frequency matrix are a function of the phase difference between the measured iPPG signals from different regions.

[0067] The weight estimator 300 outputs the frequency bin weights. The solver uses the frequency bin weights to apply joint sparsity 180 of the determined frequency coefficients under sparsity level constraint 320. Applying such joint sparsity encourages the numerical values ​​of non-zero frequency coefficients, as indicated by the sparsity level constraint, to lie in the location indicated by the frequency bin with the smallest weight.

[0068] 4 is a diagram illustrating the calculation of an iPPG signal from video intensity, according to some embodiments. Some embodiments use spatial averaging of pixel groups to reduce the camera quantization noise v nThis is based on the recognition that (t) can be reduced. To that end, the RPPG method obtains a set of iPPG signals measured from a video of a person by averaging pixel intensities across all pixels in each of a set of skin regions 402 (also referred to as N skin regions) of the person at each time step (e.g., each video frame). In some embodiments, the skin regions 402 are facial regions focused around the forehead, cheek, and chin areas of the person's face. In some embodiments, the RPPG method excludes regions along the facial contour and the eyes, nose, and mouth because these areas exhibit weak RPPG signals. The N skin regions are also referred to as "average regions" because the iPPG signal obtained from each region is calculated as the average of the intensities of the pixels within that region.

[0069] In one embodiment, a set of iPPG signals from a video frame is obtained by spatially averaging pixel intensities within each of the subject's N=48 facial regions represented by a set of skin regions 402. To obtain the N=48 facial regions, some embodiments first detect multiple (e.g., 68) facial landmarks 400 using a face registration (i.e., facial landmark detection) method, and then interpolate and extrapolate the detected landmarks to more (e.g., 145) interpolation locations that are used to subdivide the face into more regions.

[0070] For each face region j∈{1,...,N}, the iPPG signal p obtained from the mean pixel intensity j(t) is a one-dimensional time series signal, where t∈{1,...,T} is a video frame index within a time window of length T frames. The iPPG signals from N face regions are stacked into an iPPG matrix P of size T×N. These iPPG signals are then processed within overlapping time windows. Some embodiments are based on the understanding that it is beneficial to process the iPPG signals using a 10-second time window because such a time window is short enough to absorb heart rate fluctuations but long enough to be robust against noise fluctuations over time. Furthermore, the signal in each time window is normalized by subtracting the average intensity of each region's signal over time and then dividing by the corresponding average intensity. Furthermore, a bandpass filter is used to limit the signal to a cardiac frequency range that includes the physiological range of cardiac signals of interest, e.g., 42 to 240 beats per minute (bpm).

[0071] Some embodiments recognize that when facial landmarks are detected independently in each video frame, there is high-frequency jitter in the positions of detected landmarks, even when the face is stationary. This causes pixels in different small facial regions to correspond to different regions of the face for each video frame, which causes the average intensity to change over time, resulting in small errors that affect vital sign estimation. Some embodiments recognize that temporal averaging of facial landmark positions can mitigate this problem. In some embodiments, the position of each facial landmark at frame t is estimated by averaging the detected positions of the landmarks from frames t−5 to t+5.

[0072] For further robustness against slight variations in the location of face regions over time, some embodiments use spatial medians to group the mean region (e.g., N=48) into multiple larger regions to generate a clustering of iPPG signals. These larger regions are referred to as "median regions." The measured iPPG signal for each median region is obtained by calculating the median across the iPPG signal from the mean region constituting the median region for each time step. For example, the mean region N=48 indicated by 402 is grouped into five median regions 404a, 404b, 404c, 404d, and 404e. According to one embodiment, using five median regions improves performance by 9% compared to simply using 48 mean region regions. [Pre-processing by discarding noisy face regions]

[0073] Some facial regions may be severely corrupted by noise over time (e.g., due to occlusion or shadows) or may not contain a physiologically strong iPPG signal (e.g., due to facial hair). In such cases, it is not possible to recover an iPPG signal from such corrupted regions, and including these corrupted regions in vital sign estimation may corrupt the vital sign estimate. Therefore, it is beneficial to identify corrupted regions and remove them before any processing so that they do not affect the vital sign estimate.

[0074]

number

[0075] Furthermore, some facial regions are physiologically known to contain better iPPG signals. However, the "betterness" of these facial regions also depends on the specific imaging conditions, facial hair, or facial occlusion. Therefore, it is beneficial to identify which regions are most likely to contain noise and remove them before any processing to prevent them from affecting vital sign estimation.

[0076] 5 is a diagram illustrating an overview of the power spectrum curve used to determine the signal-to-noise ratio (SNR) of an iPPG signal, which is used by some embodiments to evaluate the usefulness of various regions. For example, some embodiments may use a power spectrum curve where the SNR is greater than or equal to a threshold θ SNR (For example, θ SNR =0.2), or when the maximum amplitude is less than the threshold θ amp For example, one embodiment does so by removing regions where θ amp is set to four times the average iPPG signal amplitude. Some embodiments determine the SNR 500 as the ratio of the area under the power spectrum curve in a region a 502 surrounding the largest peak in the frequency spectrum divided by the area under the curve for the remainder of the frequency spectrum within a frequency range b 504 that includes the physiological range of the heartbeat signal (e.g., 30-300 beats per minute (bpm)). [Time window fusion]

[0077] Because heartbeat signals change slowly over time, iPPG signals from multiple facial regions can be approximated to a stationary process within a short time window. Using information from previous time windows can improve the denoising of iPPG signals and remove many sudden changes caused by noise. The iPPG signals are processed using a sliding time window. For each time window, the iPPG signal to be processed is a weighted average of two sources: the already processed and denoised data from the previous time window and the noise-mixed data from the current time window that has not yet been processed. This weighted average is defined as follows:

number

[0078]

number

[0079] As part of the pre-processing within each time window, different numbers of face regions may be removed, resulting in different dimensions of the iPPG signal in successive time windows. Therefore, after processing each time window, the signal in the missing region is reconstructed by linear interpolation from neighboring regions using the weighted time window fusion described. Noise reduction using orthogonal projection

[0080] FIG. 6A illustrates an overview of the extraction of motion noise and time-varying background illumination signals according to some embodiments. Due to various sources of measurement noise, different facial regions may be noisy. For example, different facial regions may be differently degraded by noise caused by ambient illumination changes, motion registration errors, and facial expressions, resulting in high-dimensional noise. Therefore, iPPG signals from such noisy regions are also noisy. However, blood flows through facial regions with approximately the same temporal profile during the cardiac cycle. As a result, the underlying iPPG signals present in the measured intensity fluctuations from all median regions correspond to a low-rank matrix when grouped into a matrix.

[0081] Some embodiments are based on the recognition that an orthogonal projection (OP) of a noisy iPPG signal P can be used to suppress noise that is degrading the iPPG signal. In other words, a noisy iPPG signal P can be denoised by using an orthogonal projection (OP) of the noisy iPPG signal P. To do so, some embodiments orthogonally project the noisy iPPG signal P onto a noise subspace Q and subtract the projection from the noisy iPPG signal P. This is equivalent to projecting the noisy iPPG signal onto the orthogonal complement of the noise subspace. In some embodiments, the noise subspace Q includes one or more of a vertical motion signal V capturing vertical motion in the region generating the iPPG signal, a horizontal motion signal H capturing horizontal motion in the region generating the iPPG signal, and a background illumination signal B capturing light fluctuations in background regions outside the region generating the iPPG signal.

[0082] To obtain the vertical motion signal V and the horizontal motion signal H, some embodiments extract a motion vector 600 for each facial landmark. The motion vector 600 can be decomposed into two components: a horizontal motion component 604 and a vertical motion component 602. The motion noise caused by facial motion can be summarized in two time-varying five-dimensional (5D) signals: a 5D horizontal motion signal H (corresponding to the horizontal motion signal H) and a 5D vertical motion signal V (corresponding to the vertical motion signal V).

[0083] To extract the 5D horizontal motion signal H, some embodiments measure the horizontal motion of each of the (N=48) face regions 402 by spatially averaging the positions of the four corners of the region in each frame. The 48 dimensions are then reduced to a smaller number of dimensions, for example, five, with each dimension associated with a respective median region. The 48 dimensions are reduced to five dimensions by calculating the median of the motion signals across all average value regions belonging to the median region. The series of 5D signals across all time steps in the 10-second time window is a T×5 matrix H. Similarly, the 5D vertical motion signal V is calculated.

[0084] Furthermore, to approximate noise caused by time-varying illumination at different locations, a 5D time-varying background illumination signal B (corresponding to background illumination signal B) is calculated. To obtain a 5D time-varying background illumination signal B representing background ambient light intensity fluctuations, multiple background regions are selected in a face-free background. For example, five background regions 606a, 606b, 606c, 606d, and 606e are selected. Each background region is divided into small (e.g., 30×30 pixel) regions. The spatial average of the intensity values ​​of each small region is calculated. Furthermore, the median of the spatial average of the intensity values ​​of the small regions is calculated to obtain a single value for the background region. Similarly, the median is obtained for each background region 608 in each frame, resulting in a T×5 matrix B 610.

[0085] 6B is a diagram illustrating an orthogonal projection of a noisy iPPG signal P to denoise the noisy iPPG signal, according to some embodiments. Three noise components (a 5D horizontal motion signal H, a 5D vertical motion signal V, and a 5D time-varying background illumination signal B) are concatenated to generate a noise signal matrix Q=[H|V|B] 612 of dimension T×15. Some embodiments orthogonally project 614 the noisy iPPG signal P onto the noise subspace Q to generate an orthogonal projection of the noisy iPPG signal, and then subtract this orthogonal projection of the noisy iPPG signal from the noisy iPPG signal P to generate a denoised iPPG signal Z 616.

number

[0086] Some embodiments are based on the realization that the denoised iPPG signal Z can be used with 2-1 norm regularization to recover the sparse frequency spectrum (X) of the iPPG signal.

[0087] FIG. 7 illustrates an overview of an AutoSparsePPG framework using a denoised iPPG signal Z according to some embodiments. In block 700, an iPPG signal is calculated from each face region as described above with reference to FIG. 4. The iPPG signal is noisy due to measurement noise. To denoise the noisy iPPG signal, in block 702, noise B from motion noise (H, V) and light fluctuations in background regions is suppressed by orthogonally projecting the noisy iPPG signal P onto a noise subspace Q and then subtracting the orthogonal projection of the noisy iPPG signal from the noisy iPPG signal P (described in detail above with reference to FIGS. 6A and 6B). As a result, a denoised iPPG signal Z is generated. In block 704, the denoised iPPG signal Z can be used for sparse spectrum extraction. Specifically, the denoised iPPG signal Z is modeled as the sum of two components: the desired iPPG signal Y, whose frequency spectrum X has only a few non-zero frequency coefficients, and inlier noise E that was not removed by denoising 702. In one embodiment, the denoised iPPG signal Z is used as one of the inputs to the AutoSparsePPG algorithm (shown in FIG. 2A ) to recover the sparse frequency spectrum 706 of the iPPG signal and the frequency spectrum 708 of the noise.

[0088] Some embodiments aim to accurately estimate vital signs even in variable environments where dramatic lighting variations exist. For example, in a variable environment such as a vehicle interior, some embodiments provide an RPPG system suitable for estimating the vital signs of a vehicle driver or passenger. However, while driving, lighting on a person's face can change dramatically. For example, during the day, sunlight is filtered by trees, clouds, and buildings before reaching the driver's face. As a vehicle moves, direct lighting can frequently and dramatically change in both magnitude and spatial extent. At night, overhead streetlights and oncoming car headlights create large, spatially non-uniform lighting variations. Such lighting variations can be so dramatic and ubiquitous that multiple approaches to mitigating these lighting variations are impractical.

[0089] To address these challenges, in addition to or instead of the sparse reconstruction using joint sparsity described above, one embodiment uses active interior lighting in narrow spectral bands where sunlight, street lights, and headlight and taillight spectral energy are all minimal.

[0090] FIG. 8 illustrates a plot of the spectrum of sunlight at the Earth's surface used by some embodiments. For example, due to moisture in the atmosphere, sunlight reaching the Earth's surface has much less energy around the near-infrared (NIR) wavelength of 940 nm than other wavelengths. Light emitted by street lamps and vehicle lights (such as headlights) is generally within the visible spectrum, with very little power at infrared frequencies. To that end, one embodiment uses an active narrowband illumination source at or near 940 nm and a narrowband camera filter that overlaps with the wavelength of this narrowband illumination source, ensuring that most illumination variations due to environmental ambient lighting are filtered out. Furthermore, because this narrow frequency band at or near 940 nm is beyond the visible range, humans do not perceive this light source and are therefore not distracted by its presence. Furthermore, as the bandwidth of the light source used for active illumination becomes narrower, the camera's bandpass filter can become narrower, further eliminating variations due to ambient lighting. For example, some implementations use an LED light source and a camera bandpass filter with a 10 nm bandwidth.

[0091] Thus, one embodiment uses a narrow bandwidth near-infrared (NIR) light source to illuminate human skin in a narrow frequency band that includes the near-infrared frequency of 940 nm, and an NIR camera to measure the intensity of various areas of the skin in that narrow frequency band.

[0092] Some embodiments are based on the recognition that in a narrow frequency band that includes the near-infrared frequency of 940 nm, signals observed by an NIR camera are significantly weaker than signals observed by a color intensity camera, such as an RGB camera. However, experiments have demonstrated the effectiveness of the sparse reconstruction RPPG used by some embodiments in processing these weak intensity signals.

[0093] 9 shows a comparative plot of experimentally acquired iPPG signal frequency spectra in the NIR and the visible portion of the spectrum (RGB). The iPPG signal in the NIR 900 (labeled "IR rPPG signal" in the legend) is approximately 10 times weaker than the iPPG signal in the RGB 902 (labeled "RGB iPPG signal"). Accordingly, an embodiment of an rPPG system includes a near-infrared (NIR) light source for illuminating human skin in a first frequency band and a camera for measuring the intensity of each of various regions in a second frequency band overlapping the first frequency band, such that the measured intensity of a region of the skin is calculated from the intensities of pixels of an image of that region of the skin.

[0094] The first and second frequency bands include a near-infrared frequency of 940 nm. The system includes a filter for denoising the intensity measurements of each of the various regions using robust principal component analysis (RPCA). In one embodiment, the second frequency band, determined by a bandpass filter in the camera, has a passband width of less than 20 nm. For example, the bandpass filter has a narrow passband with a full width at half maximum (FWHM) of less than 20 nm. In other words, the overlap between the first and second frequency bands is less than 20 nm. When combined with sparse reconstruction, such a system can perform RPPG in a variable environment. In another embodiment, the bandpass filter has a wider passband, such as a passband with a FWHM of approximately 50 nm.

[0095] Some embodiments incorporate the recognition that optical filters, such as bandpass filters and longpass filters (i.e., filters that block the transmission of light whose wavelength is less than a cutoff frequency but allow the transmission of light whose wavelength is greater than (often equal to) a second cutoff frequency), can be highly sensitive to the angle of incidence of light passing through the filter. For example, an optical filter may be designed to transmit and block a predetermined frequency range when light is incident on the optical filter parallel to its axis of symmetry (approximately perpendicular to the surface of the optical filter), referred to as a 0° angle of incidence. As the angle of incidence changes from 0°, many optical filters exhibit a “blue shift,” in which the passband and / or cutoff frequency of the filter effectively shift to shorter wavelengths. To account for this blue shift phenomenon, some embodiments use the center frequency of the overlap between a first frequency band and a second frequency band to have a wavelength greater than 940 nm (e.g., they shift the center frequency of a bandpass optical filter or the cutoff frequency of a longpass optical filter to have a wavelength longer than 940 nm).

[0096] Furthermore, because light from different parts of the skin strikes the optical filter at different angles of incidence, the optical filter allows different transmissions of light from different parts of the skin. To compensate for this, some embodiments use a bandpass filter with a wider passband. For example, the bandpass optical filter has a passband wider than 20 nm, so that the overlap between the first frequency band and the second frequency band is greater than 20 nm.

[0097] The illumination intensity across the face can be uneven due to 3D variations in normals across the face surface, due to shadows cast on the face, and due to different parts of the face being at different distances from the NIR light source. To make the illumination more uniform across the face, some embodiments use multiple NIR light sources, such as two NIR light sources placed on each side of the face and approximately equal distances from the head. Horizontal and vertical diffusers are also placed at the NIR light sources to broaden the light beam reaching the face and minimize the illumination intensity difference between the center and periphery of the face.

[0098] Some embodiments aim to capture a well-exposed image of the skin area to measure a strong iPPG signal. However, the intensity of the illumination is inversely proportional to the square of the distance from the light source to the face. If a person is too close to the light source, the image will be saturated and will not contain an iPPG signal. If a person is far from the light source, the image will be dim and will have a weak iPPG signal. Some embodiments experimentally select the most advantageous positions of the light sources and their brightness settings to record well-exposed images at a range of possible distances between the person's skin area and the camera, while not capturing a saturated image.

[0099] Therefore, the RPPG system uses a narrow frequency band including near-infrared frequencies at 940 nm to reduce noise due to illumination variations, and the AutoSparsePPG framework (described above with reference to Figure 7) that is robust to motion noise.

[0100] 10 is a block diagram of a remote photoplethysmography (RPPG) system 1000 according to some embodiments. The system 1000 includes a processor 1020 configured to execute stored instructions and a memory 1040 storing instructions executable by the processor. The processor 1020 may be a single-core processor, a multi-core processor, a computing cluster, or any other configuration. The memory 1040 may include random access memory (RAM), read-only memory (ROM), flash memory, or other suitable memory system. The processor 1020 is connected to one or more input / output devices via a bus 1006.

[0101] The instructions stored in the memory 1040 implement an RPPG method for estimating a person's vital signs from a set of imaging photoplethysmography (iPPG) signals measured from various regions of the person's skin. The RPPG system 1000 may also include a storage device 1030 configured to store various modules, such as a solver 1031, a vital sign estimator 1032, a weight estimator 1033, and an AutoSparsePPG algorithm 1034. The above modules stored in the storage device 1030 are executed by the processor 1020 to perform vital sign estimation. The vital signs correspond to the person's pulse rate or the person's heart rate variability. The storage device 1030 may be implemented using a hard drive, an optical drive, a thumb drive, an array of drives, or any combination thereof.

[0102] According to several principles utilized by various embodiments, the solver 1031 determines frequency coefficients in frequency bins of the quantized frequency spectrum of the measured iPPG signal by solving an optimization problem to minimize the distance between the measured iPPG signal and a corresponding iPPG signal reconstructed from the determined frequency coefficients, where the determination of the frequency coefficients is performed while applying joint sparsity of the determined frequency coefficients. Such backward reconstruction of the frequency coefficients allows for the application of a common metric, i.e., joint sparsity, in the frequency domain. According to several other principles utilized by various embodiments, the solver 1031 determine frequency coefficients in frequency bins of the quantized frequency spectrum of the measured iPPG signal by minimizing a distance between the measured iPPG signal and a corresponding iPPG signal reconstructed from the determined frequency coefficients, and determining the frequency coefficients is performed while applying joint sparsity of the determined frequency coefficients under a sparsity level constraint such that the determined frequency coefficients of the various iPPG signals have non-zero values ​​in the same frequency bins.

[0103] The vital sign estimator 1032 is configured to estimate the person's vital sign from the determined frequency coefficients of the measured iPPG signals. Furthermore, the RPPG system 1000 includes a weight estimator 1033 configured to determine frequency bin weights indicating the locations of frequency bins having frequency coefficients with non-zero values ​​based on a function of the phase difference between the measured iPPG signals. The AutoSparsePPG algorithm 1034 is used for sparse spectral estimation.

[0104] The system 1000 includes an input interface 1050 for receiving a set of imaging photoplethysmography (iPPG) signals measured from various areas of the person's skin. For example, this input interface may be a network interface controller adapted to connect the iPPG system 1000 to a network 1090 via a bus 1006. Via the network 1090, the intensity measurements 1095 are downloaded and stored in a computer storage system 1030 for storage and / or further processing. Value and and stored as

[0105] Additionally or alternatively, in some implementations, the RPPG system 1000 may include an iPPG signal. No. In some implementations, a human machine interface (HMI) 1010 in the system 1000 connects the system to input devices 1011 such as a keyboard, mouse, trackball, touchpad, joystick, pointing stick, stylus, touchscreen, etc.

[0106] The RPPG system 1000 can be coupled to an output interface for rendering the person's vital signs via a bus 1006. For example, the RPPG system 1000 may include a display interface 1060 adapted to connect the system 1000 to a display device 1065. The display device 1065 may include a computer monitor, a camera, a television, a projector, or a mobile device, among others.

[0107] The RPPG system 1000 also includes an imaging interface 1070 adapted to connect the RPPG system 1000 to an imaging device 1075 and / or may be connected to the imaging interface 1070. The imaging device 1075 may include a video camera, a computer, a mobile device, a webcam, or any combination thereof.

[0108] In some embodiments, the RPPG system 1000 is connected via a bus 1006 to an application interface 1080 adapted to connect the RPPG system 1000 to an application device 1085 that can act based on the results of the remote photoplethysmography. For example, in one embodiment, the device 185 is a car navigation system that uses a person's vital signs to determine how to control, e.g., steer, a car. In another embodiment, the device 185 is a driver monitoring system that uses a driver's vital signs to determine when the driver is safe to drive, e.g., whether the driver is drowsy.

[0109] 11 is a diagram illustrating an overview of a patient monitoring system 1100 using the RPPG system 1000 in a hospital scenario, according to some embodiments. A patient 1104 is lying in a hospital bed. In such a hospital scenario, the vital signs of the patient 1104 need to be monitored remotely. The patient monitoring system 1100 uses the remote photoplethysmography measurement principle. Measuring vital signs using a camera is known as imaging photoplethysmography (iPPG). To that end, a camera 1102 is used to capture images, i.e., video sequences, of the patient 1104.

[0110] The camera 1102 may include a CCD or CMOS sensor for converting incident light and its intensity fluctuations into an electronic signal. 1104The method particularly non-invasively captures light reflected from a skin portion of the patient, whereby the skin portion particularly refers to the forehead, neck, wrist, part of the arm, or other part of the patient's skin. A light source, such as a near-infrared light source, may be used to illuminate the patient or an area of ​​interest that includes the patient's skin portion.

[0111] Based on the captured images, the RPPG system 1000 determines vital signs of the patient 1104. In particular, the RPPG system 1000 determines vital signs such as the heart rate, respiratory rate, or blood oxygenation of the patient 1104. Furthermore, the determined vital signs are typically displayed on an operator interface 1106 for displaying the determined vital signs. Such an operator interface 1106 may be a patient bedside monitor or may be a remote monitoring station in a dedicated room within a hospital or in a remote location in a telemedicine application.

[0112] 12 is a diagram illustrating an overview of a driver assistance system 1200 using the RPPG system 1000, according to some embodiments. An NIR light source and / or an NIR camera 1204 are disposed within a vehicle 1202. The NIR light source is disposed within the vehicle 1202 to illuminate the skin of a person operating the vehicle (driver), and the NIR camera 1204 is disposed within the vehicle to measure iPPG signals from various areas of the driver's skin. The RPPG system 1000 is integrated into the vehicle 1202. The RPPG system 1000 receives the measured iPPG signals and determines the driver's vital signs, such as pulse rate.

[0113] Furthermore, the processor of the RPPG system 1000 can generate one or more control action commands based on the estimated vital signs of the person driving the vehicle. The one or more control action commands include vehicle braking, steering control, generating an alert notification, initiating an emergency service request, or switching driving modes. The one or more control action commands are transmitted to a controller 1208 of the vehicle 1202. The controller 1208 can control the vehicle 1202 according to the one or more control action commands. For example, if the driver's determined pulse rate is very slow, the driver may be experiencing a heart attack. As a result, the RPPG system 1000 can generate control commands for vehicle deceleration and / or steering control (e.g., diverting the vehicle to the shoulder of a highway and stopping it) and / or initiating an emergency service request. The controller 1208 can control the vehicle accordingly.

[0114] The above description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the above description of exemplary embodiments provides those skilled in the art with an enabling description for implementing one or more exemplary embodiments. It is intended that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter, as set forth in the appended claims.

[0115] Specific details are set forth in the above description to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements of the disclosed subject matter may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments. Furthermore, like reference numbers and names in the various drawings indicate like elements.

[0116] Also, particular embodiments may be described as a process that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. While a flowchart may describe operations as a sequential process, many of these operations can be performed in parallel or simultaneously. The order of these operations may also be rearranged. A process may be terminated when its operations are completed, but may have additional steps not discussed or included in the diagram. Moreover, not all operations in any specifically described process are performed in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the end of the function may correspond to a return of the function to a calling function or a main function.

[0117] Furthermore, embodiments of the disclosed subject matter may be implemented at least in part manually or automatically. The manual or automatic implementation may be performed or at least assisted by the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored on a machine-readable medium. A processor may perform the necessary tasks.

[0118] The various methods or processes outlined herein may be coded as software executable on one or more processors utilizing any one of a variety of operating systems or platforms. Further, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine language code or intermediate code that runs on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0119] Embodiments of the present disclosure may be embodied as a method, an example of which is provided. The actions performed as part of this method may be ordered in any suitable manner. Thus, embodiments may be constructed in which actions are performed in an order different from that shown, which may include performing some actions simultaneously even though they are shown as sequential actions in the illustrative embodiments. Although the present disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adaptations and modifications may be made within the spirit and scope of the present disclosure. Therefore, it is the aspect of the appended claims to cover all such variations and modifications that fall within the true spirit and scope of the present disclosure.

Claims

1. 1. A remote photoplethysmography (RPGG) system for estimating a person's vital signs, comprising: a processor; a memory storing instructions; The processor executes the instructions to: configured to receive a set of imaging PhotoPlethysmoGraphy (iPPG) signals measured from different areas of the person's skin; If the measured iPPG signal is Z, the frequency matrix indicating the frequency spectrum of the vital sign signal in each region is X, the noise matrix indicating the time domain measurement noise in each region is E, and the inverse Fourier transform matrix is ​​F −1 , then the measured iPPG signal can be expressed as Z=F −1 X+E, Vital sign signals have sparseness at certain frequencies, The processor further comprises: Calculating a sparsity level determined from a 2-1 norm regularization of the frequency matrix and a 2-1 norm regularization of a transpose of the noise matrix; determining frequency coefficients, which are elements of the frequency matrix, by solving an optimization problem that minimizes the distance between the measured iPPG signal and an iPPG signal reconstructed from the frequency matrix and the noise matrix, while applying joint sparsity of the frequency coefficients, under the constraint that the sparsity level is less than a predetermined value; configured to output one or a combination of the determined frequency coefficients, the iPPG signal reconstructed from the frequency coefficients, and a vital sign signal corresponding to the reconstructed iPPG signal; The sparsity level is expressed as follows, where μ is a regularization parameter: [Equation 1] It is calculated by the predetermined value is determined iteratively by minimizing a distance of the reconstructed iPPG signal relative to the measured iPPG signal based on a gradient of the distance; the gradient is calculated for the frequency coefficients and the measurement noise; The processor may calculate the following formula: [Equation 2] Calculating the predetermined value based on Here, the initial values ​​of the frequency matrix and the noise matrix are: [Equation 3] and ∇ X and ∇ E are [Equation 4] RPG system, which is a gradient of.

2. the columns of the frequency matrix correspond to the different regions of the skin of the person; the rows of the frequency matrix correspond to the frequency bins of the frequency coefficients; columns of the noise matrix correspond to the different regions of the skin of the person; the rows of the noise matrix correspond to the time of the measurement; 2. The RPG system of claim 1, wherein the 2-1 norm regularization is performed on a matrix by operating a 2-norm along the columns followed by a 1-norm along the rows.

3. the columns of the frequency matrix correspond to the different regions of the skin of the person; the rows of the frequency matrix correspond to the frequency bins of the frequency coefficients; columns of the noise matrix correspond to the different regions of the skin of the person; the rows of the noise matrix correspond to the time of the measurement; The processor further comprises: configured to determine frequency bin weights indicating which frequency bins of the frequency coefficients have non-zero values ​​based on a function of the phase difference between the measured iPPG signals; 2. The RPG system of claim 1, wherein the 2-1 norm regularization is performed on a matrix by operating a 2-norm along the columns followed by a weighted 1-norm along the rows.

4. An RPG system as described in claim 3, wherein at the sparsity level, the weights of the weighted 1-norm in the noise matrix are identical.

5. An RPG system as described in claim 3 or claim 4, wherein at the sparsity level, the weights in the weighted 1-norm in the frequency matrix are identical.

6. An RPG system as described in claim 3, wherein at the sparsity level, the weights of the weighted 1-norm in the frequency matrix are a function of the phase difference between the measured iPPG signals from different regions.

7. The processor further comprises: receiving a set of iPPG signals measured from a set of skin areas of the person; configured to group the set of iPPG signals into median regions to generate a clustering of iPPG signals; The RPG system of claim 1 , wherein for each median region, the measured iPPG signal is the median across the iPPG signals measured from the skin regions forming the median region.

8. 8. The RPG system of claim 7, wherein the processor is further configured to remove the iPPG signal of a region within a time window from the set of iPPG signals if the energy of the measured iPPG signal of the region exceeds a threshold.

9. an iPPG signal in the set of iPPG signals is noisy due to measurement noise; 8. The RPG system of claim 7, wherein the processor is further configured to denoise the noisy iPPG signal in the set of iPPG signals by projecting the noisy iPPG signal onto an orthogonal complement of a noise subspace.

10. The processor further comprises: orthogonally projecting the noisy iPPG signal onto the noise subspace to generate an orthogonal projection of the noisy iPPG signal; 10. The RPG system of claim 9, configured to then subtract the orthogonal projection of the noisy iPPG signal from the noisy iPPG signal P to generate a denoised iPPG signal.

11. The noise subspace is a vertical motion signal capturing vertical motion of the region generating the iPPG signal; a horizontal motion signal capturing horizontal motion of the region generating the iPPG signal; and a background illumination signal B that captures light variations in a background area outside the area generating the iPPG signal.

12. The person corresponds to the driver of the vehicle; The RPG system of claim 1 , wherein the processor is further configured to generate one or more control action commands for a controller of the vehicle based on the driver's vital signs.

13. The RPG system of claim 12 , wherein the one or more control action commands include vehicle braking, steering control, generating an alert notification, initiating an emergency service request, or switching driving modes.

14. 1. A remote photoplethysmography (RPGG) method for estimating vital signs of a person, comprising: receiving a set of imaging photoplethysmography (iPPG) signals measured from different areas of the person's skin; If the measured iPPG signal is Z, the frequency matrix indicating the frequency spectrum of the vital sign signal in each region is X, the noise matrix indicating the time domain measurement noise in each region is E, and the inverse Fourier transform matrix is ​​F −1 , then the measured iPPG signal can be expressed as Z=F −1 X+E, Vital sign signals have sparseness at certain frequencies, The method further comprises: computing a sparsity level determined from a 2-1 norm regularization of the frequency matrix and a 2-1 norm regularization of the transpose of the noise matrix; determining frequency coefficients, which are elements of the frequency matrix, by solving an optimization problem that minimizes the distance between the measured iPPG signal and an iPPG signal reconstructed from the frequency matrix and the noise matrix, while applying joint sparsity of the frequency coefficients, under the constraint that the sparsity level is less than a predetermined value; outputting one or a combination of the determined frequency coefficients, the iPPG signal reconstructed from the frequency coefficients, and a vital sign signal corresponding to the reconstructed iPPG signal; The sparsity level is expressed as follows, where μ is a regularization parameter: [Equation 5] It is calculated by the predetermined value is determined iteratively by minimizing a distance of the reconstructed iPPG signal relative to the measured iPPG signal based on a gradient of the distance; the gradient is calculated for the frequency coefficients and the measurement noise; The predetermined value is calculated by the following formula: [Equation 6] is calculated based on Here, the initial values ​​of the frequency matrix and the noise matrix are: [Equation 7] and ∇ X and ∇ E are [Equation 8] The RPG method is the gradient of

Images