Cross-region deployment-based imaging photoplethysmography system and method for estimating vital signs

Abstract

A computer-implemented method for estimating a subject's vital signs includes receiving a sequence of imaging photoplethysmography (iPPG) signals measured from different areas of the subject's skin; performing cross-domain expansion optimization to iteratively minimize the difference between the received iPPG signals and the reconstructed iPPG signals; and then processing the reconstructed iPPG signals using an iPPG neural network trained with machine learning in the frequency domain to apply the learned structure to the frequency coefficients of the reconstructed iPPG signals. The method further includes determining the subject's vital sign signals from the frequency coefficients of the reconstructed iPPG signals and outputting the vital sign signals corresponding to the reconstructed iPPG signals once the termination condition of the cross-domain expansion optimization is reached.

Description

[Technical Field]

[0001] This disclosure relates, in general, to remote monitoring of a subject's vital signs, and more specifically, to an imaging photoplethysmography (iPPG) system and method for remote measurement of vital signs. [Background technology]

[0002] Human vital signs, such as heart rate (HR), heart rate variability (HRV), respiratory rate (RR), or blood oxygen saturation, serve as indicators of a person's current condition and potential predictors of serious medical events. For this reason, vital signs are widely monitored in hospital and outpatient treatment settings, at home, and in other health, leisure, and health-promoting settings. One method of measuring vital signs is plethysmography, which corresponds to the measurement of volume changes in human organs or body parts. There are various plethysmographic realities, such as photoplethysmography (PPG), which is an optical measurement technique that evaluates the time-varying change in the light reflectance or transmission of an area or volume of a target, and can be used to detect changes in blood volume in the microvascular bed of tissue. PPG is based on the principle that because blood absorbs and reflects light differently from surrounding tissue, correspondingly, fluctuations in blood volume with each heartbeat affect the transmission or reflectance of light.

[0003] Conventional non-invasive devices for measuring a person's vital signs often need to be attached to a person's skin, such as a fingertip, earlobe, or forehead. This can be uncomfortable for a person for several reasons. In addition, the sensor's entry window may be too large or too small for some patients, and therefore may not provide accurate readings. Furthermore, in light of the emergence of infectious diseases such as the novel coronavirus disease based on SARS-CoV-2, the use of non-contact, non-invasive techniques for measuring vital signs has become essential. In recent years, there has been growing interest in non-contact monitoring of vital signs using cameras, particularly for telemedicine, including the estimation of heart rate, respiratory rate, and blood pressure from images of a subject's face or some other body part. The main advantage of monitoring a person's vital signs using a camera rather than using conventional contact sensors is that it is easier to use. Cameras also, naturally, provide vital sign information over a larger spatial area compared to highly localized contact sensors. The granularity of the output data can also be fine-tuned based on the resolution and capabilities of the camera sensor.

[0004] In addition to health management, remote monitoring can be used in safety-critical applications such as driving or operating heavy machinery because it does not require the attachment of contact sensors to the operator's body, which may interfere with the user's normal movements. A camera that records facial images captures subtle changes in skin color corresponding to blood volume pulse waves. However, the captured images can also be impaired by noise due to several factors. For example, some vital signs, such as blood volume pulse wave signal components, represent only a small percentage of pixel intensity and can be easily masked by changes in lighting and movement. Therefore, in order to accurately measure vital signs, it is important to consider various types of noise as part of the vital sign estimation.

[0005] Several attempts in this direction have been made using blind source separation methods, model-based methods, and data-driven methods. However, these approaches have not been effective in recovering or extracting underlying pulse signals for several reasons. For example, model-based methods are not effective in recovering underlying pulse signals because the constraints designed manually are too simple and do not consider all the characteristics of the signal. On the other hand, purely data-driven deep learning-based methods are black-box methods and lack the good interpretability of the underlying approach because they offer little interpretability. Another challenge faced by conventional vital sign estimation approaches is that the region in which the measurement occurs may not be a suitable region from which the underlying pulse wave can be recovered.

[0006] Therefore, there is a need to develop a solution for remote estimation of vital signs that is effective, based on data-driven modeling of both pulse wave and structured noise, while simultaneously maintaining interpretability. Furthermore, there is a need to develop a solution that can operate simultaneously with measurements across multiple regions, modeling both the pulse wave component and the structured noise component in their respective appropriate regions, and accurately recovering vital signs. [Overview of the project] [Problems that the invention aims to solve]

[0007] The objective of some exemplary embodiments is to provide an effective solution for estimating a subject's vital signs by an approach based on remote photoplethysmography. Another objective of some exemplary embodiments is to provide such a solution that offers accurate measurement of vital signs while maintaining good interpretability of the underlying approach. Some exemplary embodiments are also aimed at providing a solution that takes into account all the characteristics of the measured signal when attempting to recover the underlying pulse signal from the measured signal. Furthermore, some exemplary embodiments are also aimed at recovering the underlying pulse signal in a region different from the measured signal region to isolate noise, thereby providing greater accuracy in vital sign estimation.

[0008] Some exemplary embodiments are based on the understanding that one way to model a pulsed signal is to model the pulsed signal as a sparse signal in the Fourier domain and estimate this sparse signal from a noisy time series obtained from various facial regions using variations of the Iterative Collapse Threshold Algorithm (ISTA). Some exemplary embodiments are based on the understanding that such a method, which models the signal as sparse in the frequency domain and the noise as sparse in the spatiotemporal domain, is based on two assumptions: (1) a true pulsed signal can be modeled using a sparse set of frequencies shared across facial regions, and (2) noise primarily affects a small number of regions. Thus, the understanding of some exemplary embodiments is that in such an approach, solving for the signal and noise components can be done via alternating gradient updates and soft thresholding projections.

[0009] Some exemplary embodiments recognize that remote estimation of signs such as pulse waves and heart rate from images, known as imaging photoplethysmography (iPPG) or remote photoplethysmography (rPPG), can be classified into blind source separation methods, model-based methods, and data-driven methods. Blind source separation assumes that the extracted time-series signal consists of both noise and an underlying pulse signal, and that these signals are statistically independent or uncorrelated. Techniques such as independent component analysis (ICA) and principal component analysis (PCA) separate the signal from the noise. Some exemplary embodiments also recognize that another approach to this task is to explicitly model light absorption and reflection in the skin, both of which aim to reduce the dependence of the extracted signal on the mean skin reflectance color. Some exemplary embodiments recognize that model-based methods model the underlying pulse waveform as a sum of a sparse set of periodic signals, and they aim to solve for the underlying pulse waveform. Some exemplary embodiments also recognize that another approach to the task of recovering an underlying pulse signal is based on a data-driven method, which learns directly from training data and is typically instantiated as a neural network, with the neural network's parameters learned by minimizing the loss between the network's output and the ground truth waveform.

[0010] Several exemplary embodiments are based on the alternative understanding that unfolding optimization algorithms leverage the power of learning while integrating learnable parameters into conventional iterative algorithms, utilizing known structures, and preserving interpretability. These algorithms repeatedly apply two steps: firstly, they ensure that the intended result is consistent with the measured values ​​by minimizing a data fidelity term using a learned or fixed forward operator; and secondly, they fit the solution to the ground truth signal after applying a signal denoising filter using the learned signal.

[0011] Based on several experiments, it is recognized in some exemplary embodiments that the second step of the signal denoising process is currently applied to data in the original region of the measurement. This is not surprising, as the purpose of the denoising process is to remove noise caused by imperfections in the measurement made in the original measurement region. However, some embodiments are based on the recognition that, in some applications, it is beneficial to apply the signal denoising process to a region different from the measurement region. For example, in some applications, the data in that different region may have a better or more pronounced structure suitable for machine learning, and / or ground truth data may be measured in that different region, and so on.

[0012] Such cross-domain optimization is unprecedented because conventionally, denoisingers should act on noise caused by noisy measurements. However, some embodiments are based on the understanding that when a denoisinger is applied in a region different from the original region of measurement, the denoisinger acts as a structure implementer and / or structure prior probability distribution. This understanding strengthens the selection of regions with significant structure. Additionally or alternatively, this understanding means that noise may be constructed separately in a similar manner to constructing the signal under consideration.

[0013] Some embodiments are based on the recognition that non-contact vital sign estimation can benefit from this cross-domain unfolding optimization. To that end, the objective of some embodiments is to adapt the unfolding optimization algorithm for imaging photoplethysmography (iPPG) to estimate subject vital signals such as pulse wave signals, heart rate, and / or other vital signs from images. In addition, or alternatively, the objective of some embodiments is to disclose an unfolded iPPG method that integrates iterative optimization updates with a deep learning-based signal prior probability distribution to estimate pulse waveforms and heart rate from facial images.

[0014] Some embodiments are based on the understanding that the two steps of the decompression algorithm are typically performed in the same domain. For example, in image processing applications, both steps of the decompression algorithm are performed in the feature domain of the image. Theoretically, this approach can be extended to time-series, noisy measurements of iPPG by performing signal denoising in the time domain. This would make sense because the noise in the noisy measurement comes from that domain.

[0015] However, after several experiments, some embodiments recognized the advantage of applying the second-step signal denoiser directly in the frequency domain to extract the target pulse signal. This is advantageous because the denoiser can be trained with ground truth data of the pulse signal measured in a contact manner. In addition, in the frequency domain, the pulse signal has a more consistent structure than in the time domain. In effect, the signal denoiser acts as a structural prior probability distribution. One result is that the noise can be selectively modeled separately using another noise estimator trained to determine the noise structure.

[0016] For these purposes, the objective of some exemplary embodiments is to provide systems, methods, and computer program products for effectively estimating the vital signs of a subject using a cross-domain unfolding optimization approach. The disclosed embodiments model the signals extracted from the subject's video as a sum of a basis pulse signal and noise. However, instead of explicitly imposing a manually designed prior probability distribution (e.g., sparsity in the frequency domain) on the signal, some exemplary embodiments use a neural network to learn the prior probability distributions on the signal and noise. Some of the disclosed embodiments solve for the basis pulse signal by unfolding a nearest-neighbor gradient descent, and the algorithm alternates between gradient descent steps and the application of a learned denoiser, which replaces the manually designed prior probability distributions and their nearest-neighbor operators. In other words, some embodiments combine a model-based approach with a data-driven deep neural network for vital sign estimation. Using this approach, some of the disclosed embodiments achieve accurate estimation of the vital signs of a subject using the challenging MMSE-HR dataset.

[0017] The models utilized in various exemplary embodiments take into account all variability in real-world data by utilizing trainable parameters in the form of deep denoisingers, which can better model signal and noise characteristics using training data, thereby leading to improved vital sign estimations. Some exemplary embodiments also provide better interpretability regarding what the model is doing to estimate vital signs by, for example, visualizing intermediate gradient descent steps and denoisinger outputs. Thus, the disclosed embodiments expand an iteration of the model-based method and provide a hybrid approach that replaces any manually designed model with a deep denoisinger that can be trained using training data.

[0018] To achieve the aforementioned objectives and advancements, several exemplary embodiments provide systems, methods, and computer program products for estimating vital sign signals of a subject. [Means for solving the problem]

[0019] Several exemplary embodiments provide a remote photoplethysmography (RPPG) system for estimating vital sign signals of a subject. The system comprises a memory for storing instructions and a processor that executes instructions to cause the RPPG system to receive a sequence of imaging photoplethysmography (iPPG) signals measured from different regions of the subject's skin in the time domain. The processor also performs cross-domain expansion optimization to iteratively minimize the difference between the received iPPG signals and the reconstructed iPPG signals, and then processes the reconstructed iPPG signals using an iPPG neural network. The reconstructed iPPG signals have frequency coefficients determined in the frequency domain and converted to the time domain of the received iPPG signals. The iPPG neural network is trained in the frequency domain using machine learning to implement learned structures on the frequency coefficients of the reconstructed iPPG signals. Furthermore, when the cross-domain optimization termination condition is reached, the processor determines the subject's vital sign signals from the frequency coefficients of the reconstructed iPPG signal and outputs the vital sign signals corresponding to the reconstructed iPPG signal via the output interface.

[0020] In yet another exemplary embodiment, a computer-implemented method for estimating the vital signs of a subject is provided. This method includes receiving a sequence of imaging photoplethysmography (iPPG) signals measured from different regions of the subject's skin in the time domain, performing cross-domain expansion optimization to iteratively minimize the difference between the received iPPG signals and reconstructed iPPG signals, and then processing the reconstructed iPPG signals using an iPPG neural network. The reconstructed iPPG signals have frequency coefficients determined in the frequency domain and converted to the time domain of the received iPPG signals. The iPPG neural network is trained in the frequency domain using machine learning to implement learned structures on the frequency coefficients of the reconstructed iPPG signals. The method further includes determining the subject's vital sign signals from the frequency coefficients of the reconstructed iPPG signals when the termination condition of the cross-domain expansion optimization is reached, and outputting the vital sign signals corresponding to the reconstructed iPPG signals via an output interface.

[0021] In some further exemplary embodiments, a non - transient computer - readable medium storing computer - executable instructions for performing a method for estimating a subject's vital signs is provided. The method includes receiving a sequence of imaging photoplethysmography (iPPG) signals measured from different regions of the subject's skin in the time domain, performing cross - region unfolding optimization that iteratively minimizes the difference between the received iPPG signals and the reconstructed iPPG signals, and then processing the reconstructed iPPG signals using an iPPG neural network. The reconstructed iPPG signals have frequency coefficients that are determined in the frequency domain and converted to the time domain of the received iPPG signals. The iPPG neural network is trained using machine learning in the frequency domain to implement a learned structure for the frequency coefficients of the reconstructed iPPG signals. The method further includes determining a vital sign signal of the subject from the frequency coefficients of the reconstructed iPPG signals when an end condition of the cross - region unfolding optimization is reached, and outputting the vital sign signal corresponding to the reconstructed iPPG signals via an output interface.

[0022] In some exemplary embodiments, the cross - region unfolding optimization estimates the noise of the received iPPG signals and uses the estimated noise to correct the reconstructed iPPG signals in order to minimize the difference between the received iPPG signals and the reconstructed iPPG signals. The noise may be processed by a noise neural network to implement an implicit structure in the noise and generate a structured component of the noise.

[0023] In some exemplary embodiments, the received iPPG signals are estimated in the frequency domain, and the noise neural network processes the noise in the frequency domain to constrain the received iPPG signals to be limited to a set of active frequency coefficients. The received iPPG signals may be converted to the time domain before being combined with the reconstructed iPPG signals in the time domain.

[0024] In some exemplary embodiments, the noise is estimated in the time domain, and the noise neural network may process the noise in the time domain to constrain the structure of the noise in a manner defined by the noise neural network.

[0025] Hereinafter, the embodiments disclosed herein will be further described with reference to the following drawings. The drawings shown are not necessarily to scale, and generally, emphasis is placed on explaining the principles of the embodiments of the present disclosure.

Brief Description of the Drawings

[0026] [Figure 1A] A block diagram of an imaging photoplethysmography (iPPG) system for estimating a subject's vital signs according to some exemplary embodiments is shown. [Figure 1B] A flowchart of a method performed by an iPPG system to estimate a subject's vital signs according to some exemplary embodiments is shown. [Figure 1C] A flowchart of a method for extracting multi-dimensional time series data from a subject's video according to some exemplary embodiments is shown. [Figure 2] Two comprehensive iterations of a cross-region expanded iPPG algorithm according to some exemplary embodiments are shown. [Figure 3] The architecture of a noise eliminator utilized by the iPPG system of FIG. 1A according to some exemplary embodiments is shown. [[ID=二十六]] [Figure 4] A tabular comparison of the performance of the expanded iPPG system of FIG. 1A and the performance of some known methods is shown. [Figure 5A] Exemplary iterations of an expanded iPPG algorithm for estimating a base pulse signal and spectrum for one test video in the MMSE-HR dataset according to some exemplary embodiments are shown. [Figure 5B]This paper presents exemplary iterations of the expanded iPPG algorithm for estimating the underlying pulse signal and spectrum for a single test video in the MMSE-HR dataset, using several exemplary embodiments. [Figure 5C] This paper presents exemplary iterations of the expanded iPPG algorithm for estimating the underlying pulse signal and spectrum for a single test video in the MMSE-HR dataset, using several exemplary embodiments. [Figure 5D] The heart rate estimation performance of the deployed iPPG system over a 10-second continuous window in test footage is shown by several exemplary embodiments. [Figure 6] This paper demonstrates the impact on heart rate estimation performance of explicitly modeling structured noise components using a separate noise filter, as illustrated by several exemplary embodiments. [Figure 7] The effect of the number of expansion iterations on heart rate estimation performance is shown in several exemplary embodiments. [Figure 8] The following are block diagrams of computing systems for realizing some components of an deployed iPPG system, based on several exemplary embodiments. [Figure 9] This figure shows a patient monitoring system using the iPPG system shown in Figure 1A, according to several exemplary embodiments. [Figure 10] This figure shows a vehicle assistance system using the iPPG system of Figure 1A, according to several exemplary embodiments. [Modes for carrying out the invention]

[0027] The drawings identified above illustrate embodiments disclosed herein, but other embodiments are contemplated, as discussed. This disclosure presents exemplary embodiments as representative, not limiting. Those skilled in the art can devise numerous other modifications and embodiments that fall within the scope and spirit of the principles of the embodiments of this disclosure.

[0028] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the following description of exemplary embodiments provides a practicable description for carrying out one or more exemplary embodiments. Various modifications that may be made in the function and arrangement of the elements are contemplated without departing from the spirit and scope of the subject matter disclosed as described in the appended claims.

[0029] The following description provides specific details for a complete understanding of the embodiments. However, those skilled in the art will understand that embodiments may be carried out without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in the form of block diagrams so as not to obscure the embodiments with unnecessary details. In other examples, well-known processes, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments. Furthermore, similar reference numbers and names in different drawings may refer to similar elements.

[0030] Furthermore, individual embodiments may be described as processes shown as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. While flowcharts may describe operations as sequential processes, many operations can be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process may terminate when its operations are complete, but it may have additional steps that are not discussed or included in the diagrams. Moreover, not all operations in any particular process described may occur in all embodiments. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the function may correspond to the function's return to the calling function or the main function.

[0031] Furthermore, embodiments of the disclosed subject matter may be implemented at least partially manually or automatically. Manual or automatic implementations may be performed using, or at least assisted by, a machine, hardware, software, firmware, middleware, microcode, a hardware description language, or any combination thereof. If implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the required tasks may be stored in a machine-readable medium. The required tasks may be performed by a processor.

[0032] In recent years, non-contact remote photoplethysmography (RPPG) devices have been introduced for unobtrusive measurements. RPPG utilizes a light source, or generally a radiation source, positioned away from the subject being measured. Similarly, detectors, such as cameras or photodetectors, can be positioned away from the person being measured. Therefore, remote photoplethysmography systems and devices are considered unobtrusive and are well-suited for routine medical and non-medical applications. One advantage of camera-based vital sign monitoring over on-body sensors is its ease of use; that is, there is no need to attach sensors, and the camera is simply pointed at the person. Another advantage of camera-based vital sign monitoring over on-body sensors is the potential to achieve motion robustness; that is, cameras have higher spatial resolution than contact sensors, which mostly involve single-element detectors. RPPG technology faces significant challenges in providing accurate measurements under motion / light distortion. In particular, pulse signals containing information from the subject's body introduce noise into the measurement for several reasons. Therefore, the vital sign component is only a small fraction of the pixel intensity and can be easily masked by changes in lighting and motion. Therefore, in order to accurately measure vital signs, it is important to consider various types of noise as part of the vital sign estimation.

[0033] The exemplary embodiments disclosed herein are effective and provide solutions for remote estimation of vital signs that are based on data-driven modeling of both pulse waves and structured noise, while simultaneously maintaining interpretability. Some exemplary embodiments also provide the ability to operate simultaneously on measurements across multiple regions to model both pulse wave components and structured noise components in their respective appropriate regions, thereby accurately recovering vital signs. Some exemplary embodiments are also directed toward the objective of recovering the underlying pulse signal and separating the noise in regions different from the measured signal region, thereby providing greater accuracy in vital sign estimation.

[0034] Figure 1A shows a block diagram of an imaging photoplethysmography (iPPG) system 100 for estimating vital signs of a subject, according to several exemplary embodiments. The subject may be human or animal. The iPPG system 100 comprises a time-series extraction module 101 and a PPG estimator module 109 for generating PPG waveforms (also called "PPG signals"), which are given images of different areas of the subject's skin as input. The generated PPG waveforms may be further processed to estimate several vital signs of the subject. Each of the time-series extraction module 101 and the PPG estimator module 109 may be implemented as a combination of hardware and software.

[0035] In some embodiments, the iPPG system 100 may optionally include a light source with visible wavelengths, near-infrared (NIR) wavelengths, or a broad spectrum including visible and NIR wavelengths. The light source may be configured to illuminate the subject's skin. The iPPG system 100 may also optionally include a camera configured to capture images 105 at each wavelength of at least one body part of the subject (such as the subject's face). In some exemplary embodiments, the imaging setup comprising the light source and camera may be outside the iPPG system 100. Whether or not the imaging setup is part of the iPPG system 100, the iPPG system 100 receives captured images 105 of the subject as input. The captured images 105 may correspond to the subject's face. The images 105 include multiple frames, and each frame contains an image of the subject's face. An example of an image of the subject's face captured in at least one frame of the images 105 is shown as image 107.

[0036] In some exemplary embodiments, the time-series extraction module 101 can segment an image 107 in each frame of video 105 into multiple spatial regions 103, which are then analyzed jointly to accurately determine the PPG waveform. The segmentation of each image 107 is based on the recognition that a particular area of ​​the body part under consideration contains the strongest PPG signal. For example, a particular area of ​​the face containing the strongest PPG signal (also called a “region of interest (ROI)”) is the area around the forehead, cheeks, and chin (as shown in Figure 1A). The segmentation of each image 107 results in a sequence of images containing different spatial regions from the multiple spatial regions 103, each spatial region containing data corresponding to at least one body part, such as the subject’s skin. Furthermore, each of the multiple spatial regions 103 may be used to determine the PPG signal.

[0037] The sensitivity of the PPG signal to noise in measuring the intensity of a subject's skin (e.g., pixel intensity in an image) is at least partially due to the independent estimation of the PPG signal from the intensity of the subject's skin measured at different spatial locations (or spatial regions). In different locations, e.g., different regions of the subject's skin, the measured intensity may be exposed to different measurement noises. If the PPG signal is estimated independently from the intensity in each spatial region (e.g., the PPG signal estimated from the intensity in one skin region is estimated independently from the intensity or estimated signal from other skin regions), the independence of the different estimates may prevent the estimator from identifying such noise that affects the accuracy in determining the PPG signal.

[0038] Noise includes one or more of the following: lighting variations, pixel blur due to human movement, etc. Furthermore, vital signs such as heart rate are a common source of intensity variations present in different areas of the skin. Therefore, replacing independent estimations with a combined estimation of PPG signals measured from intensity in different areas of the subject's skin can reduce the impact of noise on the quality of vital sign estimations.

[0039] For example, in the context of performing vital sign estimation using a subject's facial image, it may be assumed that certain facial regions are physiologically known to contain better PPG signals. However, the "goodness" of these facial regions also depends on several factors, such as the specific conditions under which the image was captured, the subject's beard, or facial occlusion. Therefore, it is beneficial to identify which regions are most likely to contain noise and remove them before any processing so that they do not affect the signal estimate. In this regard, several exemplary embodiments can incorporate principles and techniques aimed at calculating the signal-to-noise ratio (SNR) for each region of the face. Some embodiments do so by discarding a region if its SNR falls below a threshold SNR or if its maximum amplitude exceeds a threshold amplitude. Thus, only a select few facial regions may be chosen for further processing, thereby reducing the complexity of signal estimation.

[0040] Referring to Figure 1A, the iPPG system 100 jointly analyzes multiple spatial regions 103 to estimate vital signs in order to reduce the effects of noise. According to some exemplary embodiments, the vital signs may be one or a combination of the subject's pulse rate and / or the subject's heart rate variability (also referred to as the “heart rate signal”). In some embodiments, the subject’s vital signs may be one-dimensional signals, with the dimension being time.

[0041] Some embodiments are based on the understanding that vital signs can be accurately estimated by employing time analysis. Thus, the iPPG system 100 is configured to extract multidimensional time-series data 108 from a sequence of images corresponding to different areas of the subject's skin, and the multidimensional time-series data 108 is used to determine the PPG signal to accurately estimate vital signs. Some embodiments are also based on the novel fact that, when acquiring video 105 using an RGB camera, the iPPG system 100 is configured to calculate the ratio of one to one of the color channels in the video 105 to one or more other color channels in order to improve sensitivity to lighting variations.

[0042] The estimated multidimensional time series data 108 is provided to the PPG estimator module 109 to recover the signal of interest (PPG signal) from the noisy multidimensional time series data 108. The PPG estimator module includes a cross-domain expansion algorithm 109a configured to recover and output the PPG signal. The cross-domain expanded iPPG algorithm 109a is realized by expanding iterations of a model-based proximity descent algorithm for the PPG estimator module 109. Each iteration includes a gradient calculation step 109b that reduces the data fitting error term and one or more denoisingers 109c that help bring the output of the gradient step closer to a clean PPG signal. After iterating the cross-domain expanded iPPG algorithm 109a a certain number of times, several vital signs 111 of the subject can be estimated using the recovered PPG signal. For example, the estimate of vital signs in a time window can be considered as the frequency component with the maximum power in the frequency spectrum.

[0043] The number of iterations in algorithm 109a may be a dynamic variable defined by the operator, or it may be defined in relation to the estimated vital signs. In some exemplary embodiments, input from the operator may be provided to run algorithm 109a for a defined number of iterations. In some other exemplary embodiments, the threshold for the number of iterations may be obtained from a database that defines the number of iterations in relation to one or more of the subject's vital signs to be estimated, body parts considered for imaging, sex, age, or other physiological characteristics. In some exemplary embodiments, the threshold for the number of iterations may be a hyperparameter of the iPPG algorithm 109a. The value corresponding to the number of iterations may be provided as a termination condition for algorithm 109a.

[0044] Note that in some cases, the cross-domain expansion algorithm 109a estimates the iPPG signal in the frequency domain. Noise in the time series data 108 is processed in the frequency domain by algorithm 109a. Doing so restricts the time series data 108 to only a small number of frequencies. In some exemplary embodiments, the time series data in the time domain may be converted to the frequency domain, and then the difference between the output signal and its ground truth may be calculated.

[0045] Figure 1B shows a flowchart of a method 120 performed by the iPPG system 100 to estimate the vital signs 111 of a subject, according to several exemplary embodiments. Figure 1B is described in conjunction with Figure 1A. A video 105 of the subject is received 121 by the iPPG system 100. The video 105 may be a live video or a pre-recorded video of the subject and may correspond to at least one body part of the subject (e.g., face). The video 105 is captured under a suitable illumination spectrum such as NIR or visible or broad-spectrum visible and NIR wavelengths.

[0046] The iPPG system 100 extracts a sequence of images from different areas of the subject's skin. To this end, for each video 105, the iPPG system 100 acquires an image (e.g., image 107) from each image frame of the video 105. Each image is partitioned or segmented into multiple spatial regions (e.g., spatial region 103), resulting in a sequence of images corresponding to different areas of a body part (e.g., face). The partitioning of image 107 is performed such that each spatial region contains a specific area of ​​the body part that strongly exhibits a PPG signal. Thus, each of the multiple spatial regions 103 is a region of interest (ROI) for determining the PPG signal.

[0047] For each spatial region, a time-series signal is derived using the time-series extraction module 101. Thus, the time-series extraction module 101 of the iPPG system 100 converts a sequence of images into multidimensional time-series data 108 125. In some exemplary embodiments, for each image 105, the time-series extraction module 101 can extract 5-dimensional time-series data corresponding to the pixel intensity over time of five facial regions (ROIs), where these facial regions correspond to multiple spatial regions 103. In some embodiments, the multidimensional time-series signal may have more or fewer dimensions corresponding to more or fewer facial regions than five. In some embodiments, the five facial regions may correspond to the right cheek, left cheek, chin, right forehead, and left forehead. The time-series extraction module 101 is configured to convert a sequence of images corresponding to multiple spatial regions 103 into a multidimensional time-series signal 125. To this end, the pixel intensity fluctuations of pixels from each of the multiple spatial regions (also called "different spatial regions") 103 at a given point in time may be averaged to generate values ​​of different dimensions of the multidimensional time series signal at that point in time.

[0048] In some embodiments, the time series extraction module 101 may be further configured to temporally window (or segment) the multidimensional time series signal. Thus, there may be multiple segments of the multidimensional time series signal, and at least a portion of each of the multiple segments overlaps with subsequent segments of the multiple segments to form a sequence of overlapping segments. Furthermore, each of the segments may be normalized before submitting the multidimensional time series signal to the PPG estimator module 109. The windowed sequence may be a specific duration with a specific frame stride in interference (e.g., a 10-second duration with a 10-frame stride in inference (250 frames at 25 fps)), where the stride indicates a shift in the number of frames (e.g., 10 frames) over the windowed sequence (e.g., a 10-second windowed sequence).

[0049] Referring back to Figure 1A, the time-series extraction module 101 is configured to receive multiple frames of video 105 and extract a multidimensional time series 108 from the multiple frames. Figure 1C shows a flowchart of a method for extracting multidimensional time-series data from video of a subject, according to several exemplary embodiments. Captured video 105 of the subject is received 131 by the time-series extraction module 101 of the iPPG system 100 and fragmented 133 into multiple frames. Each individual frame of video 105 may correspond to an image 107 of at least one body part of the subject, and thus the frames of video 105 may result in a sequence of images. The sequence of images may correspond to different areas of the subject's skin, and each area in the sequence may contain pixels of different intensities showing variations in skin coloration. In image 107 of Figure 1A, a skin area located on the face (facial area) is shown, but it can be intended that various embodiments are not limited to using the face. In some embodiments, a sequence of images corresponding to other areas of exposed skin, such as a person's neck or wrist, may be acquired and processed by the time-series extraction module 101.

[0050] The time-series extraction module 101 detects frames containing desired body parts of the subject and selects those frames for further processing. For example, the time-series extraction module 101 may detect the subject's face within each RGB video frame. Next, landmark positioning 135 is used, and interpolation / extrapolation 137 is performed on its 68 landmark outputs for 145 landmarks. That is, in order to extract an ROI associated with a specific body part of the subject in image 107 of Figure 1A, multiple landmark positions corresponding to the specific body part of the subject are positioned 135 in each image frame of the video 105. Thus, the multiple landmark positions may vary depending on the body part used for PPG signal determination. In an exemplary embodiment, if a person's face is used to determine the PPG signal, 68 landmark positions corresponding to the person's face (i.e., 68 facial landmarks) are positioned in each image frame of the video 105.

[0051] Some embodiments are based on the understanding that image averaging reduces the effects of quantization noise from the camera generating the image 105, motion jitter due to incomplete landmark positioning, and minor deformations due to human head and face movements. In response to image averaging, multiple landmark positions are smoothed to extract a ROI (e.g., five facial regions).

[0052] Therefore, in some embodiments, before extracting ROIs from multiple landmark locations, the landmark locations are smoothed using a smoothing technique such as a moving average technique. In particular, a kernel of a predetermined size is moved across multiple landmark locations in the image to replace the pixel values ​​at each landmark location manipulated by the kernel with the average value of the pixel values ​​manipulated by the kernel.

[0053] For example, 68 landmark locations are smoothed using a moving average with a kernel of size 3 frames. Then, using the smoothed landmark locations, five ROIs located around the forehead, cheeks, and chin are extracted. Thus, in each frame of video 105, the average intensity of pixels in each of the five spatial regions is calculated. In this way, multiple spatial regions 103 (or ROIs) are extracted from each image, and these multiple spatial regions 103 form a sequence of images.

[0054] Referring back to Figure 1C, these landmarks are grouped into small spatial areas 139, and in each of them, the average pixel intensity of each illumination channel is calculated 141. For example, in some embodiments, if an RGB camera is used to acquire the image 105, the time-series extraction module 101 is configured to calculate the average pixel intensity of the red and green channels. In some exemplary embodiments, instead of using multiple illumination channels, the ratio of two illumination channels may be used as a metric for further processing. For example, one or more ratios of one color channel in the image 105 to another color channel may be calculated before forming the time series 108. In the exemplary scenario where the image 105 is captured by an RGB camera, the ratio of the red and green channels may be used for further processing.

[0055] Subsequently, the small spatial areas are grouped into spatial regions 143 based on the median intensity value of the area within each spatial region of a defined cluster size. Then, for each spatial region, multidimensional time-series data is extracted 145 corresponding to the pixel intensity over time. For example, the small spatial areas may be grouped into facial regions with K=5, and the median intensity value of those areas may be taken within each facial region. This yields a 5-dimensional time series 108 for each image. In some embodiments, a Butterworth filter with a cutoff frequency of [0.7, 2.5] Hz may be applied to the time series 108 to capture frequencies within a typical heart rate range.

[0056] Each dimension of the multidimensional time series data 108 may correspond to a different spatial region than multiple spatial regions of the subject's skin in image 107. Furthermore, each dimension may be a signal from an explicitly tracked region of interest (ROI) among multiple spatial regions of the subject's skin. Tracking reduces the amount of motion-related noise. However, the multidimensional time series data 108 may still contain significant noise due to factors such as landmark positioning errors, lighting variations, 3D head rotation, and deformations such as facial expressions.

[0057] The estimated multidimensional time series data 108 contains signal components that reflect the underlying pulse signal, which is useful for vital sign estimation, but the multidimensional time series data 108 is often noisy. Therefore, the multidimensional time series data 108 can be considered as a group of measured imaging PPG signals (measured iPPG signals). Thus, the purpose of the PPG estimator is to recover from the measured iPPG signals a noise-free iPPG signal that truly reflects the underlying pulse signal, which is useful for vital sign estimation. Therefore, the noise-free iPPG signal can be referred to as the recovered iPPG signal.

[0058] Referring back to Figure 1B, the PPG estimator module 109 of the iPPG system 100 processes the time series data 108 using a cross-domain expanded iPPG algorithm 109a 127, and the PPG estimator module 109 may process each segment from the overlapping sequence of multidimensional time series data 108 using a gradient step 109b and a denoising tool 109c. As part of the processing 127 by the PPG estimator module 109, the cross-domain expanded optimization iteratively minimizes the difference between the received / estimated iPPG signal and the reconstructed iPPG signal, and then processes the reconstructed iPPG signal using an iPPG neural network trained with machine learning in the frequency domain to implement the learned structure on the frequency coefficients of the reconstructed iPPG signal. The reconstructed iPPG signal has frequency coefficients obtained in a certain frequency domain, which are converted to the time domain of the received iPPG signal as part of step 127.

[0059] Once the frequency coefficients are obtained, the iPPG system 100, upon reaching the termination condition of the cross-domain expansion optimization algorithm 109a, estimates the subject's vital signs from the frequency coefficients of the reconstructed iPPG signal using any preferred technique 129.

[0060] To recover the target signal (recovered iPPG signal) from a noisy multidimensional time series 108, the multidimensional time series signal indicated by Z in Figure 1A is fed to the PPG estimator module 109. The PPG estimator module 109 is configured to recover and output the PPG signal from the noisy multidimensional time series signal using a cross-domain expanded iPPG algorithm 109a. Furthermore, several vital signs of a person can be estimated using the recovered PPG signal.

[0061] The cross-domain expanded iPPG algorithm 109a in Figure 1A is realized by expanding an iteration of a model-based proximity descent algorithm for the PPG estimator module 109. Each iteration includes a gradient calculation step 109b that reduces the data fitting error term and one or more denoisingers 109c that help bring the output of the gradient step closer to a clean PPG signal.

[0062] For each input time window containing S frames, a time series is extracted containing the average intensity of each of the K facial regions within each frame. When these signals are stacked onto a matrix Z of size S × K, it is assumed that these region-specific signals share a quasi-periodic pulse signal that allows for a structured representation in the Fourier domain. Therefore, observation of the heart rate signal is modeled as follows: Z=Y+E=F -1 X+E (1) Here, F -1is an oversampled inverse Fourier transform matrix of size S×N, where Y of size S×K and X of size N×K represent the pulsating signals in all K domains in the time and frequency domains, respectively, and E is an S×K real matrix representing the structured noise component that captures non-pulse-related fluctuations in the iPPG signal Z. Here, Z is measured in the time domain and processed in the Fourier domain using the Fourier transform F. This is based on the understanding that the Fourier domain acts as a structure implementer for the estimated PPG signal Y, and that X, given the Fourier transform in Y, has a simpler structure than in the measurement domain. X is the signal that is processed by the cross-domain expanded iPPG algorithm 109a instead of Y. Meanwhile, the structured noise component E is processed in the time domain by the cross-domain expanded iPPG algorithm 109a. That is, X and E, which are components of Z, are processed in different domains. In this exemplary embodiment, these two domains are the Fourier domain and the time domain, respectively. In other embodiments, these regions may also be wavelet regions or appropriate dictionaries learned from data.

[0063] The decomposition of Z into signal components X and structured noise components E in the Fourier domain must approximately satisfy the data fidelity term.

number

[0064] However, satisfying only the data fidelity term for decomposition is insufficient, since multiple combinations of X and E can be used to form Z. In other words, not all decompositions return X and E with a proper structure. For this purpose, deep denoisingers are trained to discover a proper structure in the Fourier domain.

number

number

[0065] In some embodiments, to find the heart rate, the power in all frequency bins across all K regions is summed, and the frequency with the maximum power is selected as the frequency of the signal representing the heart rate. For example, in one embodiment, the estimated vital sign is the frequency of the subject's heartbeat over the given period.

[0066] In the proposed iPPG system 100, in the expanded iPPG architecture, one or more encoder-decoder denoiser architectures are applied to the output of gradient step 109b in each iteration. Thus, the PPG estimator module 109 comprises one or more time-series denoiser neural networks (also called “denoisers”) 109c. The gradient step 109b and denoisers 109c are executed T times in succession to process multidimensional time-series data and accurately determine the PPG waveform in the cross-domain expanded iPPG system 109a, and the PPG waveform is used to estimate a person’s vital signs. The number of iterations T can be considered a hyperparameter of the iPPG algorithm 109a. According to some exemplary embodiments, but not limited to, T=3 for RMSE and PTE6 may give the desired performance.

[0067] In some exemplary scenarios, the vital sign to be estimated for a subject may be a heart rate signal, but the heart rate signal is locally periodic, and the period of the heart rate may change over time. In such cases, some embodiments are based on the recognition that a 10-second window is a good compromise for the period for extracting the current heart rate. The length of the stride frame may also vary depending on the vital sign of the subject to be estimated.

[0068] Figure 2 shows two comprehensive iterations (iteration t and iteration t+1) of the cross-domain expanded iPPG algorithm 109a implemented in the PPG estimator module 109. A multidimensional time series 108 (e.g., five dimensions of the time sequence) is supplied to the PPG estimator module 109 as a channel. The PPG estimator module 109 comprises a cross-domain expanded iPPG algorithm implemented as a sequence of expanded iterations of a neighbor gradient descent algorithm. Each iteration consists of a gradient step 109b, where the neighbor operators are replaced by trainable cross-domain denoisers 109c R and Q. The channels corresponding to the multidimensional time series signal are coupled during the gradient step and during the forward pass in the sequence of layers in the denoiser 109c. In the PPG estimator module 109, T iterations of the gradient step 109b and the denoiser 109c map the multidimensional time series signal to the desired PPG signal. For each windowed sequence (e.g., a 10-second window) of the multidimensional time series signal, the cross-domain expanded iPPG architecture 109a extracts the PPG signal from the multidimensional time series 108 obtained from the ROI.

[0069] Figure 3 shows the architectures of denoisingers R and Q used by the iPPG system of Figure 1A in several exemplary embodiments. In the cross-domain expanded iPPG, the trained neural network denoisingers R and Q are modeled using an encoder-decoder architecture. As shown in Figure 3, the encoder 302 for network R is realized through blocks 302a and 302b, the decoder for network R is realized through blocks 304a and 304b, the encoder for network Q is realized through blocks 306a and 306b, and the decoder for network Q is realized through blocks 308a and 308b. Network R is represented by the Fourier coefficient X t Since it operates in E, its network weights are complex numbers (blocks indicated by thick, thick boundaries), t The network weights for Q, which operates in this manner, are real-valued (blocks indicated by thin boundaries). The network consists of two downsampling convolutional blocks 302a and 302b for network R and 306a and 306b for network Q, with a stride of 2 and increasing the number of channels from 5 to 32, and then from 32 to 64. Two upsampling blocks 304a and 304b for network R and 308a and 308b for network Q are implemented using transposed convolutions with a stride of 2, reducing the number of channels from 64 to 32, and from 32 to 5. Each convolutional layer has a kernel size of 16 in the time dimension, followed by a ReLU nonlinearity, and then a batch normalization layer.

[0070] In some embodiments, during training, the parameters of the neural networks R and Q are set to the output signal after T unfolded iterations, Y. T =F -1 X T And, ground truth waveform Z gt The parameters of the denoisingers R and Q are updated to minimize the mean squared error loss between them. Mini-batch stochastic gradient descent using backpropagation may also be used to update the parameters of the denoisingers R and Q.

[0071] Next, the details of any training data set used in the noise eliminator and the details of the experiments related to the performance of the cross-region deployed iPPG algorithm will be described. In some exemplary embodiments, the cross-region deployed iPPG algorithm 109a can be trained using a Multimodal Spontaneous Expression-Heart Rate (MMSE-HR) data set. This data set includes 102 videos from 23 female subjects and 17 male subjects, capturing the face and simultaneous blood pressure waves from a finger sensor in states that induce various emotions. This results in substantial movement in some videos, but the cross-region deployed iPPG algorithm is robust to it. The videos were captured at 25 frames per second with a resolution of  1040×1392, and the blood pressure waves were measured at 1000 samples per second. The ground truth data is downsampled to match the frame rate of the video in an experiment where the convolutional layers are initialized to output 0 and a single skip connection is added in the top convolutional layer. The variable X input to R is initialized as the Fourier transform of Z. The noise E input to Q is initialized as a zero matrix. The mean squared error between each of the five output channels and the ground truth is calculated, and the Adam optimizer is used with a learning rate of 3×10 -4 for 8 epochs to update the parameters of the noise eliminators R and Q. The training data is augmented using augmentations called "slow down" and "speed up" so that the heart rates at the lower and upper ends not well represented in the data set can be estimated. In the case of the "slow down" augmentation, an input window of length S is cropped by a random percentage between 20% and 40% and interpolated back to the original window size S using linear interpolation. In the case of the "speed up" augmentation, given a window length S, an input window length 20% - 40% larger than the target time window is randomly selected (e.g., 1.2×S) and interpolated back to length S.

[0072] During training, each empirical ground truth waveform is divided into 10-second windows, and then the windows are shifted by 2.4 seconds to obtain the next partially overlapping window for training. The windows are loaded randomly in batches of 100 during training. During testing, the 10-second segments are reconstructed in a non-overlapping manner.

[0073] For evaluation purposes, the mean absolute error (MAE) and root mean square error (RMSE) of the ground truth and predicted heart rate calculated for a 30-second time window of the test video are reported. For this purpose, three adjacent 10-second output windows from the deployed iPPG system are concatenated to perform the evaluation for a 30-second window. The MAE and RMSE metrics are averaged across all B windows and across all test set partitions for all test video. A metric called PTE6 is also reported, which is the percentage of time when the heart rate error is less than 6 beats / minute (bpm), and this is a way to measure how often the estimated heart rate is correct.

[0074] Figure 4 shows a tabular comparison of the performance of the expanded iPPG system in Figure 1A with the performance of several known methods. Expanded iPPG significantly outperforms model-based methods such as AutoSparsePPG by reducing the MAE error from 4.55 bpm to 1.26 bpm and the RMSE from 14.42 bpm to 3.87 bpm. PTE6 also increases from 88.10% to 94.57%. Given that expanded iPPG incorporates learned components, it can also be compared to the data-driven methods CAN and InverseCAN. Expanded iPPG outperforms both CAN and InverseCAN. Compared to InverseCAN, our expanded iPPG method reduces the MAE from 2.27 bpm to 1.26 bpm and the RMSE from 4.90 bpm to 3.87 bpm.

[0075] Figures 5A–5C illustrate how the expanded iPPG iteratively estimates the underlying pulse signal and spectrum for a single test video in the MMSE-HR dataset. Specifically, Figure 5A shows the first iteration of the expanded iPPG estimation, Figure 5B shows the second subsequent iteration, and Figure 5C shows the third subsequent iteration. In each of Figures 5A–5C, the projected curve is shown as a dotted line and the ground truth curve is shown as a solid line. Comparing the outputs of the first, second, and third iterations to the true signal, the reconstructed iPPG signal progressively approaches the true signal as the iterations progress.

[0076] Figure 5D shows an example of unfolded iPPG performance (shown as a dashed line representing estimated heart rate in bpm) for test footage from the MMSE-HR dataset, over consecutive 10-second time windows. The unfolded iPPG system (dashed curve) accurately predicts ground truth heart rate (shown as a solid curve) over a wide range of heart rates and over the duration of the entire video (along the horizontal axis in seconds).

[0077] Figure 6 shows the impact on heart rate estimation performance of explicitly modeling noise using a noise remover Q (instead of modeling Fourier coefficients and noise simultaneously in R). Using both Q and R significantly improves the results compared to using R alone. The network Q learns the structure of the noise, which can be obtained by subtracting it from the signal Z in equation (1).

[0078] Figure 7 analyzes an exemplary scenario illustrating the effect of the number of expansion iterations T (continuous gradient step and denoising step) on cardiac estimation performance. The results show that expanded iterations with T=3 provide the best performance.

[0079] Figure 8 shows a block diagram of an iPPG system 800 according to an exemplary embodiment. The system 800 includes a processor 801 configured to execute stored instructions and a memory 803 that stores instructions executable by the processor 801. The processor 801 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 803 may include random-access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. The processor 801 is connected to one or more input and output devices via a bus 805.

[0080] Instructions stored in memory 803 correspond to an iPPG method for estimating a person's vital signs based on a set of iPPG signal waveforms measured from different areas of the subject's skin. The iPPG system 800 may also include a storage device 807 configured to store various modules, such as a time-series extraction module 101 and a PPG estimator module 109, the PPG estimator module 109 comprising an implemented cross-domain expanded iPPG algorithm 109a. The aforementioned modules stored in storage device 807 are executed by processor 801 to perform vital sign estimation. The vital signs may correspond to a person's pulse rate or a person's heart rate variability. Storage device 807 may be implemented using a hard drive, an optical drive, a thumb drive, an array of drives, or any combination thereof.

[0081] The time-series extraction module 101 acquires images in each frame of one or more images 809 supplied to the iPPG system 800, one or more images 809 containing images of body parts of a subject from which vital signs will be estimated. One or more images may be recorded by one or more suitable imaging devices. The time-series extraction module 101 can partition the images from each frame into multiple spatial regions corresponding to ROIs of body parts, which are strong indicators of the PPG signal, and the partitioning of images into multiple spatial regions forms a sequence of images of body parts. Each image contains a different region of skin of a body part within the image. The sequence of images can be converted into a multidimensional time-series signal as described above with reference to Figure 1C. The multidimensional time-series signal can be supplied to the PPG estimator module 109. The PPG estimator module 109 uses a cross-domain expanded iPPG algorithm 109a to process multidimensional time-series signals by performing a series of expanded iterations that continuously repeat the reduction of the data fidelity error term by a gradient step 109b and passing through one or more denoisingers 109c T times, thereby estimating a PPG waveform, and using that PPG waveform, it can estimate a person's vital signs.

[0082] The iPPG system 800 includes an input interface 811, such as an input port for receiving one or more video 809. For example, the input interface 811 may be a network interface controller adapted to connect the iPPG system 800 to a network 813 via a bus 805.

[0083] In addition, or as an alternative, in some implementations, the iPPG system 800 is connected to a remote sensor 815, such as an imaging sensor, to collect one or more images 809. In some implementations, a human-machine interface (HMI) 817 within the iPPG system 800 is connected to an input device 819, such as a keyboard, mouse, trackball, touchpad, joystick, pointing stick, stylus, or touchscreen, to receive input from a user or operator.

[0084] The iPPG system 800 may be linked to an output interface via bus 805 to render PPG waveforms. For example, the iPPG system 800 may include a display interface 821 adapted to connect the iPPG system 800 to a display device 823, which may include, but is not limited to, a computer monitor, projector, or mobile device.

[0085] The iPPG system 800 may also include, and / or be connected to, an imaging interface 825 adapted to connect the iPPG system 800 to an imaging device 827.

[0086] In some embodiments, the iPPG system 800 may be connected via a bus 805 to an application interface 829 adapted to connect the iPPG system 800 to an application system 831 that can operate based on estimated vital signs. In an exemplary scenario, the application system 831 may be a patient monitoring system that uses the patient's vital signs. In another exemplary scenario, the application system 831 may be a driver monitoring system that uses the driver's vital signs to determine whether the driver is able to drive safely, for example, whether the driver is drowsy.

[0087] The proposed approach for estimating the vital signs of a subject may be used for several control applications, some of which are described below with reference to Figures 9 and 10.

[0088] In some exemplary embodiments, the subject from which vital signs are to be estimated may be a patient. In such exemplary scenarios, the iPPG system may be used to monitor the patient's vital signs. Figure 9 shows an exemplary patient monitoring system 900 using an iPPG system 100 or 800 according to an exemplary embodiment. To monitor the vital signs of patient 901, a camera 903 is used to image patient 901 and acquire a video sequence of patient 901. The camera 903 may include a CCD sensor or CMOS sensor for converting incident light and its intensity variations into electrical signals. The camera 903 captures light reflected from a portion of the patient's skin, particularly non-invasively. Thus, the portion of skin may refer in particular to the forehead, neck, wrists, part of the arms, or any other portion of the patient's skin. A light source, such as a near-infrared light source, may be used to illuminate the area of ​​the subject, including the patient or the portion of the patient's skin.

[0089] Based on the captured images, the iPPG system 800 determines the vital signs of patient 901 in accordance with the exemplary embodiment described above. In particular, the iPPG system 800 determines vital signs such as heart rate, respiratory rate, or blood oxygenation of patient 901. Furthermore, the determined vital signs may be displayed on the operator interface 905 to present the determined vital signs. Such an operator interface 905 may be a patient bedside monitor, a remote monitoring station in a dedicated room within the hospital, or even a remote monitoring station for telemedicine applications in a remote location.

[0090] In some exemplary embodiments, the subject whose vital signs are to be estimated may be the driver or passenger of the vehicle. Figure 10 shows an occupant assistance system 1000 using an iPPG system 800 according to an exemplary embodiment. An NIR light source and / or an NIR camera 1001 are located inside the vehicle 1003. In particular, the NIR camera 1001 may be positioned in a field of view (FOV) 1007 that captures the driver or passenger 1005. The iPPG system 800 is incorporated into the vehicle 1003. The NIR light source is configured to illuminate the skin of the vehicle occupant, such as the driver or passenger 1005, and the NIR camera 1001 is configured to record images of the driver or passenger 1005 in real time. Furthermore, the NIR images are supplied to the iPPG system 800 to measure iPPG signals from different areas of the driver or passenger 1005's skin. The iPPG system 800 receives the measured iPPG signals and determines vital signs such as the pulse rate of the driver or passenger 1005.

[0091] Upon obtaining vital signs of the passenger or driver 1005, the iPPG system 800 may process the estimated vital signs to check the passenger or driver 1005's condition. In some exemplary embodiments, the processing of the estimated vital signs of the passenger or driver 1005 may be performed by the vehicle 1003's control system or a remote server communicatively coupled to the vehicle 1003's control system. For example, the estimated vital signs of the passenger or driver 1005 can be compared to an acceptable threshold to confirm the passenger or driver 1005's medical health status. If the comparison indicates that the passenger or driver 1005's medical health status is unfavorable, appropriate action may be initiated. For example, the processor of the iPPG system 800 may generate one or more control action commands based on the estimated vital signs of the driver 1005 of the vehicle 1003. One or more control action commands may include vehicle braking, steering control, generation of warning notifications, initiation of an emergency service request, or switching the driving mode from manual to automatic or automatic to manual. One or more control action commands may be sent to the controller 1005 of the vehicle 1003. The controller 1005 can control the vehicle 1003 in accordance with one or more control action commands. For example, if the determined driver's pulse rate is very low, the driver 1005 may be having a heart attack. As a result, the iPPG system 800 may generate control commands for decelerating and / or steering the vehicle (e.g., steering the vehicle to the shoulder of a main road and bringing it to a stop) and / or initiating an emergency service request.

[0092] In this way, some exemplary embodiments described herein may be used in real-world applications for medical diagnosis and welfare, vehicle assistance, and patient monitoring.

[0093] The above description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the above description of exemplary embodiments provides a practicable description for realizing one or more exemplary embodiments for those skilled in the art. Various modifications that may be made in the function and arrangement of the elements are contemplated without departing from the spirit and scope of the subject matter disclosed as described in the claims.

[0094] The above description provides specific details for a complete understanding of the embodiments. However, those skilled in the art will understand that embodiments may be carried out without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in the form of block diagrams so as not to obscure the embodiments with unnecessary details. In other examples, well-known processes, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments. Furthermore, similar reference numbers and names in various drawings indicate similar elements.

[0095] Furthermore, individual embodiments may be described as processes shown as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. While flowcharts may describe operations as sequential processes, many operations can be performed in parallel or simultaneously. In addition, the order of operations may be reordered. A process may terminate when its operations are complete, but it may have additional steps that are not discussed or included in the diagrams. Moreover, not all operations in any particular process described may occur in all embodiments. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the function may correspond to the function's return to the calling function or the main function.

[0096] Furthermore, embodiments of the disclosed subject matter may be implemented at least partially manually or automatically. Manual or automatic implementations may be performed using, or at least assisted by, a machine, hardware, software, firmware, middleware, microcode, a hardware description language, or any combination thereof. If implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the required tasks may be stored in a machine-readable medium. The required tasks may be performed by a processor.

[0097] The various methods or processes outlined herein may be coded as software executable on one or more processors using any one of various operating systems or platforms. In addition, such software may be written using any of several preferred programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine language code or intermediate code that runs on a framework or virtual machine. Typically, the functionality of program modules may be combined or distributed as desired in various embodiments.

[0098] Embodiments of this disclosure may be embodied as the methods provided in the examples. The actions performed as part of the methods may be ordered in any preferred manner. Thus, embodiments may be constructed in which the actions are performed in a different order than those illustrated, which may include performing several actions simultaneously, as shown as sequential actions in the exemplary embodiments. While this disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adaptations and modifications are possible within the spirit and scope of this disclosure. Accordingly, the claims aspect is intended to encompass all such variations and modifications that are in the true spirit and scope of this disclosure.

Claims

1. A remote photoplethysmography (RPG) system for estimating vital sign signals of a subject, Memory for storing instructions, The system comprises a processor, the processor executing the instructions to the remote photoplethysmography (RPG) system, A sequence of imaging photoplethysmography (iPPG) signals measured from different areas of the subject's skin in the time domain is received. The processor performs cross-domain expansion optimization to iteratively minimize the difference between the received iPPG signal and the reconstructed iPPG signal, the reconstructed iPPG signal having frequency coefficients, the frequency coefficients being determined in the frequency domain and converted to the time domain of the received iPPG signal before the calculation of the difference between the received iPPG signal and the reconstructed iPPG signal, the reconstructed iPPG signal being obtained as the output of an iPPG neural network trained using machine learning in the frequency domain to implement a learned structure on the frequency coefficients of the reconstructed iPPG signal, the processor further executes the instructions to the remote photoplethysmography (RPG) system, When the termination condition for the cross-region expansion optimization is reached, the vital sign signals of the subject are determined from the frequency coefficients of the reconstructed iPPG signal. An RPG system that outputs the vital sign signals corresponding to the reconstructed iPPG signals via an output interface.

2. The RPG system according to claim 1, wherein the cross-domain expansion optimization estimates the noise in the received iPPG signal and modifies the reconstructed iPPG signal with the estimated noise in order to minimize the difference between the received iPPG signal and the reconstructed iPPG signal.

3. The RPG system according to claim 2, wherein the noise is processed by a noise neural network to impose an implicit structure on the noise and generate structured components of the noise.

4. The RPG system according to claim 3, wherein the received iPPG signal is estimated in the frequency domain, the iPPG neural network processes the noise in the frequency domain to restrict the received iPPG signal to a set of active frequency coefficients, and the received iPPG signal is converted to the time domain before being combined with the reconstructed iPPG signal in the time domain.

5. The RPG system according to claim 3, wherein the noise is estimated in the time domain, and the noise neural network processes the noise in the time domain to constrain the structure of the noise in a manner defined by the noise neural network.

6. The RPG system according to claim 1, wherein the cross-domain expansion optimization uses gradient descent minimization to minimize the difference between the received iPPG signal and the reconstructed iPPG signal.

7. The RPG system according to claim 1, wherein the cross-domain expansion optimization uses proximity gradient descent minimization to minimize the difference between the received iPPG signal and the reconstructed iPPG signal.

8. The RPG system according to claim 1, wherein for at least some iterations of the cross-domain expansion optimization, the reconstructed iPPG signal is updated at least once in the time domain and at least once in the frequency domain by the iPPG neural network during the minimization of the difference between the received iPPG signal and the reconstructed iPPG signal.

9. The RPG system according to claim 3, wherein the noise neural network is trained with ground truth iPPG signals measured using contact sensing.

10. The RPG system according to claim 3, wherein the noise neural network comprises a trained neural network denoiser modeled using an encoder-decoder architecture.

11. The RPG system according to claim 10, wherein the weights of at least one of the trained neural network denoisers are complex values.

12. The RPG system according to claim 10, wherein the weights of at least one of the trained neural network denoisers are real numbers.

13. Each of the aforementioned trained neural network noise reducers is: It includes two downsampling convolutional blocks with a stride of 2, wherein the number of channels is increased from 5 to 32 in the first block of the downsampling convolutional block, and from 32 to 64 in the second block of the downsampling convolutional block, and each of the trained neural network denoisers further, The RPG system according to claim 10, comprising two upsampling blocks realized using transposed convolution with a stride of 2, wherein the number of channels is reduced from 64 to 32 in the third block of the upsampling block and from 32 to 5 in the fourth block of the upsampling block.

14. The RPG system according to claim 13, wherein each convolutional layer of the first block, the second block, the third block, and the fourth block has a kernel size of 16 in the time dimension, followed by ReLU nonlinearity and batch normalization layers.

15. The remote photoplethysmography (RPG) system further comprises a controller that is communicatively coupled to the machine, the controller being, The vital signs of the subject are received, The RPG system according to claim 1, which generates one or more control commands for controlling the machine based on the vital signs of the subject received.

16. A computer-based method for estimating vital sign signals of a subject, Receiving a sequence of imaging photoplethysmography (iPPG) signals measured from different regions of the subject's skin in the time domain, The method includes performing cross-domain expansion optimization to iteratively minimize the difference between a received iPPG signal and a reconstructed iPPG signal, wherein the reconstructed iPPG signal has frequency coefficients, which are determined in the frequency domain and converted to the time domain of the received iPPG signal before the calculation of the difference between the received iPPG signal and the reconstructed iPPG signal, and the reconstructed iPPG signal is obtained as the output of an iPPG neural network trained using machine learning in the frequency domain to implement a learned structure on the frequency coefficients of the reconstructed iPPG signal, and the method further includes, When the termination condition for the cross-region expansion optimization is reached, the vital sign signals of the subject are determined from the frequency coefficients of the reconstructed iPPG signal, A computer-implemented method, comprising outputting the vital sign signals corresponding to the reconstructed iPPG signals via an output interface.

17. Using the aforementioned cross-domain expansion optimization, the noise of the received iPPG signal is estimated, The computer-implemented method according to claim 16, further comprising correcting the reconstructed iPPG signal with estimated noise in order to minimize the difference between the received iPPG signal and the reconstructed iPPG signal.

18. The computer-implemented method according to claim 17, further comprising processing the noise with a noise neural network to impose an implicit structure on the noise and generate structured components of the noise.

19. The computer-implemented method according to claim 18, wherein the received iPPG signal is estimated in the frequency domain, the noise is processed in the frequency domain by the noise neural network to restrict the received iPPG signal to a set of active frequency coefficients, and the received iPPG signal is converted to the time domain before being combined with the reconstructed iPPG signal in the time domain.

20. A non-temporary computer-readable medium storing computer-executable instructions that, when executed by a computer, cause the computer to perform a method for estimating vital sign signals of a subject, wherein the method is Receiving a sequence of imaging photoplethysmography (iPPG) signals measured from different regions of the subject's skin in the time domain, The method includes performing cross-domain expansion optimization to iteratively minimize the difference between a received iPPG signal and a reconstructed iPPG signal, wherein the reconstructed iPPG signal has frequency coefficients, which are determined in the frequency domain and converted to the time domain of the received iPPG signal before the calculation of the difference between the received iPPG signal and the reconstructed iPPG signal, and the reconstructed iPPG signal is obtained as the output of an iPPG neural network trained using machine learning in the frequency domain to implement a learned structure on the frequency coefficients of the reconstructed iPPG signal, and the method further includes, When the termination condition for the cross-region expansion optimization is reached, the vital sign signals of the subject are determined from the frequency coefficients of the reconstructed iPPG signal, A non-temporary computer-readable medium, which includes outputting the vital sign signals corresponding to the reconstructed iPPG signals via an output interface.

Images