Non-contact oxygen saturation estimation

Abstract

A non-contact estimation of oxygen saturation comprises processing a video signal of a face of a human subject illuminated by ambient light to obtain a plurality of color components in different color spaces Statistical parameters of pixel values are calculated for the video frames and color components to generate a PPG signal, from which features are extracted. Oxygen saturation is estimated for the subject based on the features and an oxygen saturation estimation model (350) trained for estimating oxygen saturation based on input features.

Description

[0001] NON-CONTACT OXYGEN SATURATION ESTIMATION

[0002] TECHNICAL FIELD

[0003] The present invention generally relates to oxygen saturation estimation, and in particular to a method and a system for non-contact estimation of oxygen saturation and a method for generating an oxygen saturation estimation model useful in such non-contact estimation of oxygen saturation.

[0004] BACKGROUND

[0005] Oxygen saturation is the fraction of oxygen-saturated hemoglobin relative to total hemoglobin (unsaturated + saturated) in the blood. The human body requires and regulates a very precise and specific balance of oxygen in the blood. Normal arterial blood oxygen saturation (SaO2) levels in humans are 95-100 %. If the level is below 90 %, it is considered low and called hypoxemia. Arterial blood oxygen levels below 80 % may compromise organ function, such as the brain and heart. Continued low oxygen levels may lead to respiratory and / or cardiac arrest.

[0006] Oxygen saturation can be measured in different tissues, including arterial oxygen saturation (SaO2) as determined by arterial blood gas tests, venous oxygen saturation (SvO2) typically used during treatment with a heart lung machine (extracorporeal circulation), tissue oxygen saturation (StO2) measured by near infrared (NIR) spectroscopy and peripheral oxygen saturation (SpO2), which is an approximation of SaO2 and is usually measured by a pulse oximeter device. SpO2 can be calculated with pulse oximetry according to the formula:

[0007] HbO2

[0008] Sp°2~ HbO2+ Hb where HbO2 is oxygenated hemoglobin (oxyhemoglobin) and Hb is deoxygenated hemoglobin. The pulse oximeter consists of a small device that clips to the body (typically a finger, an earlobe or an infant's foot) and transfers its readings to a reading meter by wire or wirelessly. The pulse oximeter uses light-emitting diodes of different wavelengths in conjunction with a light-sensitive sensor to measure the absorption of red and infrared light in the extremity. The difference in absorption between oxygenated and deoxygenated hemoglobin makes the calculation possible according to the above presented formula. There is, though, a need for more convenient measurements of oxygen saturation, and in particular for non-contact or contactless oxygen saturation measurements that do not require attaching or connecting any measurement equipment to the body of a subject.

[0009] U.S. Patent No. 11,103,144 discloses a method of measuring a physiological parameter, such as oxygen saturation level, in a contactless manner. The method includes acquiring a plurality of image frames for a subject, acquiring a first color channel value, a second color channel value, and a third color channel value for at least one image frame included in the plurality of image frames. The method further includes calculating a first difference and a second difference on the basis of the first color channel value, the second color channel value, and the third color channel value for at least one image frame included in the plurality of image frames. The first difference represents a difference between the first color channel value and the second color channel value for the same image frame, and the second difference represents a difference between the first color channel value and the third color channel value for the same image frame.

[0010] U.S. Patent No. 10,888,280 discloses a photoplethysmography (PPG) circuit that obtains PPG signals at a plurality of wavelengths. A signal processing module obtains at least a first spectral response around a first wavelength and a second spectral response around a second wavelength. The signal processing device generates PPG input data using the PPG signals. The PPG input data includes one or more parameters obtained from each of the first spectral response and the second spectral response. A neural network processing device generates an input vector including the PPG input data and determines an output vector including health data. The health data includes an oxygen saturation level, a glucose level or a blood alcohol level.

[0011] U.S. Patent Application Publication No. 2023 / 0000377 discloses contactless image-based blood oxygen estimation. An image or video of a part of a subject captured by a camera of a computing device is received. A region of interest of the part of the subject is extracted from the image or video. Feature extraction of the region of interest is performed and a blood oxygen saturation level of the subject is estimated based on a spatial and temporal data analysis of more than two color channels. Feature extraction and estimation of the blood oxygen saturation level implement a combination of spatial averaging, color channel mixing, and temporal trend analysis.

[0012] SUMMARY It is general objective to provide a non-contact oxygen saturation estimation that does not require special lighting conditions.

[0013] This and other objectives are met by embodiments of the invention.

[0014] The present invention is defined in the independent claims. Further embodiments of the invention are defined in the dependent claims.

[0015] An aspect of the invention relates to a method for non-contact estimation of oxygen saturation. The method comprises converting a video signal of a face of a human subject illuminated by ambient light from an initial color space comprising multiple color components to multiple additional color spaces. Each additional color space comprises multiple color components and the video signal comprises a plurality of video frames. The method also comprises computing, for each video frame and each color component of the initial color space and the multiple additional color spaces, statistical parameters of pixel values in the video frame. The method further comprises generating, for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, a photoplethysmography (PPG) signal representing the statistical parameter of a respective color component for the plurality of video frames. The method additionally comprises determining features representative of a respective PPG signal and estimating oxygen saturation for the human subject based on the determined features and an oxygen saturation estimation model trained for estimating oxygen saturation based on input features.

[0016] Another aspect of the invention relates to computer-implemented method of generating an oxygen saturation estimation model. The method comprises, for each video signal of a plurality of video signals of faces of a plurality of human subjects illuminated by ambient light, wherein each video signal comprises a plurality of video frames, converting the video signal from an initial color space comprising multiple color components to multiple additional color spaces, wherein each additional color space comprises multiple color components. The method also comprises computing, for each video frame and each color component of the initial color space and the multiple additional color spaces, statistical parameters of pixel values in the video frame. The method further comprises generating, for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, a PPG signal representing the statistical parameter of a respective color component for the plurality of video frames. The method additionally comprises determining features representative of each PPG signal and a respective feature importance score for each feature. The oxygen saturation estimation model is trained based on features selected based on the feature importance scores and actual oxygen saturation values obtained for the plurality of human subjects.

[0017] A further aspect of the invention relates to a non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to convert a video signal of a face of a human subject illuminated by ambient light from an initial color space comprising multiple color components to multiple additional color spaces. Each additional color space comprises multiple color components and the video signal comprises a plurality of video frames. The processor is also caused to compute, for each video frame and each color component of the initial color space and the multiple additional color spaces, statistical parameters of pixel values in the video frame. The processor is further caused to generate, for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, a PPG signal representing the statistical parameter of a respective color component for the plurality of video frames. The processor is additionally caused to determine features representative of a respective PPG signal and estimate oxygen saturation for the human subject based on the determined features and an oxygen saturation estimation model trained for estimating oxygen saturation based on input features.

[0018] Yet another aspect of the invention relates to a non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to, for each video signal of a plurality of video signals of faces of a plurality of human subjects illuminated by ambient light, wherein each video signal comprises a plurality of video frames, convert the video signal from an initial color space comprises multiple color components to multiple additional color spaces, wherein each additional color space comprising multiple color components. The processor is also caused to compute, for each video frame and each color component of the initial color space and the multiple additional color spaces, statistical parameters of pixel values in the video frame. The processor is further caused to generate, for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, a PPG signal representing the statistical parameter of a respective color component for the plurality of video frames. The processor is additionally caused to determine features representative of each PPG signal and a respective feature importance score for each feature and train the oxygen saturation estimation model based on features selected based on the feature importance scores and actual oxygen saturation values obtained for the plurality of human subjects.

[0019] An aspect of the invention relates to a system for non-contact estimation of oxygen saturation. The system comprises a camera configured to record a video signal of a face of a human subject illuminated by ambient light. The system also comprises at least one memory configured to store an oxygen saturation estimation model trained for estimating oxygen saturation based on input features and the video signal recorded by the camera. The system further comprises at least one processor configured to convert the video signal stored in the at least one memory from an initial color space comprising multiple color components to multiple additional color spaces. Each additional color space comprises multiple color components and the video signal comprises a plurality of video frames. The processor is also caused to compute, for each video frame and each color component of the initial color space and the multiple additional color spaces, statistical parameters of pixel values in the video frame. The processor is further caused to generate, for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, a PPG signal representing the statistical parameter of a respective color component for the plurality of video frames. The processor is additionally caused to determine features representative of a respective PPG signal and estimate oxygen saturation for the human subject based on the determined features and the oxygen saturation estimation model stored in the at least one memory.

[0020] The present invention enables non-contact or contactless estimation of oxygen saturation without the need for special lighting, such as a dedicated infrared light source. In clear contrast, contactless estimation of oxygen saturation can be conducted in ambient light conditions. Hence, no dedicated light source with special light spectrum is needed as ambient light sources and even daylight could be used as “light source” when conducting the contactless oxygen saturation estimation.

[0021] BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The embodiments, together with further objects and advantages thereof, may best be understood by making reference to the following description taken together with the accompanying drawings, in which:

[0023] Fig. 1 is a flow chart illustrating a method for non-contact estimation of oxygen saturation according to an embodiment;

[0024] Fig. 2 is a flow chart illustrating a computer-implemented method of generating an oxygen saturation estimation model according to an embodiment;

[0025] Fig. 3 is a flow chart illustrating additional, optional steps of the method shown in Fig. 1 and 2;

[0026] Fig. 4 is a flow chart illustrating an embodiment of determining an illumination intensity value of Fig. 3; Fig. 5 is a flow chart illustrating additional, optional steps of the method shown in Fig. 2;

[0027] Fig. 6 is a schematic illustration of a device configured to generate an oxygen saturation estimation model according to an embodiment;

[0028] Fig. 7 is a schematic illustration of a device for non-contact estimation of oxygen saturation and / or generation of an oxygen saturation estimation model according to an embodiment; and

[0029] Fig. 8 is a schematic illustration of a system for non-contact estimation of oxygen saturation according to an embodiment.

[0030] DETAILED DESCRIPTION

[0031] The present invention generally relates to oxygen saturation estimation, and in particular to a method and a system for non-contact or contactless estimation of oxygen saturation and a method for generating an oxygen saturation estimation model useful in such non-contact or contactless estimation of oxygen saturation.

[0032] The current techniques for estimating oxygen saturation in a human subject are either contact-dependent techniques or require special measurement conditions. The contact-dependent techniques use a pulse oximeter device clipped to a body extremity of the human subject to perform the oxygen saturation estimations by measuring absorption of red and infrared light in the body extremity. Contactless techniques have been proposed in the art to estimate tissue oxygen saturation (StO2) by near infrared (NIR) spectroscopy. These contactless techniques therefore require the presence of an infrared light source in order to perform the StO2 measurements.

[0033] The present invention enables contactless estimation of oxygen saturation but does not require the presence of a dedicated infrared light source. In clear contrast, the oxygen saturation estimation of the invention can be conducted in ambient light conditions. Hence, no dedicated light source with special light spectrum is needed as ambient light sources and even daylight could be used as “light source” when conducting the oxygen saturation estimation.

[0034] In an embodiment, oxygen saturation as estimated in accordance with the invention is peripheral oxygen saturation (SpC>2). An aspect of the invention relates to a method for non-contact estimation of oxygen saturation, see Fig. 1 . The method comprises converting, in step S1 , a video signal of a face of a human subject illuminated by ambient light from an initial color space comprising multiple color components to multiple additional color spaces. Each additional color space comprises multiple color components. Furthermore, the video signal comprises a plurality of video frames. The method also comprises computing statistical parameters, also referred to as statistics, of pixel values in a video frame in step S2. This step S2 is performed for each video frame of the plurality of video frames and each color component of the initial color space and the multiple additional color spaces, which is schematically indicated by the loop L1. The method also comprises generating, in step S3 and for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, a photoplethysmography (PPG) signal representing the statistical parameter of a respective color component for the plurality of video frames. This step S3 is, as mentioned above, performed for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, which is schematically indicated by the loop L2. A next step S4 comprises determining features representative of the respective PPG signal. The method further comprises estimating oxygen saturation for the human subject in step S5 based on the determined features and an oxygen saturation estimation model trained for estimating oxygen saturation based on input features.

[0035] According to the invention, the video signal of the face of the human subject is recorded while the human subject, or at least the face of the human subject, is illuminated by ambient light. The ambient light is preferably ambient visible light, i.e., light having wavelengths in the range of 400 to 700 nm. The ambient light illuminating the face of the human subject could be from one or more light sources or lamps present in the room or facility where the oxygen saturation estimation is conducted. The at least one light source could, for instance, be one or more light sources arranged in the ceiling, one or more light sources arranged at a wall and / or one or more stand-alone light sources. The at least one light source is not arranged to specifically illuminate the subject but merely to provide background or ambient illumination. The present invention is, however, not limited to having one or more light sources for conducting the noncontact estimation of oxygen saturation. In clear contrast, daylight from one or more windows could be sufficient as ambient light illuminating the skin of the face of the human subject.

[0036] The video signal is recorded by a camera. Any camera capable of recording the face of a human subject during ambient illumination could be used according to the invention. The camera could be a dedicated camera or could be part of another device, such as the camera of a smart phone, tablet or laptop. The video signal as recorded by the camera is in an initial color space also referred to as a color format. A color space describes a specific, measurable, and fixed range of possible colors and luminance values. Its most basic practical function is to describe the capabilities of a capture or display device to reproduce color information. Hence, a color space is a range of colors on a spectrum that can be interpreted and displayed on a visual plane. A color space identifies a particular combination of a color model and a mapping function, the expression “color space” is often used informally to identify a color model. A "color model" is an abstract mathematical model describing the way colors can be represented as tuples of numbers, e.g., triples in red, green, blue (RGB) or quadruples in cyan, magenta, yellow, key (CMYK). Adding a specific mapping function between a color model and a reference color space establishes within the reference color space a definite "footprint", known as a gamut, and for a given color model, this defines a color space. The usual reference standard is the CIELAB color space or CIE XYZ color space.

[0037] As mentioned above, each color space defines a color as tuples of numbers, also referred to as color components or color channels. For instance, red, green and blue are three color components or channels of the RGB color space.

[0038] The video signal of the face of the human subject is in an initial color space comprising multiple color components, such as RGB color space and comprising red, green and blue color components. This video signal is converted into multiple, i.e., at least two, additional color spaces other than the initial color space. Illustrative, but non-limiting, examples of such additional color spaces include the HSV, CIELAB, YIQ, CMYK and CIE XYZ color spaces. These additional color spaces comprise respective color components, such as hue, saturation and value for the HSV color space; L*, a* and b* for the CIELAB color space; Y, I and Q for the YIQ color space; cyan, magenta, yellow and key (black) for the CMYK color space, and X, Y and Z for the CIE XYZ color space. Hence, the number of color components do not necessarily have to be the same for the initial color space and all additional color spaces.

[0039] The video signal is, thus, converted from the initial color space to at least two additional color spaces in step S1 . This step S1 could convert the video signal into two additional color spaces, into three additional color spaces, into four additional color spaces, into five additional color spaces, or indeed into six or more additional color spaces.

[0040] The following step S2 computes statistical parameters of pixel values in a video frame. This step S2 is performed for each video frame of the plurality of video frames in the video signal and for each color component of the initial color space and the multiple additional spaces. This means that, for each video frame, M different sets of statistical parameters are computed in step S2, wherein M represents the total number of color components in the initial color space and the multiple color spaces, i.e. , M = CCI+ CCL, wherein CCi indicates the number of color components of the initial color space, CCi indicates the number of color components of additional color space no. / G [1 , / V] and N>2 represents the number of additional color spaces.

[0041] The statistical parameters computed in step S2 for a given color component are then used in step S3 to generate a PPG signal representing the statistical parameter of a respective color component for the plurality of video frames. As an example, if the video signal comprises P video frames and each set of statistical parameters comprise Q statistical parameters, then the statistical parameters are used in step S3 to generate Q PPG signals for the given color component. As a consequence, M*Q PPG signals are preferably generated in step S3.

[0042] A PPG signal is a signal, from which volumetric variations of blood circulation representing blood volume changes in the microvascular bed of the monitored tissue of the human subject can be detected. Accordingly, a PPG signal generally has a variation in amplitude representing volumetric variations of blood circulation. Accordingly, a cardiac cycle can be detected in such PPG signal as the portion of the signal between consecutive valleys (or consecutive peaks) in the PPG signal.

[0043] Features representative of the PPG signals are then determined in step S4 and used in step S5 to estimate the oxygen saturation for the human subject based on a trained oxygen saturation estimation model. The oxygen saturation estimation model has been trained for estimating oxygen saturation based on input features, which is further described herein in connection with Fig. 2. Hence, the non-contact estimation of oxygen saturation uses an oxygen saturation estimation model that outputs an estimate of oxygen saturation given input features.

[0044] Fig. 2 is a flow chart illustrating a computer-implemented (Cl) method of generating an oxygen saturation model. The method comprises steps S11 to S14, which are performed for each video signal of a plurality of video signals of faces of a plurality of human subjects illuminated by ambient light. Each video signal comprises a plurality of video frames. Step S11 comprises converting the video signal from an initial color space comprising multiple color components to multiple additional color spaces. Each additional color space comprises multiple color components. The method also comprises computing, in step S12 and for each video frame and each color component of the initial color space and the multiple additional color spaces, which is schematically indicated by the loop L3, statistical parameters of pixel values in the video frame. The method further comprises generating, in step S13 and for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, which is schematically illustrated by the loop L4, a PPG signal representing the statistical parameter of a respective color component for the plurality of video frames. The method additionally comprises determining features representative of each PPG signal and a respective feature importance score, also referred to as feature permutation score herein, for each feature in step S14. The oxygen saturation estimation model is trained in step S15 based on features selected based on the feature importance scores and actual oxygen saturation values obtained for the plurality of human subjects.

[0045] Thus, an oxygen saturation estimation model is trained based features extracted from PPG signals obtained for different human subjects. Steps S11 to S13 correspond to steps S1 to S3 in Fig. 1 but are performed for a plurality of video signals of faces of a plurality of human subjects illuminated by ambient light. Correspondingly, step S14 corresponds to step S4 in Fig. 1 but step S4 does not necessarily comprise determining feature importance scores. Accordingly, a plurality of features are determined for the PPG signals in step S14 for each such video signal and human subject. The oxygen saturation estimation model is then trained using the plurality of features and the actual oxygen saturation values of the human subjects in step S15. The training in step S15 thereby learns the oxygen saturation estimation model to correlate the features with oxygen saturation values.

[0046] The oxygen saturation estimation model can be trained in Fig. 2 to accurately estimate oxygen saturation of a human subject based on a video signal of a face of a human subject illuminated merely by ambient light. Thus, by providing a plurality of video signal signals from various human subjects, the oxygen saturation estimation model will learn how features of the PPG signals generated in step S13 reflect changes in oxygen saturation.

[0047] The actual oxygen saturation values input to the oxygen saturation estimation model during the training step S15 are preferably measured according to well-known oxygen saturation methods or techniques, for instance pulse oximetry measurements using a pulse oximeter device.

[0048] The oxygen saturation estimation model may be implemented according to various embodiments. For instance, the oxygen saturation estimation model is a computer-implemented oxygen saturation model and could be in the form a machine learning (ML) model. Generally, ML algorithms build a mathematical model based on training data, i.e., input frequency domain features and statistical parameters of time domain features according to the invention, in order to make predictions or decisions without being explicitly programmed to do so. There are various types of ML algorithms that differ in their approach, the type of data they input and output, and the type of task or problem that they are intended to solve. Illustrative, but non-limiting, examples of such ML algorithms include supervised learning algorithms, unsupervised learning algorithms, semi-supervised learning algorithms, reinforcement learning algorithms, self-learning algorithms, feature learning algorithms, sparse dictionary learning algorithms, anomaly detection algorithms, and association rule learning algorithms.

[0049] Performing machine learning involves creating a model, which is trained on training data and can then process additional data to make predictions or decisions. Various types of ML models could be used according to the embodiments, including, but not-limited to artificial neural networks, decision trees, support vector machines, regression analysis, Bayesian networks and Genetic algorithms.

[0050] Furthermore, deep learning, also known as deep structured learning, is a ML method based on artificial neural networks with representation learning. Learning can be supervised, semi-supervised or unsupervised. Deep learning architectures, such as deep neural networks, deep belief networks, recurrent neural networks and convolutional neural networks, could be used to train and implement the oxygen saturation estimation model. "Deep" in deep learning comes from the use of multiple layers in the network. Deep learning is concerned with an unbounded number of layers of bounded size, which permits practical application and optimized implementation, while retaining theoretical universality under mild conditions. In deep learning the layers are also permitted to be heterogeneous and to deviate widely from biologically informed connectionist models, for the sake of efficiency, trainability and understandability.

[0051] Hence, in an embodiment, step S15 in Fig. 2 comprises training an oxygen saturation estimation ML model, such as a random forest (RF) based oxygen saturation model. Hence, in a preferred embodiment, the oxygen saturation estimation model trained in step S15 of Fig. 2 and used in step S5 in Fig. 1 is preferably a RF-based oxygen saturation model.

[0052] Random forests or random decision forests are an ensemble learning method for classification, regression and other tasks that operates by constructing a multitude of decision trees at training time. For classification tasks, the output of the random forest is the class selected by most trees. For regression tasks, the mean or average prediction of the individual trees is returned. Random decision forests correct for decision trees’ habit of overfitting to their training set. Random forests generally outperform decision trees.

[0053] Hence, by using multiple decision trees for prediction, the RF-based oxygen saturation estimation model eliminates prediction bias that occurs if a single decision tree is used for decision making. Also, the random selection of data for training and testing reduces variance in the data that prevents overfitting.

[0054] Another advantage of using the RF algorithm is that it performs feature selection during training. Features that are most correlated with the training targets are selected by the RF algorithm using permutation scores, also referred to as importance scores herein. RF permutes feature values to estimate if the permutation deteriorates the prediction performance compared to a baseline. The features that are not correlated show no changes when the values are permutated suggesting that there is no difference between the permuted values and the original sequence of values. This suggests that the feature is a noise that does not contribute to training and can be discarded. On the other hand, the permutation of features that are correlated with the training targets results in reducing the prediction accuracy.

[0055] Fig. 5 is a flow chart illustrating additional steps of the method in Fig. 2 for selecting features to train the oxygen saturation estimation model according to an embodiment. This embodiment continues from step S14 in Fig. 2. The embodiment comprises conducting steps S41 to S44 in Fig. 5 for t = 1 to T. The parameter T represents a number of decision trees in the RF-based oxygen saturation estimation model. Step S41 of Fig. 5 comprises computing a prediction error Et = Yt - Ytfor a decision tree t. The parameter Yt represents an actual oxygen saturation value and the parameter Ytrepresents a prediction of the oxygen saturation value. Step S42 comprises selecting a feature f among the plurality of features and permuting feature values until dtf = 0. Step S43 comprises estimating a new prediction error Ett and step S44 comprises computing a difference dtf = Etf - Et. Hence, permutations for a particular feature f are performed until the difference dtf is equal to zero. At that point, the method continues to step S45, which comprises computing a mean dt and standard deviation ot over the T decision trees and computing a feature permutation score or feature importance score as If = -dtfof. This feature permutation score If is optionally compared to a threshold value Tf. Furthermore, the embodiment as shown in Fig. 5 comprises discarding the feature f if the feature permutation score It is equal to lower than the threshold value 77 in step S46. However, if the feature permutation importance It is above the threshold value 77the feature is kept in the optional step S47 and is thereby selected for usage when training the RF-based oxygen saturation estimation model. Generally, a value of the feature permutation score It close to zero indicates a low prediction ability of the particular feature f. Hence, features as determined in step S14 of Fig. 2 resulting in a feature permutation score If above zero generally have high prediction ability for usage by the RF-based oxygen saturation estimation model when predicting or estimating oxygen saturation based on video signals.

[0056] An illustrative, but non-limiting, example of a threshold value Tf that can be used according to the embodiments is 0.08.

[0057] Fig. 3 is a flow chart illustrating additional, optional steps of the methods shown in Figs. 1 and 2. In an embodiment, the methods of Figs. 1 and 2 comprise the additional step S20, which comprises recording the face of the human subject using a camera to get a recorded video signal. The face of the human subject is then detected in step S21 in the recorded video signal using a face detection algorithm outputting a bounding box covering a face region of the recorded video signal. An optional, but preferred, step S22 comprises blurring the video data within the bounding box of the recorded video signal to obtain the video signal of the face of the human subject. These steps S20 to S21 or S20 to S22 disclose an embodiment of obtaining the video signal that is used in steps S1 and S11 of Figs. 1 and 2.

[0058] The recording in step S20 can be performed by any camera, such as a camera of a smart phone, tablet or laptop. The face in the recorded video signal is then detected using a face detection algorithm, such as the Viola-Jones algorithm (Viola and Jones, Rapid Object Detection using a Boosted Cascade of Simple Features, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001. Vol. 1. IEEE Comput. Soc.), which returns a bounding box that covers the face region. Optionally, the background in the face is removed by shortening the video resolution using the bounding box coordinates. In more detail, the center pixel of the bounding box is selected. A new box is created using the number of pixels required in the Eulerian magnification algorithm to use at least 6 pyramids (2 to power 6). Hence, the new box only keeps those pixels that are required for magnification, while discarding the rest of the pixels. The optional, but preferred step S22 blurs the face within the bounding box to remove wrinkles, scars, and other skin contours. Such blurring can be performed, for instance, using a Gaussian blur filter.

[0059] In an embodiment, the methods of Figs. 1 and 2 comprise an additional step S24 as shown in Fig. 3. In an embodiment, this step S24 is additional to steps S20 to S22 discussed above. In another embodiment, this step S24 can be performed in the methods of Figs. 1 and 2 without steps S20 to S22. Step S24 comprises magnifying the colors of the video signal using a video magnification algorithm to obtain a magnified video signal. In such an embodiment, steps S1 and S11 comprise converting the magnified video signal from the initial color space to the multiple additional color spaces.

[0060] Such color magnification magnifies subtle color and motion variations in video signals in order to enhance such variations in the video signal.

[0061] In an embodiment, the methods of Figs. 1 and 2 comprise step S23 of Fig. 3, which comprises determining an illumination intensity value for a video frame of the video signal. In such an embodiment, step S24 comprises magnifying the colors of the video signal using an Eulerian video magnification algorithm using the illumination intensity value as an Eulerian alpha weight parameter to obtain the magnified video signal.

[0062] Fig. 4 illustrates an embodiment of this step S23 in Fig. 3. The method continues from step S21 or S22 in Fig. 3 or indeed from start. A next step S30 comprises converting a first video frame of the video signal from the initial color space into the HSV color space. In such an embodiment, the illumination intensity value is determined in step S31 based on the V color component values of the first video frame.

[0063] For instance, the Eulerian alpha weight parameter can be calculated using the illumination information of the first frame of the video signal. Eulerian alpha is a weight parameter in the Eulerian video magnification algorithm (Wadhwa et al., Eulerian Video Magnification and Analysis, Communications of the ACM, 60(1): 87-95, 2017) that can be tuned to adjust the strength of magnification according to the skin tone. Generally, dark skin requires a small value of alpha and bright skin requires a high value of alpha. The illumination intensity can be determined by converting the color format of the first video frame from RGB color space to HSV color space and then finding the value of the V component that represents the illumination intensity value. If V is high, e.g., 1 or close to 1 for bright skin, the magnifier value is set high and if V is low, e.g., 0 or close to 0 for dark skin, the magnifier value is set to low. Thus, the V component from the HSV color space could be used as magnifier value if in a range of 0 to 1 , or could otherwise be rescaled to be within this range. This magnifier value is then used in the Eulerian magnification algorithm to magnify skin color in the video signal.

[0064] In an embodiment, steps S1 and S11 comprise converting the video signal from an RGB color space into multiple color spaces selected from the group consisting of HSV, CEILAB, YIQ, CMYK and CIE XYZ color spaces. In various embodiment, two, three, four or indeed all five of the additional color spaces are selected when using the RGB color space as the initial color space. In an embodiment, steps S2 and S12 comprise computing, for each video frame and each color component of the initial color space and the multiple additional color spaces, multiple statistical parameters selected from the group consisting of mean of pixel values, standard deviation of pixel values, mean absolute deviation of pixel values and kurtosis of pixel values in the video frame. In various embodiments, two, three or indeed all four of the statistical parameters above are computed in steps S2 and S12.

[0065] For instance, the temporal frequency and power information in the video frames of the (magnified) video signal can be analyzed in various color spaces including RGB, HSV, CEILAB, YIQ, CMYK and CIE XYZ color spaces. First, the original RGB format of the (magnified) video signal can be converted to the other color spaces in steps S1 and S11 . Then, for each color channel, i.e., color component, of a color space, statistical parameters are computed in each video frame in step S2 or S12, such as by computing the mean, standard deviation, mean absolute deviation, and kurtosis of pixel values. A PPG signal is then generated in steps S3 and S13 using the pixel statistics (statistical parameters) of the color component over the video frames. For instance, four PPG signals (one for each of mean, standard deviation, mean absolute deviation, and kurtosis of pixel values) can generated for each color component of the color spaces resulting in a total of 76 (4 statistical parameters x 19 color components) PPG signals for the RGB, HSV, CEILAB, YIQ, CMYK and CIE XYZ color spaces.

[0066] In an embodiment, steps S3 and S13 further comprise generating a respective PPG signal presenting peak magnitude and a respective PPG signal representing peak location of histograms of hue, saturation and value in video frames in the HSV color space.

[0067] Histograms of hue (H), saturation (S), and value (V) can be used to extract additional PPG signals. For instance, the peak of the histogram is detected in each video frame and then the peak magnitude and location are determined to generate PPG signals. Three PPG signals can be generated using the peak magnitude of histograms for hue, saturation, and value respectively. Similarly, three PPG signals can be generated using the peak location for hue, saturation, and value respectively. In such an example, a total of 82 PPG signals can used for feature extraction in steps S4 and S14.

[0068] In an embodiment, steps S4 and S14 comprise extracting features using power analysis of each PPG signal. In this embodiment, features are extracted using a power analysis of the power spectrum of each PPG signal. The power analysis of PPG signals can, for instance, be performed using spectral centroid, periodic energy, jitter and shimmer, cepstral separation difference, zero crossing rate, and percentiles of power distribution.

[0069] Hence, in an embodiment, steps S4 and S14 comprise determining, for each PPG signal, multiple features selected from the group consisting of spectral centroid of the PPG signal, periodic energy of a cardiac cycle in the PPG signal, jitter of the PPG signal, shimmer of the PPG signal, rhythm of the PPG signal, rate of change in systolic peak magnitude, signal rhythm, cepstral separation difference feature, zero crossing rate, and percentiles of power distribution of the PPG signal.

[0070] In an embodiment, the features input into the oxygen saturation estimation model in step S5 and used to train the oxygen saturation estimation model in step S15 are multiple features selected from the group consisting of the mean of the spectral centroid from the PPG signal generated using the standard deviation of the Y color component of the XYZ color space, the mean of the spectral centroid from the PPG signal generated using the standard deviation of the L* color component of the CEILAB color space, the median of the periodic energy of cardiac cycles from the PPG signal generated using the hue histogram value of the HSV color space, the 70thpercentile of power distribution from the PPG signal generated using the standard deviation of the a* color component of the CEILAB color space, the mean of rhythm given from the PPG signal generated using the kurtosis of the Q color component of the YIQ color space, the standard deviation of rhythm from the PPG signal generated using the standard deviation of the red color component of the RGB color space, the median of the 25thpercent of the cardiac cycle height from the PPG signal generated using the hue histogram value of the HSV color space, the mean of the spectral centroid from the PPG signal generated using the standard deviation of the Y color component of the YIQ color space, AR of the cepstral separation difference (CSD) from the PPG signal generated using the mean of hue value of the HSV color space, 90thpercentile of power distribution calculated from the PPG signal generated using the standard deviation of the Y color component of the CMYK color space and zero crossing rate from the PPG signal generated using the mean of the Q color component of the YIQ color space.

[0071] In various embodiments, two, three, four, five, six, seven, eight, nine, ten, or indeed all eleven of the above-mentioned features are input into the oxygen saturation estimation model in step S5 and used to train the oxygen saturation estimation model in step S15. Fig. 6 is a schematic illustration of a device 100 configured to generate an oxygen saturation estimation model 150 according to an embodiment. The device 100 comprises a memory 120 configured to, at least temporarily, store sets 140 of features. The memory 120 also comprises the trained oxygen saturation estimation model 150. The device 100 in Fig. 6 has been shown with a single memory 120. The embodiments are, however, not limited thereto. In clear contrast, the device 100 could comprise or be, wirelessly or with wire, connected to multiple memories 120, such as a memory system of multiple memories. The device 100 also comprises a processor 110 configured to convert received video signals into multiple additional color spaces, compute statistical parameters, generate PPG signals, determine features and train the oxygen saturation estimation model 150 based on the input data. The device 100 further comprises a general input and output (I / O) unit 130 configured to communicate with external devices. The I / O unit 130 could represent a transmitter and receiver, or transceiver, configured to conduct wireless communication. Alternatively, or in addition, the I / O unit 130 could be configured to conduct wired communication and may then, for instance, comprise one or more input and / or output ports.

[0072] Fig. 7 is a schematic block diagram of a device 200, such as computer, comprising a processor 210 and a memory 220 that can be used to generate an oxygen saturation estimation model and / or estimate oxygen saturation using such an oxygen saturation estimation model. In such an embodiment, the training or generation and / or estimation could be implemented in a computer program 240, which is loaded into the memory 220 for execution by processing circuitry including one or more processors 210 of the device 200. The processor 210 and the memory 220 are interconnected to each other to enable normal software execution. An I / O unit 230 is preferably connected to the processor 210 and / or the memory 220 to enable reception of video signals.

[0073] The term processor should be interpreted in a general sense as any circuitry, system or device capable of executing program code or computer program instructions to perform a particular processing, determining or computing task. The processing circuitry including one or more processors 210 is, thus, configured to perform, when executing the computer program 240, well-defined processing tasks such as those described herein.

[0074] The processor 210 does not have to be dedicated to only execute the above-described steps, functions, procedure and / or blocks, but may also execute other tasks.

[0075] In an embodiment, the computer program 240 comprises instructions, which when executed by a processor 210, cause the processor 210 to convert a video signal of a face of a human subject illuminated by ambient light from an initial color space comprising multiple color components to multiple additional color spaces. Each additional color space comprises multiple color components and the video signal comprises a plurality of video frames. The processor 210 is also caused to compute, for each video frame and each color component of the initial color space and the multiple additional color spaces, statistical parameters of pixel values in the video frame. The processor 210 is further caused to generate, for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, a PPG signal representing the statistical parameter of a respective color component for the plurality of video frames. The processor 210 is additionally caused to determine features representative of a respective PPG signal and estimate oxygen saturation for the human subject based on the determined features and an oxygen saturation estimation model trained for estimating oxygen saturation based on input features.

[0076] In another embodiment, the computer program 240 comprises instructions, which when executed by a processor 210, cause the processor 210 to, for each video signal of a plurality of video signals of faces of a plurality of human subjects illuminated by ambient light, wherein each video signal comprises a plurality of video frames, convert the video signal from an initial color space comprises multiple color components to multiple additional color spaces, wherein each additional color space comprising multiple color components. The processor 210 is also caused to compute, for each video frame and each color component of the initial color space and the multiple additional color spaces, statistical parameters of pixel values in the video frame. The processor 210 is further caused to generate, for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, a PPG signal representing the statistical parameter of a respective color component for the plurality of video frames. The processor 210 is additionally caused to determine features representative of each PPG signal and a respective feature importance score for each feature and train the oxygen saturation estimation model based on features selected based on the feature importance scores and actual oxygen saturation values obtained for the plurality of human subjects.

[0077] The proposed technology also provides a non-transitory computer-readable storage medium 250 comprising the computer program 240. By way of example, the software or computer program 240 may be realized as a computer program product, which is normally carried or stored on the non-transitory computer-readable medium 250, in particular a non-volatile medium. The non-transitory computer- readable medium 250 may include one or more removable or non-removable memory devices including, but not limited to a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc (CD), a Digital Versatile Disc (DVD), a Blu-ray disc, a Universal Serial Bus (USB) memory, a Hard Disk Drive (HDD) storage device, a flash memory, a magnetic tape, or any other conventional memory device. The computer program 240 may, thus, be loaded into the operating memory 220 of the computer 200 for execution by the processor 210 thereof.

[0078] Hence, an embodiment relates to a non-transitory computer-readable medium 250 storing instructions that, when executed by a processor 210, cause the processor 210 to convert a video signal of a face of a human subject illuminated by ambient light from an initial color space comprising multiple color components to multiple additional color spaces. Each additional color space comprises multiple color components and the video signal comprises a plurality of video frames. The processor 210 is also caused to compute, for each video frame and each color component of the initial color space and the multiple additional color spaces, statistical parameters of pixel values in the video frame. The processor 210 is further caused to generate, for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, a PPG signal representing the statistical parameter of a respective color component for the plurality of video frames. The processor is additionally caused to determine features representative of a respective PPG signal and estimate oxygen saturation for the human subject based on the determined features and an oxygen saturation estimation model trained for estimating oxygen saturation based on input features.

[0079] Another embodiment relates to a non-transitory computer-readable medium 250 storing instructions that, when executed by a processor 210, cause the processor 210 to, for each video signal of a plurality of video signals of faces of a plurality of human subjects illuminated by ambient light, wherein each video signal comprises a plurality of video frames, convert the video signal from an initial color space comprising multiple color components to multiple additional color spaces, wherein each additional color space comprises multiple color components. The processor 210 is also caused to compute, for each video frame and each color component of the initial color space and the multiple additional color spaces, statistical parameters of pixel values in the video frame. The processor 210 is further caused to generate, for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, a PPG signal representing the statistical parameter of a respective color component for the plurality of video frames. The processor 210 is additionally caused to determine features representative of each PPG signal and a respective feature importance score for each feature and train the oxygen saturation estimation model based on features selected based on the feature importance scores and actual oxygen saturation values obtained for the plurality of human subjects. In an embodiment, the instructions cause the processor 210 to select features among the plurality of features to train a random forest based oxygen saturation estimation model. In such an embodiment, the processor 210 is caused to, for t = 1 to T, wherein T represents a number of decision trees in the random forest based oxygen saturation estimation model, compute a prediction error Et = Yt - Ytfor a decision tree t, wherein Yt is an actual oxygen saturation value and Ytis a prediction of the oxygen saturation value and select a feature f among the plurality of features and permuting feature values until dtt = 0; estimate a new prediction error Etf, and compute a difference dtt = Etf - Et. The processor 210 is also caused to compute a mean df and standard deviation orover the T decision trees and compute a feature permutation importance as If = -dtfof and discard the feature f if / r is equal to lower than a threshold value Tf, wherein 77 is preferably 0.08.

[0080] In an embodiment, the instructions cause the processor 210 to detect the face of the human subject in a recorded video signal using a face detection algorithm outputting a bounding box covering a face region of the recorded video signal. In an embodiment, the instructions cause the processor 210 to blur the video data within the bounding box of the recorded video signal to obtain the video signal of the face of the human subject.

[0081] In an embodiment, the instructions cause the processor 210 to record the face of the human subject using a camera to get the recorded video signal.

[0082] In an embodiment, the instructions cause the processor 210 to magnify the colors of the video signal using a video magnification algorithm to obtain a magnified video signal. The instructions also cause the processor 210 to convert the magnified video signal from the initial color space to the multiple additional color spaces.

[0083] In an embodiment, the instructions cause the processor 210 to determine an illumination intensity value for a video frame of the video signal. The instructions also cause the processor 210 to magnify the colors of the video signal using an Eulerian video magnification algorithm using the illumination intensity value as Eulerian alpha weight parameter to obtain the magnified video signal.

[0084] In an embodiment, the instructions cause the processor 210 to convert a first video frame of the video signal from the initial color space into the HSV color space and determine the illumination intensity value based on the V color component values of the first video frame. In an embodiment, the instructions cause the processor 210 to convert the video signal from an RGB color space into multiple color spaces selected from the group consisting of HSV, LAB, YIQ, CMYK and XYZ color spaces.

[0085] In an embodiment, the instructions cause the processor 210 to compute, for each video frame and each color component of the initial color space and the multiple additional color spaces, multiple statistical parameters selected from the group consisting of mean of pixel values, standard deviation of pixel values, mean absolute deviation of pixel values and kurtosis of pixel values in the video frame.

[0086] In an embodiment, the instructions cause the processor 210 to generate a respective PPG signal representing peak magnitude and a respective PPG signal representing peak location of histograms of hue, saturation and value in video frames in the HSV color space.

[0087] In an embodiment, the instructions cause the processor 210 to extract features using a power analysis of each PPG signal.

[0088] In an embodiment, the instructions cause the processor 210 to determine, for each PPG signal, multiple features selected from the group consisting of spectral centroid of the PPG signal, periodic energy of a cardiac cycle in the PPG signal, jitter of the PPG signal, shimmer of the PPG signal, rhythm of the PPG signal, rate of change in systolic peak magnitude, signal rhythm, cepstral separation difference feature, zero crossing rate, and percentiles of power distribution of the PPG signal.

[0089] The present invention also relates to a system 300 for non-contact estimation of oxygen saturation, see Fig. 8. The system 300 comprises a camera 360 configured to record a video signal of a face of a human subject illuminated by ambient light. The system 300 also comprises at least one memory 320 configured to store an oxygen saturation estimation model 350 trained for estimating oxygen saturation based on input features. The at least one memory 320 is also configured to store the video signal 340 recorded by the camera 360. The system 300 further comprises at least one processor 310. The at least one processor 310 is configured to convert the video signal 340 stored in the at least one memory 320 from an initial color space comprising multiple color components to multiple additional color spaces. Each additional color space comprises multiple color components and the video signal comprises a plurality of video frames. The at least one processor 310 is also configured to compute, for each video frame and each color component of the initial color space and the multiple additional color spaces, statistical parameters of pixel values in the video frame. The at least one processor 310 is further configured to generate, for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, a PPG signal representing the statistical parameter of a respective color component for the plurality of video frames. The at least one processor 310 is additionally configured to determine features representative of a respective PPG signal and estimate oxygen saturation for the human subject based on the determined features and the oxygen saturation estimation model 350 stored in the at least one memory 320.

[0090] The memory 320 and the at least one processor 310 may be implemented in a device 370, such as a computer, smart phone, tablet or laptop, of the system 300. This device 370 may then be connected, wirelessly or using wires, to the camera 360 using an I / O unit 330.

[0091] The camera 360 could be any camera 360 that is able to record a video signal of the face of the human subject illuminated by ambient light. The camera 360 is preferably a camera 360 capable of recording at least 100 frames per seconds, preferably at least 125 frames per seconds, such as at least 150 frames per seconds, and more preferably at least 200 frames per seconds, such as at least 250 frames per seconds or at least 300 frames per seconds. An illustrative, but non-limiting, example of a camera 360 that could be used according to the invention is a Basler MED ace camera. In an embodiment, the camera 360 is a camera 360 of the device 370, such as of a smart phone, tablet or laptop.

[0092] The various embodiments of the invention discussed in the foregoing in connection with the Figs. 1-7 also apply to the system 300 as disclosed in Fig. 8.

[0093] EXAMPLES

[0094] The method is based on training a machine learning algorithm to estimate SpO2 using features extracted from multiple PPG signals produced from a selfie video that is recorded in an ambient environment.

[0095] Data collection

[0096] Recorded human subjects were seated one meter from the camera. The recording was performed using a high-speed Basler Med-Ace camera equipped with a Sony complementary metal oxide semiconductor (CMOS) sensor connected to a computer. The camera was configured at 390 frames per second (fps) with a resolution of 640 x 480 pixels in the RGB color space. The recording duration was 15 seconds.

[0097] Face detection and blurring First, the face in the video signal was detected using the Viola-Jones algorithm (Viola and Jones, Rapid Object Detection using a Boosted Cascade of Simple Features, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001. Vol. 1. IEEE Comput. Soc.) in MATLAB that returns a bounding box that covers the face region. The background in the face region was removed by shortening the video resolution using the bounding box coordinates. In more detail, the center pixel of the bounding box was selected. A new box was created using the number of pixels required in the Eulerian magnification algorithm to use at least 6 pyramids (2 to power 6). Hence, the new box only kept those pixels that were required for magnification, while discarding the rest of the pixels. Then, the face video was blurred to remove wrinkles, scars, and other skin contours using a Gaussian blur filter.

[0098] Color magnification

[0099] Before magnifying the colors of the face in the video signal, the magnifier constant termed Eulerian alpha was calculated using the illumination information of the first frame of the video signal. Eulerian alpha is a weight parameter in the Eulerian video magnification algorithm (Wadhwa et al., Eulerian Video Magnification and Analysis, Communications of the ACM, 60(1): 87-95, 2017) that can be tuned to adjust the strength of magnification according to the skin tone. Generally, dark skin requires a small value of alpha and bright skin requires a high value of alpha. The illumination intensity was determined by converting the color format of the frame from RGB to HSV color space and then finding the value of the V component representing the illumination intensity value. If V is high, e.g., 1 or close to 1 for bright skin, the magnifier value is set high and if V is low, e.g., 0 or close to 0 for dark skin, the magnifier value is set to low. This magnifier value was then used in the Eulerian magnification algorithm in MATLAB to magnify skin color in the face of the video signal.

[0100] Conversion to color spaces

[0101] The temporal frequency and power information in the video frames of the magnified video signal were analyzed in various color spaces including RGB (Red / Green / Blue), HSV (Hue / Saturation / Value),CEI LAB (Lumination / AB: unique color coordinates), YIQ (Luminance / ln-phase / Quadrature), CMYK (Cyan / Magenta / Yellow / Black), and CIE XYZ (XZ: illuminants / Y: relative luminance). First, the original RGB format of the magnified video signal was converted to the other color spaces. Then, for each color channel of a color space, statistical parameters were computed in each video frame by computing the mean, standard deviation, mean absolute deviation, and kurtosis of pixel values. A color channel refers to a color component in a color space for example red, green, and blue are three channels of the RGB color space. A PPG signal was generated by using the pixel statistics (statistical parameters) of the color component over the video frames. In this way, four PPG signals were generated for each pixel statistic for each component of the color space resulting in a total of 76 (4 statistical parameters x 19 channels) PPG signals for all color spaces.

[0102] Further, histograms of hue (H), saturation (S), and value (V) were used to extract another six PPG signals. The peak of the histogram was detected in each video frame and then the peak magnitude and location were determined to generate PPG signals. Three PPG signals were generated using the peak magnitude of histograms for hue, saturation, and value respectively. Similarly, three PGG signals were generated using the peak location for hue, saturation, and value respectively. A total of 82 PPG signals were used for feature extraction.

[0103] Features extraction

[0104] Features were extracted using a power analysis of the power spectrum of each PPG signal of the 82 PPG signals. The power analysis of signals was performed using spectral centroid, periodic energy, jitter and shimmer, cepstral separation difference, zero crossing rate, and percentiles of power distribution as further shown in equations (1 ) to (11 ).

[0105] The spectral centroid is a measure of the center of mass or the brightness of PPG signal s. It was computed for a window i of the PPG signal s as the weighted mean of the signal frequencies determined using the Fourier transform, divided by the sum of the weights as given in equation 1 : where w[ ] is the weight or magnitude of bin number k and F[k\ is the center frequency of the bin.

[0106] The periodic energy of cardiac cycle c in PPG signal s was computed using equation 2, where n is the total number of cardiac cycles in the PPG signal s. The cardiac cycle c is defined as the portion of the PPG signal s between two valleys, which are identified in the PPG signal s using a peak finding algorithm. The rhythm in the PPG signal s was quantified using the jitter and shimmer of the PPG signal s. The jitter J was computed using equation 3, where / is the peak location of the cardiac cycle (window) i and is detected using a peak finding algorithm:

[0107] Ji — k k-1 (3)

[0108] Similarly, the shimmer was computed using equation 4, where p is the peak magnitude of the cardiac cycle (window) / :

[0109] Si = Pi - Pi-i (4)

[0110] Then, rhythm R was computed as an absolute difference between jitter and shimmer in each cardiac cycle, given in equation 5. The average R was computed to measure the signal rhythm:

[0111] Ri = \Ji - St \ (5)

[0112] Another parameter for measuring rhythm in heartbeats was computed by measuring the rate of change in the magnitude of systolic peaks. The rate of change rin systolic peak magnitude was computed using equation 6:

[0113] Then, the difference d in rate r of cardiac cycles was computed using equation 7 to measure the signal rhythm: di = rt- rj-i (7)

[0114] Cepstral separation difference (CSD) was originally developed to estimate voice depression in Parkinson’s disease. CSD was calculated as the difference between the source and filter power spectrums of the signal: ff[w] = log|E[w] | — log|H[w] | (8) where w is the log-power coefficient, R is termed as the CSD, E, and H are the source and filter log spectrums, respectively. One of the features was the average peaks of R abbreviated as AR, which estimates the power of the difference between the source and filter. AR was used as CSD feature.

[0115] Zero crossings are the number of times the PPG signal s crosses the horizontal zero axis. Zero crossing rate Z was computed using equation 9:

[0116] A power distribution over different frequencies Di(f) of PPG signal s was computed by using the power spectrum Pj( ) calculated in equations 10 and 11 :

[0117] PiC = \DFT(s,n \ (10)

[0118] Pj D (11)

[0119] Dt ) fopiW where f is the Nyquist frequency. Every 10thpercentile of the distribution was computed for analysis. Apart from the power analysis, the height of cardiac cycles in PPG signals at the 25thand 75thof the total length of the cycle was estimated and used for analysis.

[0120] Feature selection

[0121] Features from the 82 PPG signals were extracted using the equations 1 to 11 above. Feature selection was performed based on the feature permutation score, sometimes referred to as feature importance score. 11 features were the top scorers. These features included:

[0122] 1 . The mean of the spectral centroid (equation 1) from the PPG signal generated using the standard deviation of the Y channel (relative luminance) of CIE XYZ color space;

[0123] 2. The mean of the spectral centroid (equation 1) from the PPG signal generated using the standard deviation of the L* channel (luminance) in the CIELAB color space;

[0124] 3. The median of the periodic energy of cardiac cycles (equation 2) from the PPG signal generated using the hue histogram value in the HSV color space; 4. 70thpercentile of power distribution (equation 11) from the PPG signal generated using the standard deviation of the a* channel in the CIELAB color space;

[0125] 5. The mean of rhythm given (equation 5) from the PPG signal generated using the kurtosis of the Q channel (quadrature) of the YIQ color space;

[0126] 6. The standard deviation of rhythm (equation 6) from the PPG signal generated using the standard deviation of the red channel in the RGB color space;

[0127] 7. The median of the 25thpercent of the cardiac cycle height from the PPG signal generated using the hue histogram value in the HSV color space;

[0128] 8. The mean of the spectral centroid (equation 1) from the PPG signal generated using the standard deviation of the Y channel (luminance) of the YIQ color space;

[0129] 9. AR of the CSD (equation 8) from the PPG signal generated using the mean of the hue channel in the HSV color space;

[0130] 10. 90thpercentile of the power distribution (equation 11) calculated from the PPG signal generated using the standard deviation of the Y channel (yellow) CMYK color space; and

[0131] 11 . Zero crossing rate (equation 9) from the PPG signal generated using the mean of the Q channel (quadrature) in the YIQ color space.

[0132] Machine learning

[0133] The 11 selected features were extracted from two hundred samples and were used to train a random forest (RF) model. The model was validated using leave-one cross-validation. Correlation analysis of the model using Pearson correlation between the ground truth and the SPO2 value estimated by the random forest algorithm produced a coefficient value of 0.71 .

[0134] Training Random Forests for estimating oxygen saturation

[0135] Random forests (RF) are an ensemble-based method of machine learning. An RF algorithm operates by dividing the training data into random subsets and training multiple decision trees by using these subsets through a process called Bagging. Bagging splits training data in a way that two-thirds of the data that is randomly selected from the full training set is used for training a decision tree in the forests. The rest of the one-third of the data is used for testing that decision tree. The test data are termed out-of-bag (OOB) samples. An error in predicting an / *hOOB sample is computed using equation 12.

[0136] El[Y] =Yl- Yi(12) where Yl is the actual value of the OOB sample, and is the prediction of the OOB sample by an ithdecision tree. An average value of predictions produced by all the decision trees in the forests is the prediction from the model as shown in Fig. 12.

[0137] For oxygen saturation estimation, which is a regression problem, the overall performance of the RF algorithm was analyzed based on the R2coefficient computed using equation 13. where E[ Y| is the average OOB prediction error. By using multiple decision trees for prediction, the algorithm eliminates prediction bias that occurs if a single decision tree is used for decision making. Also, the random selection of data for training and testing reduces variance in the data that prevents overfitting.

[0138] Another advantage of using the RF algorithm is that it performs feature selection during training. Features that are most correlated with the training targets are selected by the RF algorithm using permutation scores. RF permutes feature values to estimate if the permutation deteriorates the prediction performance compared to a baseline. The features that are not correlated show no changes when the values are permutated suggesting that there is no difference between the permuted values and the original sequence of values. This suggests that the feature is a noise that does not contribute to training and can be discarded. On the other hand, the permutation of features that are correlated with the training targets results in reducing the prediction accuracy.

[0139] A feature’s permutation score was computed as follows: for an RF with a total of T decision trees and a total number of F features for t = 1 to T compute the baseline OOB prediction error Et for a tree f; select a feature f and permute feature values; estimate a new OOB prediction error Etf, compute the difference between the baseline and new prediction error using dtf = Etf- Et f dtf=O stop permutations; end compute mean df and standard deviation Of over T trees; feature permutation importance is computed as h= -d f, end A value of If equals or near to 0 suggests low prediction ability of feature f.

[0140] Eleven features produced permutation importance scores above 0.08. These features were selected for training the RF algorithm. The embodiments described above are to be understood as a few illustrative examples of the present invention. It will be understood by those skilled in the art that various modifications, combinations and changes may be made to the embodiments without departing from the scope of the present invention. In particular, different part solutions in the different embodiments can be combined in other configurations, where technically possible. The scope of the present invention is, however, defined by the appended claims.

Claims

CLAIMS1 . A method for non-contact estimation of oxygen saturation, the method comprising: converting (S1) a video signal of a face of a human subject illuminated by ambient light from an initial color space comprising multiple color components to multiple additional color spaces, wherein each additional color space comprises multiple color components and the video signal comprises a plurality of video frames; computing (S2), for each video frame and each color component of the initial color space and the multiple additional color spaces, statistical parameters of pixel values in the video frame; generating (S3), for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, a photoplethysmography (PPG) signal representing the statistical parameter of a respective color component for the plurality of video frames; determining (S4) features representative of a respective PPG signal; and estimating (S5) oxygen saturation for the human subject based on the determined features and an oxygen saturation estimation model trained for estimating oxygen saturation based on input features.

2. A computer-implemented method of generating an oxygen saturation estimation model, the method comprising: for each video signal of a plurality of video signals of faces of a plurality of human subjects illuminated by ambient light, wherein each video signal comprises a plurality of video frames: converting (S11) the video signal from an initial color space comprising multiple color components to multiple additional color spaces, wherein each additional color space comprising multiple color components; computing (S12), for each video frame and each color component of the initial color space and the multiple additional color spaces, statistical parameters of pixel values in the video frame; generating (S13), for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, a photoplethysmography (PPG) signal representing the statistical parameter of a respective color component for the plurality of video frames; and determining (S14) features representative of each PPG signal and a respective feature importance score for each feature; and training (S15) the oxygen saturation estimation model based on features selected based on the feature importance scores and actual oxygen saturation values obtained for the plurality of human subjects.

3. The method according to claim 1 or 2, further comprising:recording (S20) the face of the human subject using a camera to get a recorded video signal; detecting (S21) the face of the human subject in the recorded video signal using a face detection algorithm outputting a bounding box covering a face region of the recorded video signal; and blurring (S22) the video data within the bounding box of the recorded video signal to obtain the video signal of the face of the human subject.

4. The method according to any one of claims 1 to 3, further comprising magnifying (S24) the colors of the video signal using a video magnification algorithm to obtain a magnified video signal, wherein converting (S1 , S11) the video signal comprises converting (S1 , S11) the magnified video signal from the initial color space to the multiple additional color spaces.

5. The method according to claim 4, further comprising determining (S23) an illumination intensity value for a video frame of the video signal, wherein magnifying (S24) the colors of the video signal comprises magnifying (S24) the colors of the video signal using an Eulerian video magnification algorithm using the illumination intensity value as Eulerian alpha weight parameter to obtain the magnified video signal.

6. The method according to claim 5, wherein determining (S23) the illumination intensity value comprises: converting (S30) a first video frame of the video signal from the initial color space into the HSV color space; and determining (S31) the illumination intensity value based on the V color component values of the first video frame.

7. The method according to any one of claims 1 to 6, wherein converting (S1 , S11 ) the video signal comprises converting (S1, S11) the video signal from an RGB color space into multiple color spaces selected from the group consisting of HSV, CIELAB, YIQ, CMYK and CIE XYZ color spaces.

8. The method according to any one of claims 1 to 7, wherein computing (S2, S12) statistical parameters comprises computing (S2, S12), for each video frame and each color component of the initial color space and the multiple additional color spaces, multiple statistical parameters selected from the group consisting of mean of pixel values, standard deviation of pixel values, mean absolute deviation of pixel values and kurtosis of pixel values in the video frame.

9. The method according to any one of claims 1 to 8, wherein generating (S3, S13) the PPG signal further comprises generating (S3, S13) a respective PPG signal representing peak magnitude and a respective PPG signal representing peak location of histograms of hue, saturation and value in video frames in the HSV color space.

10. The method according to any one of claims 1 to 9, wherein determining (S4, S14) features comprises extracting (S4, S14) features using a power analysis of each PPG signal.

11. The method according to any one of claims 1 to 10, wherein determining features (S4, S14) comprises determining (S4, S14), for each PPG signal, multiple features selected from the group consisting of spectral centroid of the PPG signal, periodic energy of a cardiac cycle in the PPG signal, jitter of the PPG signal, shimmer of the PPG signal, rhythm of the PPG signal, rate of change in systolic peak magnitude, signal rhythm, cepstral separation difference feature, zero crossing rate, and percentiles of power distribution of the PPG signal.

12. The method according to any one of claims 1 to 11 , wherein the oxygen saturation estimation model is a random forest based oxygen saturation estimation model.

13. The method according to claim 12 when dependent on claim 2, wherein training (S15) the random forest based oxygen saturation estimation model comprises selecting features to train the random forest based oxygen saturation estimation model by: for t = 1 to 7, wherein 7 represents a number of decision trees in the random forest based oxygen saturation estimation model, computing (S41) a prediction error Et = Yt - Ytfor a decision tree t, wherein Yt is an actual oxygen saturation value and Ytis a prediction of the oxygen saturation value; selecting (S42) a feature f among the plurality of features and permuting feature values until dtf= Q estimating (S43) a new prediction error Etf, computing (S44) a difference dtf = Etf- Et computing (S45) a mean df and standard deviation or over the 7 decision trees and computing a feature permutation score as h = -dtfof, and discarding (S46) the feature f if If is equal to lower than a threshold value 77, wherein Tt is preferably 0.08.

14. A non-transitory computer-readable medium (250) storing instructions that, when executed by a processor (210), cause the processor (210) to convert a video signal of a face of a human subject illuminated by ambient light from an initial color space comprising multiple color components to multiple additional color spaces, wherein each additional color space comprises multiple color components and the video signal comprises a plurality of video frames; compute, for each video frame and each color component of the initial color space and the multiple additional color spaces, statistical parameters of pixel values in the video frame; generate, for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, a photoplethysmography (PPG) signal representing the statistical parameter of a respective color component for the plurality of video frames; determine features representative of a respective PPG signal; and estimate oxygen saturation for the human subject based on the determined features and an oxygen saturation estimation model trained for estimating oxygen saturation based on input features.

15. A non-transitory computer-readable medium (250) storing instructions that, when executed by a processor (210), cause the processor (210) to for each video signal of a plurality of video signals of faces of a plurality of human subjects illuminated by ambient light, wherein each video signal comprises a plurality of video frames: convert the video signal from an initial color space comprising multiple color components to multiple additional color spaces, wherein each additional color space comprising multiple color components; compute, for each video frame and each color component of the initial color space and the multiple additional color spaces, statistical parameters of pixel values in the video frame; generate, for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, a photoplethysmography (PPG) signal representing the statistical parameter of a respective color component for the plurality of video frames; and determine features representative of each PPG signal and a respective feature importance score for each feature; and train the oxygen saturation estimation model based on features selected based on the feature importance scores and actual oxygen saturation values obtained for the plurality of human subjects.

16. The non-transitory computer-readable medium according to claim 15, wherein the non-transitory computer-readable medium (250) stores instructions that, when executed by the processor (210), causethe processor (210) to select features among the plurality of features to train a random forest based oxygen saturation estimation model by: for t = 1 to T, wherein T represents a number of decision trees in the random forest based oxygen saturation estimation model, compute a prediction error Et = Yt - Ytfor a decision tree t, wherein Yt is an actual oxygen saturation value and Ytis a prediction of the oxygen saturation value; select a feature f among the plurality of features and permuting feature values until dtt = 0; estimate a new prediction error Etr, compute a difference dtr = Ett- Et compute a mean dt and standard deviation or over the T decision trees and compute a feature permutation importance as h= -dtfof, and discard the feature f if / r is equal to lower than a threshold value Tf, wherein 77 is preferably 0.08.

17. The non-transitory computer-readable medium according to any one of claims 14 to 16, wherein the non-transitory computer-readable medium (250) stores instructions that, when executed by the processor (210), cause the processor (210) to: detect the face of the human subject in a recorded video signal using a face detection algorithm outputting a bounding box covering a face region of the recorded video signal; and blur the video data within the bounding box of the recorded video signal to obtain the video signal of the face of the human subject.

18. The non-transitory computer-readable medium according to any one of claims 14 to 17, wherein the non-transitory computer-readable medium (250) stores instructions that, when executed by the processor (210), cause the processor (210) to record the face of the human subject using a camera to get the recorded video signal.

19. The non-transitory computer-readable medium according to any one of claims 14 to 18, wherein the non-transitory computer-readable medium (250) stores instructions that, when executed by the processor (210), cause the processor (210) to: magnify the colors of the video signal using a video magnification algorithm to obtain a magnified video signal; and convert the magnified video signal from the initial color space to the multiple additional color spaces.

20. The non-transitory computer-readable medium according to claim 19, wherein the non-transitory computer-readable medium (250) stores instructions that, when executed by the processor (210), cause the processor (210) to: determine an illumination intensity value for a video frame of the video signal; magnify the colors of the video signal using an Eulerian video magnification algorithm using the illumination intensity value as Eulerian alpha weight parameter to obtain the magnified video signal.21 . The non-transitory computer-readable medium according to claim 20, wherein the non-transitory computer-readable medium (250) stores instructions that, when executed by the processor (210), cause the processor (210) to: convert a first video frame of the video signal from the initial color space into the HSV color space; and determine the illumination intensity value based on the V color component values of the first video frame.

22. The non-transitory computer-readable medium according to any one of claims 14 to 21 , wherein the non-transitory computer-readable medium (250) stores instructions that, when executed by the processor (210), cause the processor (210) to convert the video signal from an RGB color space into multiple color spaces selected from the group consisting of HSV, LAB, YIQ, CMYK and XYZ color spaces.

23. The non-transitory computer-readable medium according to any one of claims 14 to 22, wherein the non-transitory computer-readable medium (250) stores instructions that, when executed by the processor (210), cause the processor (210) to compute, for each video frame and each color component of the initial color space and the multiple additional color spaces, multiple statistical parameters selected from the group consisting of mean of pixel values, standard deviation of pixel values, mean absolute deviation of pixel values and kurtosis of pixel values in the video frame.

24. The non-transitory computer-readable medium according to any one of claims 14 to 23, wherein the non-transitory computer-readable medium (250) stores instructions that, when executed by the processor (210), cause the processor (210) to generate a respective PPG signal representing peak magnitude and a respective PPG signal representing peak location of histograms of hue, saturation and value in video frames in the HSV color space.

25. The non-transitory computer-readable medium according to any one of claims 14 to 24, wherein the non-transitory computer-readable medium (250) stores instructions that, when executed by the processor (210), cause the processor (210) to extract features using a power analysis of each PPG signal.

26. The non-transitory computer-readable medium according to any one of claims 14 to 25, wherein the non-transitory computer-readable medium (250) stores instructions that, when executed by the processor (210), cause the processor (210) to determine, for each PPG signal, multiple features selected from the group consisting of spectral centroid of the PPG signal, periodic energy of a cardiac cycle in the PPG signal, jitter of the PPG signal, shimmer of the PPG signal, rhythm of the PPG signal, rate of change in systolic peak magnitude, signal rhythm, cepstral separation difference feature, zero crossing rate, and percentiles of power distribution of the PPG signal.

27. A system (300) for non-contact estimation of oxygen saturation, the system (300) comprising: a camera (360) configured to record a video signal of a face of a human subject illuminated by ambient light; at least one memory (320) configured to store: an oxygen saturation estimation model (350) trained for estimating oxygen saturation based on input features; and the video signal (340) recorded by the camera (360); and at least one processor (310) configured to: convert the video signal (340) stored in the at least one memory (320) from an initial color space comprising multiple color components to multiple additional color spaces, wherein each additional color space comprises multiple color components and the video signal comprises a plurality of video frames; compute, for each video frame and each color component of the initial color space and the multiple additional color spaces, statistical parameters of pixel values in the video frame; generate, for each statistical parameter and for each color component of the initial color space and the multiple additional color spaces, a photoplethysmography (PPG) signal representing the statistical parameter of a respective color component for the plurality of video frames; determine features representative of a respective PPG signal; and estimate oxygen saturation for the human subject based on the determined features and the oxygen saturation estimation model (350) stored in the at least one memory (320).

28. The system according to claim 27, wherein the at least one processor (310) is configured to: detect the face of the human subject in a recorded video signal using a face detection algorithm outputting a bounding box covering a face region of the recorded video signal; and blur the video data within the bounding box of the recorded video signal to obtain the video signal of the face of the human subject.

29. The system according to claim 27 or 28, wherein the at least one processor (310) is configured to: magnify the colors of the video signal using a video magnification algorithm to obtain a magnified video signal; and convert the magnified video signal from the initial color space to the multiple additional color spaces.

30. The system according to claim 29, wherein the at least one processor (310) is configured to: determine an illumination intensity value for a video frame of the video signal; magnify the colors of the video signal using an Eulerian video magnification algorithm using the illumination intensity value as Eulerian alpha weight parameter to obtain the magnified video signal.31 . The system according to any one of claims 27 to 30, wherein the at least one processor (310) is configured to: convert a first video frame of the video signal from the initial color space into the HSV color space; and determine the illumination intensity value based on the V color component values of the first video frame.

32. The system according to any one of claim 27 to 31 , wherein the at least one processor (310) is configured to convert the video signal from an RGB color space into multiple color spaces selected from the group consisting of HSV, LAB, YIQ, CMYK and XYZ color spaces.

33. The system according to any one of claim 27 to 32, wherein the at least one processor (310) is configured to compute, for each video frame and each color component of the initial color space and the multiple additional color spaces, multiple statistical parameters selected from the group consisting of mean of pixel values, standard deviation of pixel values, mean absolute deviation of pixel values and kurtosis of pixel values in the video frame.

34. The system according to any one of claim 27 to 33, wherein the at least one processor (310) is configured to generate a respective PPG signal representing peak magnitude and a respective PPG signal representing peak location of histograms of hue, saturation and value in video frames in the HSV color space.

35. The system according to any one of claims 27 to 34, wherein the at least one processor (310) is configured to extract features using a power analysis of each PPG signal.

36. The system according to any one of claims 27 to 35, wherein the at least one processor (310) is configured to determine, for each PPG signal, multiple features selected from the group consisting of spectral centroid of the PPG signal, periodic energy of a cardiac cycle in the PPG signal, jitter of the PPG signal, shimmer of the PPG signal, rhythm of the PPG signal, rate of change in systolic peak magnitude, signal rhythm, cepstral separation difference feature, zero crossing rate, and percentiles of power distribution of the PPG signal.

Images