Camera-based remote blood oxygen saturation monitoring system and method

Abstract

The invention provides a system. The system comprises a camera, a color image and signal processing system, a physiological activity image processing system, an oxyhemoglobin saturation estimator and a report output module. The camera captures a color image frame of a subject. A color image and signal processing system extracts physiological signals by analyzing light intensity, detecting and resizing facial images, inserting facial landmarks, creating facial tiles, selecting tiles according to the landmarks, and extracting signals from different color channels. The physiological activity image processing system generates a PAI in which physiological information of a subject is embedded by forming one-dimensional signals, serially connecting the filtered one-dimensional signals into a two-dimensional image of each color channel and combining the two-dimensional images along the dimensions of the color channels. An oxyhemoglobin saturation estimator receives the PAI using a deep learning model to estimate an oxyhemoglobin saturation level of the subject. The report output module provides a readable estimation report.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and is used to extract physiologically relevant information from images and then estimate the blood oxygen saturation of a subject using a deep learning / machine learning model. Background Art

[0002] Human vital signs, such as blood oxygen saturation (SpO2), heart rate, respiratory rate, blood pressure and body temperature, are standard parameters used to assess human health. Specifically, SpO2 readings reflect whether the human body is taking in enough oxygen to maintain normal activities. SpO2 readings are commonly used indicators in trauma management and early detection of diseases such as hypoxemia, sleep apnea and heart disease. Currently, pulse oximeters are commonly used for non-invasive measurement of SpO2. However, this contact-based device has limited availability and is not suitable for long-term monitoring. If used for a long time, it may cause discomfort and is not suitable for people with sensitive skin. In addition, the use of contact-based devices for health monitoring may cause the spread of infectious diseases. Therefore, there is a demand in the relevant market for non-contact SpO2 measurement methods.

[0003] Over the past decade, a variety of contactless SpO2 measurement methods have been proposed. Researchers have used various types of cameras, ranging from infrared cameras and high-quality monochrome cameras equipped with special filters to commercially available webcams, to estimate SpO2 by capturing subtle changes in facial light intensity and combining them with ratio-of-ratio analysis principles. However, remote SpO2 measurement based on deep learning is still in its infancy. In contrast, deep learning technology has demonstrated leading performance in detecting other vital signs such as heart rate.

[0004] Therefore, a non-contact SpO2 measurement system is needed that can use deep learning technology to improve accuracy and provide a more comfortable and hygienic solution for continuous health monitoring. Summary of the Invention

[0005] Based on the above, there are still technical problems in extracting physiological related information from images. The purpose of the present invention is to provide a system and method to solve these problems.

[0006] In general, to extract a subject's SpO2, the system first captures a color video of the subject. Computer vision techniques are then applied to the video frames to locate facial regions of interest (ROIs) and track these regions continuously for a period of time. Next, the images of the located ROIs are fed into a processing pipeline with image and signal processing algorithms to create a Physiological Activity Image (PAI). This PAI, embedded with the subject's physiological information, is then fed into a deep learning model to predict the subject's SpO2 percentage (%).

[0007] According to a first aspect of the present invention, a remote blood oxygen saturation monitoring system based on deep learning is provided. The system includes a camera, a color image and signal processing system, a physiological activity image processing system, a blood oxygen saturation estimator, and a report output module. The camera is used to capture color image frames of a subject. The color image and signal processing system is used to extract physiological signals based on the color image frames of the subject and is used to: perform light intensity analysis on the color image frames of the subject; identify the positions of the face and facial landmarks within the color image frames to detect facial images; resize the detected facial images and interpolate and calculate approximate facial landmarks; create facial blocks based on the resized facial images; select the facial blocks based on the interpolated facial landmarks; and set different color channels and then extract the physiological signals from the selected facial blocks. The physiological activity image processing system is used to generate a physiological activity image (PAI) using the extracted physiological signals. The PAI embeds the subject's physiological information and is used to: form 1D (one-dimensional) signals from the extracted physiological signals and apply bandpass filtering to each 1D signal; sequentially concatenate the filtered 1D signals of the same color channel to create a 2D (two-dimensional) image for each color channel; and concatenate the 2D images of different color channels along the dimension of the color channel to generate the PAI. A blood oxygen saturation estimator is configured to receive the PAI and perform inference to obtain an estimated blood oxygen saturation level of the subject, wherein the blood oxygen saturation estimator has a deep learning model for estimating SpO2. A report output module is configured to receive the inference result of the estimated blood oxygen saturation and, based on the inference result, provide a readable report on the subject's blood oxygen saturation status.

[0008] According to a second aspect of the present invention, a remote blood oxygen saturation monitoring method based on deep learning is provided. The method comprises the following steps: using a camera to capture a color image frame of a subject; using a color image and signal processing system to extract physiological signals based on the color image frame of the subject, which comprises the following steps: (1) performing light intensity analysis on the color image frame of the subject; (2) identifying the position of the face and facial landmarks within the color image frame to detect a facial image; (3) adjusting the size of the detected facial image and interpolating and calculating approximate facial landmarks; (4) creating a facial block based on the resized facial image; (5) selecting the facial block based on the interpolated facial landmarks; and (6) setting different color channels and then extracting the physiological signal from the selected facial block. The method also includes: using a physiological activity image processing system to generate a physiological activity image (PAI) using the extracted physiological signals, which embeds the physiological information of the subject, including the following steps: (1) forming a 1D (one-dimensional) signal from the extracted physiological signals and applying bandpass filtering to each of the 1D signals; (2) sequentially concatenating the filtered 1D signals of the same color channel to create a 2D (two-dimensional) image for each color channel; and (3) concatenating the 2D images of different color channels along the dimension of the color channel to generate the PAI. The method also includes: receiving the PAI for inference by a blood oxygen saturation estimator to obtain an estimated blood oxygen saturation level of the subject, wherein the blood oxygen saturation estimator has a deep learning model for estimating SpO2; and receiving the inference result of the estimated blood oxygen saturation and using a report output model to provide a readable report on the blood oxygen saturation status of the subject based on the inference result.

[0009] Through the above configuration, the technical aspects of the present invention provide a non-contact method for SpO2 estimation. By capturing a subject's facial image and utilizing PAI, the solution provided by the present invention enables direct, non-contact SpO2 estimation. This non-contact method can be used with common digital imaging devices (such as webcams or smartphone cameras), demonstrating high potential for integration into telemedicine services, health programs, clinical applications, and even insurance assessments, thereby improving accessibility and efficiency in healthcare and related industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The embodiments of the present invention are described in more detail below with reference to the accompanying drawings, in which:

[0011] Figure 1 A schematic diagram showing the architecture of a remote blood oxygen saturation monitoring system based on deep learning according to an embodiment of the present invention is shown;

[0012] Figure 2 A schematic diagram showing a process for estimating blood oxygen saturation of a subject according to an embodiment of the present invention;

[0013] Figure 3 A schematic diagram showing a process of converting an original color frame into a physiological signal using a color image and signal processing system according to an embodiment of the present invention;

[0014] Figure 4 Shows the absorption coefficient of oxygenated hemoglobin and deoxygenated hemoglobin at different wavelengths;

[0015] Figure 5 A schematic diagram showing forming a PAI from extracted physiological signals by using a physiological activity image processing system according to an embodiment of the present invention;

[0016] Figure 6 A flowchart showing forming a physiological activity image from a captured video of a subject according to an embodiment of the present invention; and

[0017] Figure 7 A flowchart is shown for training a deep learning model for estimating blood oxygen saturation using captured subject videos according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In the following description, a camera-based remote blood oxygen saturation monitoring system and method are used as preferred examples for illustration. Those skilled in the art will appreciate that modifications, including additions and / or substitutions, may be made without departing from the scope and spirit of the present invention. Specific details may be omitted to avoid obscuring the present invention; however, this disclosure is written to enable those skilled in the art to practice the teachings herein without undue experimentation.

[0019] Figure 1 The following is a schematic diagram illustrating the architecture of a remote blood oxygen saturation monitoring system 100 based on deep learning according to an embodiment of the present invention. The system 100 includes a camera 110, a color image and signal processing system 120, a physiological activity image processing system 130, a blood oxygen saturation estimator 140, and a report output module 150.

[0020] Camera 110 is configured to capture color image frames of a subject to be analyzed. The term "subject," as used herein, refers to a person undergoing blood oxygen saturation monitoring. Camera 110 serves as the primary input device for system 100, providing raw visual data for subsequent processing steps. In some embodiments, camera 110 can record one or more color videos containing color image frames of the subject.

[0021] The color image and signal processing system 120 communicates with the camera 110, wherein the color image frames captured by the camera 110 can serve as an input source for the color image and signal processing system 120. The color image and signal processing system 120 is configured to extract physiological signals based on the color image frames of the subject. Specifically, in the process of processing the color image frames, the color image and signal processing system 120 can perform several key functions. The color image and signal processing system 120 is configured to: perform light intensity analysis on the color image frames of the subject and then identify the location of the face and facial landmarks; resize the detected facial image and interpolate to calculate approximate facial landmarks; create facial blocks in the resized facial image; select a specific facial block based on the interpolated facial landmarks; and extract physiological signals from the selected facial block to provide necessary data for subsequent processing stages.

[0022] The physiological activity image processing system 130 communicates with the color image and signal processing system 120, wherein after the physiological signals are extracted, they are transmitted from the color image and signal processing system 120 to the physiological activity image processing system 130. The physiological activity image processing system 130 is configured to generate a physiological activity image (PAI) using the extracted physiological signals, which embeds the physiological information of the subject. Specifically, during the generation of PAI, the physiological activity image processing system 130 can perform several key functions. The physiological activity image processing system 130 is configured to: form a 1D (one-dimensional) signal from the extracted physiological signals and apply bandpass filtering to each 1D signal; concatenate the filtered 1D signals of the same color channel in sequence to create a 2D (two-dimensional) image; and concatenate 2D images of different color channels along the color channel dimension to generate PAI.

[0023] The blood oxygen saturation estimator 140 communicates with the physiological activity image processing system 130 to receive the PAI. The PAI can be input into the blood oxygen saturation estimator 140 for inference / deduction / inference to obtain an estimated blood oxygen saturation level of the subject. Specifically, the blood oxygen saturation estimator 140 includes a deep learning model configured for SpO2 estimation. To this end, the deep learning model can be trained using an SpO2 training set containing processed PAI, and further SpO2 estimation can be performed. Once the deep learning model is trained, the blood oxygen saturation estimator 140 is configured to use the trained model to estimate the subject's SpO2 based on the PAI input from the physiological activity image processing system 130.

[0024] The report output module 150 communicates with the blood oxygen saturation estimator 140 to receive the inference / calculation results with the estimated blood oxygen saturation. The report output module 150 is configured to compile and present the results in a user-friendly format, providing a readable report on the subject's blood oxygen saturation status.

[0025] Figure 2 A schematic diagram of a process for estimating the blood oxygen saturation of a subject according to an embodiment of the present invention is shown. Figure 2 The process shown can be used as described above Figure 1 The estimation process includes steps S201, S202, S203, S204, S205, and S206, which are used to estimate the SpO2 of the subject.

[0026] The estimation process may begin with step S201, which involves capturing a color video of the subject. In this step, the camera 110 is first activated to record the color video of the subject, thereby obtaining an original color frame as a basic input source for the subsequent physiological signal extraction stage.

[0027] In step S202, the raw color frame captured by camera 110 is transmitted to color image and signal processing system 120 for feature extraction and region recognition. Color image and signal processing system 120 locates and tracks pixels on the subject's face and identifies regions of interest (ROIs) within the raw color frame.

[0028] Once the ROI of the subject's face is determined, the process proceeds to step S203. Color image and signal processing system 120 extracts physiological signals from the ROI based on the color image frames and records these signals. At this stage, color image and signal processing system 120 converts the raw color frames into physiological signals, which helps minimize noise in the frames and provides higher-quality data for physiological signal extraction.

[0029] Next, the process proceeds to step S204. In step S204, the extracted physiological signals are transmitted from the color image and signal processing system 120 to the physiological activity image processing system 130. The physiological activity image processing system 130 processes the physiological signals to form a physiological activity image (PAI) that embeds the subject's physiological information and is used as a single 2D input source.

[0030] After the PAI is formed / generated, the process proceeds to step S205. The PAI is input into the oxygen saturation estimator 140. This allows SpO2 estimation by feeding the 2D input source into the oxygen saturation estimator 140 and developing a model prediction. The oxygen saturation estimator 140 can use the PAI to infer / calculate / infer oxygen saturation, thereby predicting the subject's SpO2 level. This stage utilizes deep learning techniques, and the oxygen saturation estimator 140 can apply a trained model to achieve this step.

[0031] Finally, in step S206, the SpO2 level estimated by the blood oxygen saturation estimator 140 is output to the report output module 150. The report output module 150 can generate and transmit a report accordingly, completing the process of estimating the SpO2 of the subject.

[0032] The following describes in detail how each component processes data.

[0033] Figure 3 A schematic diagram of a process of converting an original color frame into a physiological signal using a color image and signal processing system according to an embodiment of the present invention is shown. Figure 3 The process includes steps S301, S302, S303, S304, and S305, which is aimed at signal conversion.

[0034] Once the color image and signal processing system 120 receives the subject's raw color frame from the camera 110, the process may begin at step S301. In this step, the color image and signal processing system 120 applies face detection and facial landmark detection to the raw color frame, thereby obtaining a detected facial image having a ROI of a facial representation. This is done because the ROI on the facial representation will be used in the next step of extracting physiological signals. In the facial representation, the ROI is located by a computer vision algorithm that approximates the facial landmarks on the subject's facial representation. These facial landmarks are descriptors of the locations of facial features (e.g., nose, eyes, mouth, etc.).

[0035] Color image and signal processing system 120 can use any type of computer vision technology to perform face detection and facial landmark detection to identify the subject's face and the locations of facial landmarks. In some embodiments, color image and signal processing system 120 includes a face detection model (e.g., a face detection model based on a pre-trained convolutional neural network (CNN)) and a facial landmark predictor for face detection.

[0036] During face detection and facial landmark detection, the raw color frames are first processed using a face detection model, which detects the bounding box of the facial representation within the raw color frame. Once the facial representation is localized, the facial landmark predictor applies a detection algorithm to identify key facial points, such as the area around the eyes, the tip of the nose, and the area near the mouth, and determines facial landmarks to define ROIs on the facial representation for subsequent analysis.

[0037] The process proceeds to step S302. The color image and signal processing system 120 processes the detected facial image by resizing it to a specific size through interpolation, thereby obtaining a resized facial image. To maintain consistency, the color image and signal processing system 120 also interpolates the corresponding estimated facial landmarks so that the facial landmarks remain aligned with the resized facial image.

[0038] In some embodiments, the color image and signal processing system 120 may use various interpolation techniques to resize the detected facial image to any specific size. For example, the color image and signal processing system 120 may include an interpolation model that may use bilinear interpolation or bicubic interpolation to achieve smooth scaling while maintaining the visual quality of the facial image.

[0039] In some embodiments, the color image and signal processing system 120 may interpolate the approximated facial landmarks using any suitable interpolation technique. For example, the interpolation model of the color image and signal processing system 120 may employ linear interpolation or a spline-based approach to adjust the spatial positions of the facial landmarks relative to the resized facial image so that the landmarks correctly correspond to the resized facial dimensions.

[0040] The process proceeds to step S303. The color image and signal processing system 120 performs facial block segmentation on the resized facial image. Each resized facial image can be segmented into multiple facial blocks, and all facial blocks have the same size (for example, have the same pixel size). The facial blocks are labeled in sequence according to their positions. In addition, facial blocks from different color frames but occupying the same position are assigned the same label, which is irrelevant to which frame they belong to. For example, if there are two color frames, assuming they are named first color frame and second color frame; facial blocks at the same position in the first color frame and the second color frame can share the same label to express that they are located at the same position.

[0041] In some embodiments, the color image and signal processing system 120 creates facial blocks from the detected facial image by segmenting the detected facial image into facial blocks of any specific size, for example, each facial block is a square block (e.g., 32×32 pixels or 16×16 pixels).

[0042] The process then proceeds to step S304. The color image and signal processing system 120 selects facial blocks near any pre-defined ROIs based on the interpolated facial landmarks. In this case, the color image and signal processing system 120 performs screening by selecting facial blocks near any pre-defined ROIs, where the ROIs are defined based on the interpolated facial landmarks created in step S301. These pre-defined ROIs may include areas around the eyes, the tip of the nose, or areas near the mouth, as these areas typically provide more robust physiological signals. Therefore, facial blocks corresponding to these areas may be selected to enhance the performance of the extraction process.

[0043] In some embodiments, for naturally paired facial features, such as the left and right eyes or the left and right cheeks, a pair of facial regions may be selected, with each facial region encompassing one of the paired features. For example, two facial regions may be predefined, one encompassing the left eye and the other encompassing the right eye, to facilitate symmetry or difference analysis of the paired features.

[0044] The process then proceeds to the final step, step S305. For each selected facial patch, the color image and signal processing system 120 extracts the average pixel intensity values ​​for each color channel (e.g., red, green, and blue) within the facial patch and records these values. Once the average pixel intensity values ​​for all facial patches in all required color channels have been extracted, the color image and signal processing system 120 packages the average pixel intensity values ​​into physiological signals, completing the physiological signal extraction process. In other words, the average pixel intensity values ​​are used to construct the physiological signals.

[0045] The average pixel intensity value of the physiological signal collected by the color image and signal processing system 120 in steps S301 - S305 is an important factor in estimating SpO 2 .

[0046] Figure 4 Shows the absorption coefficients of oxyhemoglobin and deoxyhemoglobin at different wavelengths. Figure 4Commercial pulse oximeters are equipped with light-emitting diodes that generate two different wavelengths of light: 660 nm (red) and 960 nm (infrared) to measure the different absorption coefficients of oxyhemoglobin (HbO2) and deoxyhemoglobin (Hb). A photodetector within the pulse oximeter analyzes the absorption of light at these two wavelengths and generates an absorbance, from which SpO2 (as a percentage) can be determined. However, since infrared wavelengths cannot be captured in color video, blue wavelengths can be used instead of the infrared component. This is because the absorption coefficients of HbO2 and Hb at these two wavelengths are very similar.

[0047] Figure 5 A schematic diagram illustrating forming a PAI from extracted physiological signals using a physiological activity image processing system according to an embodiment of the present invention is shown. The process of forming a PAI includes steps S501, S502, S503, S504, and S505.

[0048] Once the physiological activity image processing system 130 receives the extracted physiological signal from the color image and signal processing system 120 , the process starts from step S501 . Based on the physiological signal, the physiological activity image processing system 130 may use the average pixel intensity values ​​extracted from different color channels of the facial region as input.

[0049] The process then moves to step S502. For each color channel, the physiological activity image processing system 130 extracts the average pixel intensity value from facial blocks with the same label, thereby forming a 1D signal for each facial block's label. Therefore, the number of 1D signals corresponds to the number of labels assigned to the facial blocks after segmenting the detected facial image in step S303. For example, if a facial image is segmented into N facial blocks and thus assigned N labels, the number of 1D signals is A*N, where A is related to the number of color channels; and A and N are positive integers.

[0050] The process then proceeds to step S503. The physiological activity image processing system 130 includes a bandpass filter, and the 1D signal is further processed through the bandpass filter to eliminate noise while retaining physiologically relevant information. In some embodiments, the bandpass filtering performed by the physiological activity image processing system 130 can utilize a bandpass filter with arbitrary order, low cutoff frequency, and high cutoff frequency to reduce noise while retaining physiologically relevant information.

[0051] The process goes to step S504. The physiological activity image processing system 130 has a concatenator for forming a 2D (two-dimensional) image. Specifically, for each color channel (for example, red, green, or blue), the concatenator concatenates the 1D signals corresponding to each facial block label in sequence to generate a 2D image. Since different color channels are applied, multiple 2D images are formed. If the number of color channels is B, the number of 2D images will also be B. In some embodiments, the physiological activity image processing system 130 concatenates the filtered 1D signals of the same color channel in sequence to form a 2D image, and involves arranging these 1D signals in an arbitrary order, which are placed adjacent to each other along the spatial dimension to construct a 2D image.

[0052] The process then proceeds to the final step, step S505. The 2D images of different color channels are further concatenated along the color channel dimension by a concatenator to form a physiological activity image. This concatenation allows for the formation of a multi-channel physiological activity image, where each channel corresponds to a specific color. In some embodiments, the physiological activity image processing system 130 concatenates the 2D images of different color channels along the color channel dimension to form a PAI. This involves arranging these 2D images in any order along the color channel axis to create the final PAI.

[0053] Figure 6 A flow chart of forming a physiological activity image from a captured video of a subject according to an embodiment of the present invention is shown. Figure 6 Specific intermediate products and final products of steps S301 to S305 and S501 to S505 are shown.

[0054] Facial videos serve as input to the process of forming PAI. The process begins by applying face detection and facial landmark detection to the captured video of the subject. For each video frame, the detected facial image is resized to a specific size through interpolation, and facial landmarks are estimated by interpolation relative to the resized facial image. Facial patch segmentation is then performed on the resized facial image. Each facial patch has the same size and is labeled sequentially based on its position. Furthermore, facial patches located at the same location but from different frames will have the same label.

[0055] If these facial blocks are close to any pre-set ROI where the interpolated facial landmarks are located, these facial blocks are further selected.

[0056] From these selected facial patches, a 1D signal is formed based on the average of the pixel intensity values ​​across the different color channels extracted for each facial patch label. For each color channel, the 1D signals for each facial patch label are sequentially concatenated to produce a 2D image. Finally, the PAI is formed by concatenating the 2D images for each color channel along the color channel dimension.

[0057] Figure 6 The flow chart in provides a physical and concrete representation of the solution proposed by the present invention.

[0058] From the above description, using the provided process, a PAI data point can be obtained for each subject. A PAI dataset can be acquired for a specific group of subjects. Furthermore, a collection method (not limited to a specific method, such as a contact-based method) can be used to obtain SpO2 baseline ground truth values ​​for the specific group of subjects. In this manner, the prepared PAI dataset and the corresponding SpO2 baseline ground truth dataset can be input into the blood oxygen saturation estimator 140 for training.

[0059] Figure 7 A flowchart of training a deep learning model for estimating blood oxygen saturation using captured subject videos according to an embodiment of the present invention is shown.

[0060] exist Figure 7 In the present embodiment, before using PAI to train a deep learning model for SpO2 estimation, stages (a), (b), and (c) are performed. Stage (a) involves capturing / acquiring facial images using camera 110; stage (b) involves extracting physiological signals from facial videos using color image and signal processing system 120; and stage (c) involves generating one or more PAIs using the extracted physiological signals from physiological activity graph processing system 130.

[0061] After stage (c) and before training, a training set containing the corresponding SpO2 baseline true values ​​and processed PAI of the same group of subjects may be prepared.

[0062] During the training process of the deep learning model, at stage (d), the processed PAI and SpO2 true values ​​are input into the deep learning model of the blood oxygen saturation estimator 140. The deep learning model can automatically extract features from the PAI and learn and form a functional relationship (e.g., a loss function) to map the PAI to the estimated SpO2. The deep learning model is also configured to use the provided SpO2 true values ​​to guide itself to improve the proposed function. Specifically, the deep learning model uses a loss function (e.g., mean squared error) to calculate the difference between its predicted SpO2 estimate and the true value. The calculation of the difference can be performed using an iterative process, allowing the deep learning model to learn and improve its performance over time.

[0063] Once the model is trained, the blood oxygen saturation estimator 140 can use the trained deep learning model to estimate the subject's SpO2. After training is complete, as in stage (e), the camera 110, the color image and signal processing system 120, and the physiological activity image processing system 130 are utilized to process and generate the subject's PAI, which is then input into the blood oxygen saturation estimator 140. The trained model of the blood oxygen saturation estimator 140 can then be used to obtain an estimated blood oxygen saturation level for any given subject.

[0064] The functional units and modules of the apparatus and methods according to the embodiments disclosed herein can be implemented using computing devices, computer processors, or electronic circuits, including but not limited to application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, and other programmable logic devices configured or programmed according to the teachings of the present disclosure. Those skilled in the art of software or electronics can easily prepare computer instructions or software codes to be executed in computing devices, computer processors, or programmable logic devices based on the teachings of the present disclosure.

[0065] All or part of the methods according to the embodiments may be executed in one or more computing devices, including server computers, personal computers, laptop computers, and mobile computing devices (such as smartphones and tablet computers).

[0066] Embodiments may include computer storage media, transient and non-transitory memory devices having computer instructions or software code stored therein, which may be used to program or configure a computing device, computer processor, or electronic circuit to perform any of the processes of the present invention. Storage media, transient and non-transitory memory devices may include, but are not limited to, floppy disks, optical disks, Blu-ray disks, DVDs, CD-ROMs, magneto-optical disks, ROMs, RAMs, flash memory devices, or any type of medium or device suitable for storing instructions, code, and / or data.

[0067] Each functional unit and module according to various embodiments may also be implemented in a distributed computing environment and / or a cloud computing environment, where all or part of the machine instructions are executed in a distributed manner by one or more processing devices that are interconnected by a communication network, such as an intranet, a wide area network (WAN), a local area network (LAN), the Internet, and other forms of data transmission media.

[0068] The foregoing description of the present invention is provided for the purpose of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations will be apparent to those skilled in the art.

[0069] The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to understand the invention for various embodiments and with various modifications as are suited to the particular use contemplated.

Claims

1. A remote blood oxygen saturation monitoring system based on deep learning, characterized in that: include: a camera for capturing color image frames of the subject; A color image and signal processing system for extracting physiological signals from the color image frames of the subject and for: performing a light intensity analysis on the color image frames of the subject; identifying the locations of faces and facial landmarks within the color image frame to detect facial images; resizing the detected facial image and calculating approximate facial landmarks by interpolation; creating a facial block based on the resized facial image; Selecting the facial block according to the facial landmark calculated by interpolation; and Setting different color channels, and then extracting the physiological signal from the selected facial block; A physiological activity image processing system is configured to generate a physiological activity image (PAI) using the extracted physiological signals, wherein the PAI embeds the physiological information of the subject and is configured to: forming 1D (one-dimensional) signals from the extracted physiological signals, and applying bandpass filtering to each of the 1D signals; Concatenating the filtered 1D signals of the same color channel in sequence to create a 2D (two-dimensional) image for each color channel; as well as Concatenating the 2D images of different color channels along the dimension of the color channel to generate the PAI; a blood oxygen saturation estimator configured to receive the PAI and perform inference to obtain an estimated blood oxygen saturation level of the subject, wherein the blood oxygen saturation estimator has a deep learning model for estimating SpO2; as well as The report output module is configured to receive an inference result of estimating the blood oxygen saturation and provide a readable report on the blood oxygen saturation status of the subject based on the inference result.

2. The system according to claim 1, wherein: The color image and signal processing system includes a face detection model and a facial landmark predictor for face detection and facial landmark detection.

3. The system according to claim 2, characterized in that The face detection model is configured to process the color image frame to detect a bounding box of a facial representation within the color image frame, and the facial landmark predictor applies a detection algorithm to identify key facial points and determine the facial landmarks to define a region of interest (ROI) on the facial representation.

4. The system according to claim 1, wherein: The color image and signal processing system is further configured to first resize the detected facial image and then resize the facial landmarks so that the facial landmarks remain aligned with the resized facial image to maintain consistency.

5. The system according to claim 1, wherein: The color image and signal processing system selects the facial region near a predetermined ROI, wherein the predetermined ROI is defined by identifying the face and the locations of the facial landmarks, and wherein the predetermined ROI includes areas around eyes, nose, mouth, or a combination thereof.

6. The system according to claim 1, wherein: The color image and signal processing system extracts the physiological signal from the facial block by extracting the average pixel intensity value of each different color channel within the facial block.

7. The system according to claim 6, characterized in that The color image and signal processing system packages the average pixel intensity value into the physiological signal, thereby completing the extraction of the physiological signal.

8. The system according to claim 1, wherein: The physiological activity image processing system performs bandpass filtering on each of the 1D signals using a bandpass filter with any order and low or high cutoff frequency to eliminate noise and retain physiological related information.

9. The system according to claim 1, wherein: The deep learning model of the blood oxygen saturation estimator is trained using a training set, which includes a group of test subjects' processed PAI and the SpO2 baseline true values ​​of the same group of test subjects, so that the estimated SpO2 can be presented based on the SpO2 baseline true values.

10. The system according to claim 9, characterized in that The deep learning model of the blood oxygen saturation estimator is also configured to extract features from the processed PAI in the training set and establish a functional relationship with the SpO2 baseline true value, thereby mapping the PAI from the physiological activity image processing system to the SpO2 estimate.

11. A remote blood oxygen saturation monitoring method based on deep learning, characterized in that: include: Using a camera to capture color image frames of the subject; Extracting physiological signals from the color image frames of the subject using a color image and signal processing system comprises the following steps: performing a light intensity analysis on the color image frames of the subject; identifying the locations of faces and facial landmarks within the color image frame to detect facial images; resizing the detected facial image and calculating approximate facial landmarks by interpolation; creating a facial block based on the resized facial image; Selecting the facial block according to the facial landmark calculated by interpolation; and Setting different color channels, and then extracting the physiological signal from the selected facial block; Using a physiological activity image processing system, generating a physiological activity image (PAI) using the extracted physiological signal, wherein the PAI embeds the physiological information of the subject, comprises the following steps: forming 1D (one-dimensional) signals from the extracted physiological signals, and applying bandpass filtering to each of the 1D signals; Concatenate the filtered 1D signals of the same color channel in sequence to create a 2D (two-dimensional) image for each color channel; and Concatenating the 2D images of different color channels along the dimension of the color channel to generate the PAI; receiving the PAI for inference by an oxygen saturation estimator to obtain an estimated oxygen saturation level of the subject, wherein the oxygen saturation estimator has a deep learning model for estimating SpO2; and An inference result of estimating blood oxygen saturation is received, and a report output model is used to provide a readable report about the blood oxygen saturation status of the subject based on the inference result.

12. The method according to claim 11, characterized in that The color image and signal processing system includes a face detection model and a facial landmark predictor for face detection and facial landmark detection.

13. The method according to claim 12, characterized in that Also includes: processing the color image frame to detect a bounding box of a facial representation within the color image frame using the face detection model; as well as A detection algorithm is executed by the facial landmark predictor to identify key facial points and determine the facial landmarks, thereby defining a region of interest (ROI) on the facial representation.

14. The method according to claim 11, characterized in that Also includes: The detected facial image is first resized, and then the facial landmarks are resized so that the facial landmarks can remain aligned with the resized facial image to maintain consistency.

15. The method according to claim 11, characterized in that The color image and signal processing system selects the facial region near a predetermined ROI, wherein the predetermined ROI is defined by identifying the face and the locations of the facial landmarks, and wherein the predetermined ROI includes areas around eyes, nose, mouth, or a combination thereof.

16. The method according to claim 11, characterized in that The color image and signal processing system extracts the physiological signal from the facial block by extracting the average pixel intensity value of each different color channel within the facial block.

17. The method according to claim 16, characterized in that The color image and signal processing system packages the average pixel intensity value into the physiological signal, thereby completing the extraction of the physiological signal.

18. The method according to claim 11, characterized in that The physiological activity image processing system performs bandpass filtering on each of the 1D signals using a bandpass filter with any order and low or high cutoff frequency to eliminate noise and retain physiological related information.

19. The method according to claim 11, wherein Also includes: The deep learning model of the blood oxygen saturation estimator is trained using a training set, wherein the training set includes a group of test subjects' processed PAI and the same group of test subjects' SpO2 baseline true values, so that the estimated SpO2 can be presented based on the SpO2 baseline true values.

20. The method according to claim 19, characterized in that Also includes: Features are extracted from the processed PAI in the training set and functionally related to the SpO2 reference ground truth value, thereby mapping the PAI from the physiological activity image processing system to the SpO2 estimate.

Images