Face video blood oxygen saturation detection method and system based on deep learning

Abstract

The application discloses a face video blood oxygen saturation detection method and system based on deep learning, and relates to the technical field of computer vision; the method comprises the following steps: shooting a face video; acquiring frame-by-frame pictures of segmented face key regions in the face video; extracting iPPG signals of the face key regions through the frame-by-frame pictures of the face key regions, and equally dividing the iPPG signals of each face key region into multiple segments; training a CNN-BiLSTM-GRU blood oxygen saturation prediction model; inputting the segmented iPPG signals into the CNN-BiLSTM-GRU blood oxygen saturation prediction model respectively, and obtaining multiple preliminary blood oxygen saturation prediction values for each face key region; inputting the multiple preliminary blood oxygen saturation prediction values of each face key region into discriminators respectively, and obtaining final blood oxygen saturation prediction values through signal screening; and the method can effectively and reliably predict oxygen saturation values from the face video, and can meet the long-term detection of blood oxygen saturation values in daily life.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a face video blood oxygen saturation detection method and system based on deep learning. BACKGROUND

[0002] Blood oxygen saturation, generally represented by SpO2, refers to the percentage of oxygen combined with hemoglobin in the blood. It is a sensitive physiological parameter that can reflect the functional status of the respiratory and circulatory systems. Through non-invasive SpO2 monitoring, the operation of the pulmonary and cardiovascular systems can be understood. In the study of pulmonary diseases such as chronic obstructive pulmonary disease (COPD) and pulmonary embolism, SpO2 shows the exchange of oxygen between alveoli and small blood vessels, which helps to assess the severity of the disease. For cardiovascular diseases, especially heart failure, reduced blood oxygen saturation may indicate poor heart function. By monitoring SpO2 levels over a long period of time, potential health problems such as sleep apnea syndrome (SDB) can be detected. In addition, in special circumstances such as high altitude environments, long-distance flights and occupational exposure, SpO2 monitoring is of great significance to the evaluation of health status.

[0003] Currently, the traditional blood oxygen saturation detection method mainly uses a pulse oximeter (Pulse Oximeter) to implement. This device measures the oxygen content in the blood by placing a small sensor (usually a clip on the fingertip) on the patient's skin. It uses infrared and red light to assess the oxygen saturation in hemoglobin, and then displays the results on the instrument screen. In addition, there are some contact devices such as pulse waveform analyzers and non-invasive respiratory ventilators that can be used for blood oxygen saturation monitoring. However, the traditional contact blood oxygen saturation detection method requires detection by professional medical personnel, making it impossible for patients to conduct long-term continuous detection in a home environment. At the same time, the contact detection method can cause discomfort or pain due to puncture during the detection process of the patient, so a non-contact blood oxygen saturation detection method that can be used in a home environment has strong practicality and convenience.

[0004] However, the existing non-contact blood oxygen saturation detection methods have the following defects: the detection environment for blood oxygen saturation is relatively harsh, and cannot be applied to most environments. For example, the publication number: CN113397535A (a non-contact blood oxygen saturation detection method) requires the use of a facial diagnosis instrument to extract iPPG signals, and cannot meet the daily blood oxygen saturation monitoring in a home environment; most non-contact blood oxygen saturation detection methods focus on using complex preprocessing methods to denoise the original signal to improve the accuracy of blood oxygen saturation prediction, but also increase the detection time of blood oxygen saturation and reduce the accuracy when the detection environment changes. For example, the publication number CN114140859A (a non-contact blood oxygen saturation measurement method and system) focuses on solving the denoising of the pulse wave signal to improve the accuracy of blood oxygen saturation monitoring, but also increases the detection time of blood oxygen saturation, and has weak anti-interference ability in different detection environments, and is not highly applicable to the detection environment.

[0005] Based on the above problems, the present application provides a face video blood oxygen saturation detection method and system based on deep learning, which can be applied to non-contact effective detection of blood oxygen saturation in various environments. The deep learning network model constructed by the method combines CNN, BiLSTM and GRU network, and designs a blood oxygen saturation discriminator, which improves the accuracy and reliability of the detection results while ensuring the detection speed. SUMMARY

[0006] The present application aims to provide a face video blood oxygen saturation detection method and system based on deep learning to solve the problems of the non-contact blood oxygen saturation detection method in the prior art, such as the harsh detection environment for blood oxygen saturation and the inability to be applied to most environments.

[0007] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0008] The face video blood oxygen saturation detection method based on deep learning comprises the following steps:

[0009] S1, photographing a face video;

[0010] S2, obtaining frame-by-frame pictures of segmented face key regions in the face video;

[0011] S3, extracting iPPG signals of the face key regions from the frame-by-frame pictures of the face key regions, and dividing the iPPG signals of each face key region into multiple segments;

[0012] S4, training a CNN-BiLSTM-GRU blood oxygen saturation prediction model;

[0013] S5. Input the segmented iPPG signals into the CNN-BiLSTM-GRU blood oxygen saturation prediction model respectively, and obtain multiple preliminary blood oxygen saturation prediction values ​​for each key facial region.

[0014] S6. Input multiple preliminary blood oxygen saturation prediction values ​​of key facial regions into the discriminator. After signal filtering, the final blood oxygen saturation prediction value is obtained.

[0015] Preferably, the step S2 of acquiring frame-by-frame images of key facial regions in a face video specifically includes the following steps:

[0016] S201. Use the DLIB81 facial landmark detection model to locate the position of the face in the face video;

[0017] S202. Detect 81 key points on the face using the DLIB81 facial landmark detection model, select the 81 key points and connect them. When connecting, draw the outlines of three key parts on the face, including the nose, left cheek and right cheek.

[0018] S203. Based on three key facial features, the face is segmented into three key facial regions;

[0019] S204. Save the frame-by-frame images of the face in the face video after segmenting the face into three key facial regions.

[0020] Preferably, training the CNN-BiLSTM-GRU blood oxygen saturation prediction model in S4 specifically includes the following steps:

[0021] S401. Construct a CNN-BiLSTM-GRU blood oxygen saturation prediction model; the CNN-BiLSTM-GRU blood oxygen saturation prediction model includes a CNN part, a BiLSTM part, a GRU part, and an output part;

[0022] S402, Training the CNN-BiLSTM-GRU blood oxygen saturation prediction model; When training the CNN-BiLSTM-GRU blood oxygen saturation prediction model, a fixed learning rate of 0.001, a batch size of 16, a training epoch of 250, and the Adam optimizer are used.

[0023] S403. Save the final model weights of the trained CNN-BiLSTM-GRU blood oxygen saturation prediction model.

[0024] Preferably, in step S401, a CNN-BiLSTM-GRU blood oxygen saturation prediction model is constructed as follows:

[0025] The CNN part uses three one-dimensional convolutional layers to extract features from the iPPG signal, while using the LeakyRelu activation function to add non-linear factors and MaxPooling to reduce the dimensionality of the feature map.

[0026] The BiLSTM part is connected after the CNN part and uses a bidirectional LSTM network with 128 units;

[0027] The GRU section is connected after the BiLSTM section and uses a GRU network with 64 cells.

[0028] The output section uses a Dropout layer to prevent overfitting and two linear layers, with a LeakyReLU activation function in each of the two linear layers to increase the non-linearity of the model, to obtain the predicted blood oxygen saturation value.

[0029] Preferably, step S6, which involves inputting the preliminary predicted blood oxygen saturation value of each key facial region into the discriminator, specifically includes the following steps:

[0030] S601. Remove the maximum and minimum values ​​of the preliminary blood oxygen saturation prediction values ​​for each key facial region, and take the average of the remaining preliminary blood oxygen saturation prediction values ​​as the second-stage blood oxygen saturation prediction value for that key facial region.

[0031] S602. Calculate the variance of the second-stage blood oxygen saturation prediction value for each key facial region, remove the second-stage blood oxygen saturation prediction value of the key facial region with the largest variance, and take the average of the second-stage blood oxygen saturation prediction values ​​of the remaining two key facial regions as the final blood oxygen saturation prediction value of the face video.

[0032] A deep learning-based facial video blood oxygen saturation detection system includes:

[0033] A face signal acquisition module is used to capture facial videos using a camera;

[0034] The facial key region selection module is used to select the facial key regions in the facial video and extract the iPPG signal of each facial key region.

[0035] The signal segmentation module is used to divide the iPPG signal extracted from each key facial region into multiple segments on an average basis.

[0036] The blood oxygen saturation prediction module is used to input the multi-segment iPPG signals after each key facial region is segmented into the trained CNN-BiLSTM-GRU blood oxygen saturation prediction model to obtain multiple preliminary blood oxygen saturation prediction values ​​for each key facial region.

[0037] The blood oxygen saturation discriminator module is used to filter the preliminary blood oxygen saturation prediction values ​​of key facial regions to obtain the final blood oxygen saturation prediction values.

[0038] Preferably, the facial key region selection module includes:

[0039] The DLIB81 face landmark detection model was used to detect 81 landmarks on the face in the face video.

[0040] 81 key points were selected and connected to segment out the three key facial regions;

[0041] Save frame-by-frame images of the corresponding key facial regions after segmentation;

[0042] Extract the iPPG signal of each key facial region from frame-by-frame images of each key facial region.

[0043] Preferably, the construction of the CNN-BiLSTM-GRU blood oxygen saturation prediction model includes:

[0044] The CNN layer uses three one-dimensional convolutional layers for feature extraction, uses the LeakyRelu activation function to add non-linear factors, and uses MaxPooling to reduce the dimensionality of the feature map;

[0045] The BiLSTM layer uses a bidirectional LSTM network with 128 units;

[0046] The GRU layer uses a GRU network with 64 cells;

[0047] The regression layer uses a Dropout layer to prevent overfitting of the model, with a Dropout coefficient of 0.3, and two linear layers, each containing a LeakyReLU activation function to increase the non-linearity of the model.

[0048] Preferably, the training of the CNN-BiLSTM-GRU blood oxygen saturation prediction model includes:

[0049] Training was performed using the Adam optimizer with a fixed learning rate of 0.001.

[0050] The batch size is 16, the training epochs are 250, and the final model weights after training are saved.

[0051] Preferably, the blood oxygen saturation discriminator module includes:

[0052] The maximum and minimum values ​​of the preliminary blood oxygen saturation predictions obtained for each key facial region are removed, and the average of the remaining preliminary blood oxygen saturation predictions is taken as the second-stage blood oxygen saturation prediction value for that key facial region.

[0053] Calculate the variance of the second-stage blood oxygen saturation prediction value for each face region, delete the second-stage blood oxygen saturation prediction value of the key face region with the largest variance, and then calculate the average of the second-stage blood oxygen saturation prediction values ​​for the remaining two face regions as the final blood oxygen saturation prediction value.

[0054] Compared with the prior art, the beneficial effects of the present invention are:

[0055] (1) The present invention selects three key areas (nose, left cheek and right cheek) in the face region to extract iPPG signals, which can improve the effectiveness of iPPG signal extraction and avoid factors that affect the accuracy of blood oxygen saturation judgment, such as the forehead being covered by hair and the chin being covered by beard, thus improving the accuracy of blood oxygen saturation prediction.

[0056] (2) In this invention, the iPPG signal of each key facial region is divided into multiple segments to filter out the blood oxygen saturation value predicted by invalid signals during the blood oxygen saturation discrimination process, so as to reduce the error of blood oxygen saturation prediction.

[0057] (3) The CNN-BiLSTM-GRU blood oxygen saturation prediction model constructed in this invention can effectively extract the depth features in the iPPG signal. The BiLSTM layer combines forward LSTM and reverse LSTM to better obtain the detailed information before and after the iPPG signal. The GRU layer introduces a gating mechanism to solve the gradient vanishing problem in the recurrent neural network. Finally, the feature fusion is performed through the regression layer to improve the accuracy of blood oxygen saturation prediction.

[0058] (4) The discriminator constructed in this invention can effectively eliminate the blood oxygen saturation values ​​predicted by invalid facial key areas caused by changes in light, face movement, artifacts, etc., as well as the blood oxygen saturation values ​​predicted by invalid pulse wave signals in each facial key area, making the final result of blood oxygen saturation prediction more accurate and more reliable. Attached Figure Description

[0059] Figure 1 This is a flowchart of the deep learning-based facial video blood oxygen saturation detection method in this invention;

[0060] Figure 2 This is a schematic diagram of the connection lines of key facial features in this invention;

[0061] Figure 3This is a schematic diagram of the CNN-BiLSTM-GRU blood oxygen saturation prediction model in this invention;

[0062] Figure 4 This is a schematic diagram of the deep learning-based facial video blood oxygen saturation detection method of the present invention. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] Example 1:

[0065] See Figure 1 , 4 A deep learning-based method for detecting blood oxygen saturation in facial videos includes the following steps:

[0066] Step S110: Use a camera to record a 30-second video of a person's face;

[0067] Specifically, the subject should record a video of their face in a well-lit environment for 30 seconds at a frame rate of 30 frames per second. During the recording, the face should not shake significantly or leave the shooting area, and there should be no obstructions on the nose and cheeks.

[0068] Step S120: Segment the key facial regions using the DLIB81 facial landmark detection model;

[0069] Specifically, see Figure 2 The DLIB81 facial landmark detection model was applied to each frame of a 30-second facial video. 81 landmarks were selected, and the nose, left cheek, and right cheek of the face were segmented by connecting the landmarks. Each segmented frame was saved.

[0070] Step S130: Acquire the iPPG signal and divide it into multiple signal segments;

[0071] Specifically, the average value of the iPPG signal of each frame of the key facial region image is used as the iPPG signal at that moment, thereby forming the iPPG signal of each facial region. Then, the iPPG signals of the three key facial regions are divided into six segments on average.

[0072] Step S140: Train the constructed CNN-BiLSTM-GRU blood oxygen saturation prediction model.

[0073] Specifically, the constructed CNN-BiLSTM-GRU blood oxygen saturation prediction model is as follows: Figure 3 As shown, the model consists of a CNN layer, a BiLSTM layer, a GRU layer, and a regression layer.

[0074] The CNN layer uses three one-dimensional convolutional layers for feature extraction, uses the LeakyRelu activation function to add non-linear factors, and uses MaxPooling to reduce the dimensionality of the feature map;

[0075] The BiLSTM layer uses a bidirectional LSTM network with 128 units;

[0076] The GRU layer uses a GRU network with 64 cells;

[0077] The regression layer uses a Dropout layer to prevent overfitting of the model, with a Dropout coefficient of 0.3, and two linear layers, each containing a LeakyReLU activation function to increase the non-linearity of the model.

[0078] The model uses the publicly available Belgian dataset VV-smll, which contains 30-second facial videos of 100 volunteers in various indoor and outdoor environments, along with the blood oxygen saturation values ​​for each volunteer, as the training and validation sets. 70% of the data is used for training, and 30% for validation. The Adam optimizer is used during training with a fixed learning rate of 0.001, a batch size of 16, and 250 training epochs. The model weights after the last training epoch are saved.

[0079] Step S150: Obtain preliminary blood oxygen saturation prediction values;

[0080] Specifically, the six iPPG signals segmented from each key facial region (including the nose, left cheek, and right cheek) are input into the trained CNN-BiLSTM-GRU blood oxygen saturation prediction model. The CNN-BiLSTM-GRU blood oxygen saturation prediction model outputs six preliminary blood oxygen saturation prediction values ​​for each key facial region.

[0081] Step S160: Obtain the final predicted blood oxygen saturation value;

[0082] Specifically, the six preliminary blood oxygen saturation prediction values ​​for each key facial region are input into the discriminator to eliminate blood oxygen saturation values ​​predicted by invalid iPPG signals. The discriminator removes the maximum and minimum values ​​of the preliminary blood oxygen saturation obtained for each key facial region, and then uses the average of the remaining four blood oxygen saturation values ​​for each region as the second-stage blood oxygen saturation prediction value for that key facial region. The variance of the second-stage blood oxygen saturation value for each facial region is calculated, and the second-stage blood oxygen saturation value of the key facial region with the largest variance is removed. The average of the second-stage blood oxygen saturation values ​​of the remaining two key facial regions is used as the final blood oxygen saturation value of the subject for output.

[0083] The deep learning-based facial video blood oxygen saturation detection method of this invention uses a camera to capture a facial video, employs the DLIB81 facial landmark detection model to detect facial landmarks, and segments the face into three regions: the nose, left cheek, and right cheek by connecting selected landmarks. Frame-by-frame images of the segmented landmark regions are saved, and iPPG signals are extracted from them. The iPPG signal of each landmark region is divided into multiple segments, which are then input into a trained CNN-BiLSTM-GRU blood oxygen saturation prediction model to obtain multiple preliminary blood oxygen saturation prediction values ​​for each facial landmark region. Finally, the preliminary blood oxygen saturation prediction values ​​are input into a discriminator. After signal filtering, invalid iPPG signals caused by lighting changes, face movement, artifacts, etc., are removed, resulting in a final blood oxygen saturation prediction value with smaller errors.

[0084] The above description is only for the purpose of helping to understand the method and core essence of the present invention, but the scope of protection of the present invention is not limited thereto. For those skilled in the art, any equivalent substitutions or modifications made to the technical solution and inventive concept disclosed in the present invention within the scope of the technology disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A deep learning-based method for detecting blood oxygen saturation in facial videos, characterized in that, Includes the following steps: S1. Capture facial video; S2. Obtain frame-by-frame images of the key facial regions segmented from the face video; S3. Extract the iPPG signal of the key facial region from frame-by-frame images of the key facial region, and divide the iPPG signal of each key facial region into multiple segments on an average basis to screen out the blood oxygen saturation value predicted by invalid signals during the blood oxygen saturation discrimination process. S4. Train the CNN-BiLSTM-GRU blood oxygen saturation prediction model; A CNN-BiLSTM-GRU blood oxygen saturation prediction model was constructed. This model comprises a CNN component, a BiLSTM component, a GRU component, and an output component; details are as follows: The CNN part uses three one-dimensional convolutional layers to extract features from the iPPG signal, while using the LeakyRelu activation function to add non-linear factors and MaxPooling to reduce the dimensionality of the feature map. The BiLSTM part is connected after the CNN part and uses a bidirectional LSTM network with 128 units; The GRU section is connected after the BiLSTM section and uses a GRU network with 64 cells. The output part, after passing through a Dropout layer and two linear layers, and including a LeakyRelu activation function in the two linear layers to improve the nonlinearity of the model, obtains the predicted blood oxygen saturation value. S5. Input the segmented iPPG signals into the CNN-BiLSTM-GRU blood oxygen saturation prediction model respectively, and obtain multiple preliminary blood oxygen saturation prediction values ​​for each key facial region. S6. Input the multiple preliminary blood oxygen saturation prediction values ​​of each key facial region into the discriminator. After signal filtering, the final blood oxygen saturation prediction value is obtained. S601. Remove the maximum and minimum values ​​of the preliminary blood oxygen saturation prediction values ​​for each key facial region, and take the average of the remaining preliminary blood oxygen saturation prediction values ​​as the second-stage blood oxygen saturation prediction value for that key facial region. S602. Calculate the variance of the second-stage blood oxygen saturation prediction value for each key facial region, remove the second-stage blood oxygen saturation prediction value of the key facial region with the largest variance, and take the average of the second-stage blood oxygen saturation prediction values ​​of the remaining two key facial regions as the final blood oxygen saturation prediction value of the face video.

2. The deep learning-based facial video blood oxygen saturation detection method according to claim 1, characterized in that, The step S2, which involves acquiring frame-by-frame images of key facial regions in a face video, specifically includes the following steps: S201. Use the DLIB81 facial landmark detection model to locate the position of the face in the face video; S202. Detect 81 key points on the face using the DLIB81 facial landmark detection model, select the 81 key points and connect them. When connecting, draw the outlines of three key parts on the face, including the nose, left cheek and right cheek. S203. Based on three key facial features, the face is segmented into three key facial regions; S204. Save the frame-by-frame images of the face in the face video after segmenting the face into three key facial regions.

3. The deep learning-based facial video blood oxygen saturation detection method according to claim 1 or 2, characterized in that, The training of the CNN-BiLSTM-GRU blood oxygen saturation prediction model in S4 specifically includes the following steps: S401. Construct a CNN-BiLSTM-GRU blood oxygen saturation prediction model; S402, Training the CNN-BiLSTM-GRU blood oxygen saturation prediction model; When training the CNN-BiLSTM-GRU blood oxygen saturation prediction model, a fixed learning rate of 0.001, a batch size of 16, a training epoch of 250, and the Adam optimizer are used. S403. Save the final model weights of the trained CNN-BiLSTM-GRU blood oxygen saturation prediction model.

4. A deep learning-based facial video blood oxygen saturation detection system, characterized in that, include: A face signal acquisition module is used to capture facial videos using a camera; The facial key region selection module is used to select the facial key regions in the facial video and extract the iPPG signal of each facial key region. The signal segmentation module is used to divide the iPPG signal extracted from each key facial region into multiple segments on an average basis. The blood oxygen saturation prediction module is used to input the multi-segment iPPG signals after each key facial region is segmented into the trained CNN-BiLSTM-GRU blood oxygen saturation prediction model to obtain multiple preliminary blood oxygen saturation prediction values ​​for each key facial region. The construction of the CNN-BiLSTM-GRU blood oxygen saturation prediction model includes: The CNN layer uses three one-dimensional convolutional layers for feature extraction, uses the LeakyRelu activation function to add non-linear factors, and uses MaxPooling to reduce the dimensionality of the feature map; The BiLSTM layer uses a bidirectional LSTM network with 128 units; The GRU layer uses a GRU network with 64 cells; The regression layer uses a Dropout layer to prevent overfitting of the model, with a Dropout coefficient of 0.3, and two linear layers, each containing a LeakyReLU activation function to improve the non-linearity of the model. The blood oxygen saturation discriminator module is used to filter the preliminary blood oxygen saturation prediction values ​​of each key facial region to obtain the final blood oxygen saturation prediction value. The blood oxygen saturation discriminator module includes: The maximum and minimum values ​​of the preliminary blood oxygen saturation predictions obtained for each key facial region are removed, and the average of the remaining preliminary blood oxygen saturation predictions is taken as the second-stage blood oxygen saturation prediction value for that key facial region. Calculate the variance of the second-stage blood oxygen saturation prediction value for each face region, delete the second-stage blood oxygen saturation prediction value of the key face region with the largest variance, and then calculate the average of the second-stage blood oxygen saturation prediction values ​​for the remaining two face regions as the final blood oxygen saturation prediction value.

5. The deep learning-based facial video blood oxygen saturation detection system according to claim 4, characterized in that, The facial key region selection module includes: The DLIB81 face landmark detection model was used to detect 81 landmarks on the face in the face video. 81 key points were selected and connected to segment out the three key facial regions; Save frame-by-frame images of the corresponding key facial regions after segmentation; Extract the iPPG signal of each key facial region from frame-by-frame images of each key facial region.

6. The deep learning-based facial video blood oxygen saturation detection system according to claim 4, characterized in that, The training of the CNN-BiLSTM-GRU blood oxygen saturation prediction model includes: Training was performed using the Adam optimizer with a fixed learning rate of 0.

001. The batch size is 16, the training epochs are 250, and the final model weights after training are saved.

Images