Multi-feature fusion blood oxygen estimation method based on visible light video

Abstract

The invention belongs to the technical field of computer vision, and discloses a visible light video-based multi-feature fusion blood oxygen estimation method, which comprises the following steps of: 1, acquiring and preprocessing rPPG signals; 2, constructing a blood oxygen estimation model based on mathematical theory; 3, constructing a blood oxygen estimation model based on data driving; and 4, comprehensively processing prediction results of the blood oxygen concentration estimation model based on the mathematical principle and the blood oxygen concentration estimation model based on the data driving method, and outputting a double-model fusion result. According to the method, rPPG signal acquisition, data preprocessing, feature extraction and a deep learning model are combined, a mathematical-based blood oxygen estimation model and a data driving method-based blood oxygen estimation model are respectively constructed, and finally, prediction results of the two models are fused, so that higher prediction precision and higher robustness are achieved, and the method is suitable for popularization and application. And the generalization ability of the model is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a multi-feature fusion blood oxygen estimation method based on visible light video. Background Art

[0002] With the development of neural network technology in various fields, traditional blood oxygen calculation methods, such as the ratio method and polynomial fitting based on finger clip oximeters, are no longer the only choice. More and more researchers have begun to input the original photoplethysmography (PPG) signal into a convolutional neural network (CNN) or other types of neural networks for training and prediction. It analyzes the signal obtained from the wavelength illumination reflected from the tissue and correlates it with the blood oxygen saturation (SpO2) in the blood. Advances in PPG technology have brought about remote photoplethysmography (remote PPG, or rPPG), a non-contact method that uses camera information instead of contact sensors to monitor changes in blood volume in the skin to measure heart rate pulse. The introduction of this method has opened up a new path for the research of non-contact blood oxygen detection and improved the accuracy and reliability of detection.

[0003] In traditional blood oxygen saturation estimation, researchers widely use the Beer-Lambert Law to explain the difference in absorbance of light of different wavelengths, especially red light and infrared light, when penetrating blood. Figure 1 As shown in the figure, when blood oxygen decreases, the amplitude of the Red pulse wave increases and the IR pulse wave decreases, and the ratio of the two is strongly correlated with the blood oxygen value. This phenomenon is based on the significant difference in the absorption coefficients of oxygenated hemoglobin (HbO2) and deoxygenated hemoglobin (Hb) in the blood for these two wavelengths of light. By accurately measuring the absorbance of light of a specific wavelength, the value of blood oxygen saturation (SpO2) can be inferred. The advantage of this method is that it is highly interpretable because it is based on a clear physical law, making the predicted results easy to understand. In addition, the required equipment and algorithms are relatively simple, easy to operate and implement. However, this method also has some limitations, such as being sensitive to external interference, and motion artifacts and changes in ambient light may affect the accuracy of the measurement results.

[0004] In recent years, many scholars have proposed some methods for detecting oxygen saturation in mathematical methods based on the Beer-Lambert principle, such as a method for detecting oxygen saturation through video images captured by a multi-band camera. They simulated the scattering of light on the skin through Monte Carlo and selected appropriate bands to simulate pulse waves during oxygen saturation changes; for example, an innovative blind source separation method for AC components, which does not depend on the standard deviation of the red and blue channels. They used the energy coefficient retained in the mixing matrix as input to improve the accuracy of the estimation algorithm; for example, an iPPG technology based on reflective mode polarization imaging, in which they used a CMOS sensor polarizer to implement phase detection on each pixel, and the estimation of SpO2 was based on the phase information of the backscattered light from the fingertips of the subject; for example, an SpO2 estimation method based on a multivariate linear regression algorithm, which is based on the reflectivity on the green and red channels and requires the weighting coefficients of the regression model to be obtained through a calibration process; for example, using an ordinary RGB color camera, based on the RGB video signal obtained by shooting the palm with a white LED light source, three video pulse waves were obtained, and the AC and DC components and SpO2 were calculated through three types of ratio functions.

[0005] In the field of physiological signal detection, traditional methods based on mathematical models are highly interpretable but sensitive to signal interference. To overcome this limitation, researchers have begun to turn to monitoring using videos of palms or fingertips. However, compared with facial videos, these methods may be less convenient because they may require subjects to place their palms or fingertips in specific locations, limiting their convenience in practical applications.

[0006] In the data-driven model, the pain point of the model being sensitive to signal interference is solved by learning. For example, an innovative non-contact oxygen saturation monitoring method using photoplethysmography (PPG) uses the Euler video amplification method to enhance skin color changes caused by the cardiac cycle; for example, a video processing algorithm is used to amplify pixel intensity to detect oxygen saturation, and an image processing algorithm is further used to estimate the saturation. However, this method requires calibration of each subject under specific acquisition conditions; for example, a machine learning-based method for estimating SpO2 in facial videos, which is based on multiple pre-trained convolutional neural network (CNN) models to extract features from images of different regions of interest (ROIs), and then uses an XGBoost-based regression model for estimation; for example, a multi-model deep learning strategy that uses residual and coordinate attention networks (RCA) for feature extraction, combined with a color channel model (CCM) and a network-based model (NBM) for estimation, and classifies the output into 100 clusters corresponding to possible values ​​of SpO2; For example, the Euler video magnification technique was used to enhance the changes in facial skin pixel intensity, and machine learning regression research including generalized additive models (GAM) and extra tree regressors was conducted. They also explored deep learning methods such as 3D CNN and used infrared thermal imagers to collect images to explore in low-light conditions.

[0007] However, the models proposed so far have the following shortcomings: 1. Insufficient generalization ability: The prediction effect of hypoxemia is not significant. The reason is that the data set has a small amount of data in the case of hypoxemia.

[0008] 2. In the past, in order to achieve high-precision blood oxygen estimation, it was necessary to collect multi-sample data sets, including finger contact signals, palm videos, or control of light hitting the face, which posed a major obstacle to practical applications.

[0009] 3. Previous models were very inaccurate in predicting low blood oxygen levels, and the data sets used tended to use stable, normal blood oxygen data, which does not conform to the actual application scenarios of blood oxygen detection. Summary of the invention

[0010] The purpose of the present invention is to provide a multi-feature fusion blood oxygen estimation method based on visible light video to solve the above-mentioned technical problems.

[0011] In order to solve the above technical problems, the specific technical solution of a multi-feature fusion blood oxygen estimation method based on visible light video of the present invention is as follows: A multi-feature fusion blood oxygen estimation method based on visible light video includes the following steps: Step 1: rPPG signal acquisition and preprocessing: extract the remote photoplethysmography (rPPG) signal by monitoring the numerical changes of the average pixels in different areas of the face, and perform data preprocessing to enhance the face video data related to blood oxygen; Step 2: Construct a mathematical-based blood oxygen estimation model: Based on the method of evaluating rPPG signals, the features of 6 rPPG signals of each channel were selected, and a mathematical-based method was designed to manually extract these features and use them in the blood oxygen detection process; Step 3: Build a data-driven blood oxygen estimation model: extract the characteristic signals of physiological signals from the cleaned rPPG signal and use them to calculate blood oxygen; Step 4: Comprehensively process the prediction results of the blood oxygen concentration estimation model based on mathematical principles and the blood oxygen concentration estimation model based on data-driven methods, and output the dual model fusion result.

[0012] Furthermore, the step 1 includes the following specific steps: Step 1.1: rPPG signal acquisition: 12 facial regions were selected as regions of interest (ROIs); Step 1.2: Data preprocessing; Step 1.3: Enhance the blood oxygen-related face video data.

[0013] Furthermore, the step 1.2 includes the following specific steps: By using the fast Fourier transform (FFT) combined with a bandpass filter, the filter cutoff frequency range is designed to be 0.5 to 4 Hz, which covers the spectrum of human heart rate and its harmonic frequencies. Fast Fourier filtering is divided into 4 steps: Fast Fourier Transform: (1) in x ( t ) represents the input signal in the time domain, FFT represents the fast Fourier transform, Represents the input signal in the frequency domain; Bandpass filter settings: (2) Indicates the mask setting of the bandpass filter, indicating the frequency The signals between are retained, and the signals in the rest range are discarded; Frequency domain filtering: (3) Will The mask setting is applied to the input signal in the frequency domain to obtain a signal that only retains the frequency range we need. ; Inverse Fast Fourier Transform: (4) Will Converted into a signal in the time domain through inverse Fourier transform y ( t ); These filtered rPPG signals were normalized, interpolated, and mean filtered: (5) The above formula represents the normalized formula, where Represents the original signal y ( t ), Represents the original signal y The variance in (t); (6) The above formula represents the interpolation formula, where t Indicates the signal position that needs to be interpolated, t 0. t 1 means that the known signal positions on both sides need to be interpolated. y ( t 0), y ( t 1) Indicates t 0. t 1. Signal strength at location; (7) The above formula is a formula for average filtering. In range I ( t ) to find the average, where I’ ( t ) represents the average filtering t Position signal, I ( t+u ) indicates that t Location distance u signal strength.

[0014] Furthermore, the step 1.3 includes the following specific steps: By collecting face videos under different blood oxygen levels, and applying image processing technology to simulate the visual effects under different blood oxygen conditions, including color adjustment, brightness change, horizontal scaling, multi-angle rotation, flipping, image blur, noise, geometric distortion and optical noise enhancement, and on this basis, a combination of multiple data enhancement methods is used to expand the data set: Color adjustment: By analyzing the changes in blood oxygen saturation, the color of the video frame is adjusted to simulate the face video effect shot under different ambient lighting conditions. The technical means of converting RGB color space to HSV color space is adopted: in the HSV color space, the color tone is adjusted to simulate the visual effects under different color lights; Brightness adjustment: By adjusting the brightness and contrast of the video frame, the effect of changes in ambient light brightness on the face video is simulated to enhance the model's learning ability for scenes with drastic changes in ambient light. In the brightness adjustment process, the method of adjusting the hue after HSV color space conversion is used to achieve brightness changes; Horizontal scaling: To simulate the dynamic effect of a face swaying left and right, the size of the video frame is adjusted. The cv2.resize function in the OpenCV library is used to select the scaling parameters and then scale the image to achieve the desired scaling effect. Multi-angle rotation: When simulating the head shaking movement, the implementation of the rotation operation is involved. By calling the cv2.getRotationMatrix2D(center, angle, 1.0) function, a rotation matrix is ​​constructed. Then, the cv2.warpAffine function is used to apply the rotation matrix to the original image to perform an affine transformation to obtain the rotated image. Flip: simulate the left-right reversal during video recording, using the transforms.RandomHorizontalFlip method; Image blur: use the Gaussian blur effect, using the GaussianBlur function in the transforms library, and accurately control the intensity and range of the blur effect by accurately setting the kernel size and standard deviation, thereby simulating the image visual effects under object occlusion, low pixel resolution, and severe weather conditions; Adding noise: Using image processing technology, add random noise to the image artificially. Use the ColorJitter method in the transforms module to adjust the brightness and contrast parameters to achieve the purpose of simulating noise. Geometric distortion: simulates the dynamic changes of the face during oral expression, so that the model can learn the deformation rules of the facial skin during speech expression. This process is achieved with the help of the transforms.RandomAffine method, where the shear parameter is used to adjust the strength of the shear transformation.

[0015] Furthermore, the step 2 includes the following specific steps: Using the mean of a 10-second rPPG signal as the DC component and the variance as the AC component, a mathematical model-based method is designed to manually extract the features of the DC component and the AC component and use them in the blood oxygen detection process. According to the method of evaluating rPPG signals, 6 features of rPPG signals of each channel are selected, which are: ① Waveform S+ from trough to trough; ② The peak-to-peak waveform S-; ③The height difference H+ between two consecutive troughs; ④The height difference between two consecutive peaks H-; ⑤The time difference between the peak and the trough T1; ⑥ The height difference between the peak and the trough T2; And use the rPPG signal quality assessment algorithm on the data preprocessed signal to judge the signal quality, with a score between 0 and 1, where 0 means very poor and 1 means very good; Select rPPG signals with better quality for feature extraction. The selected signals with better quality must meet the following requirements: 1. There are two obvious peaks and troughs within the duration; 2. The rPPG signal quality is above 0.6 points; Finally, add the DC and AC features of each channel: mean and , different channels are marked with subscripts of r, g, and b, with a total of 24 features, as shown in formula (8): (8) And the physiological signal features obtained after multiple linear layers and activation layers f phy , its formula is expressed as: f phy =ReLU( W 2×ReLU( W 1× C + b 1)+ b 2) (9) in, W 1, W 2, b 1, b 2 represent the weights and biases of different linear layers, ReLU is the activation function, and C is the manually extracted feature signal of the input.

[0016] Furthermore, the step 3 includes the following specific steps: Step 3.1: Construct a data-driven blood oxygen estimation model structure framework: used to extract characteristic signals of physiological signals from the cleaned rPPG signals and use them to calculate blood oxygen; Step 3.2: Group and resample the collected data: adjust the number of samples in the minority category and the majority category to achieve a balanced state of the data set; Step 3.3: Propose a targeted loss calculation method, namely weighted loss.

[0017] Furthermore, the step 3.1 includes the following specific steps: The data-driven blood oxygen estimation model includes a residual module and two attention modules, wherein the two attention modules are a channel attention module and a spatial attention module; The reflected photoplethysmogram (rPPG) signals obtained from the face, after being aligned in a preprocessing step, are then passed through a residual module backbone network for feature extraction. The design of the residual module backbone network introduces a regularization mechanism that allows the network to learn a residual function, i.e., the residual between the input and the output; Each residual block uses three one-dimensional convolutional layers to achieve feature dimensionality reduction, extraction, and dimensionality increase. The specific formula is expressed as: RB( x )= W 3*ReLU( W 2*ReLU( W 1* x )) (10) in, W 1, W 2, W 3 represent the weights of different convolutional layers, * represents the convolution operation, ReLU is the activation function, and x Is the input feature signal, the first convolutional layer W 1 is responsible for dimensionality reduction to reduce the amount of calculation; the second convolution layer W 2 Perform feature extraction; the third convolution layer W 3 is responsible for dimensionality increase and restoring the dimension of the feature map. The sizes of the three convolution kernels are 1, 3, and 1 respectively; When the residual blocks are used in cascade, two different configurations are distinguished. In the first case, the number of channels needs to be adjusted. Formula F1 is expressed as: F1=RB( x )+ W s * x (11) Among them, RB represents the single residual block operation process, see formula (10), and W s is the weight matrix used to adjust the residual connection to match the number of channels. At the same time, the residual connection ensures that the input feature signal x Flow; In the second case, there is no need to adjust the number of channels, and the formula F2 is expressed as: F2=RB( x )+ x (12) That is, the residual connection is directly applied to the original input feature map x, and the whole residual module extracts the feature map process Res( x ) is expressed by the following formula: (13) Res( x ) represents the result of feature extraction of the residual module as the backbone network, It does not mean to sum, but to repeat the process seven times; The channel attention mechanism and the spatial attention mechanism are closely connected in a front-to-back manner. Subsequently, the combination is connected to a three-layer residual network, which is used to increase or reduce the dimension of the channel to achieve effective feature extraction. This process is called a single attention extraction. In the subsequent steps, the attention extraction process is repeated twice. The first attention extraction increases the number of channels from 32 dimensions to 64 dimensions to enhance the model's ability to capture features; then, the second attention extraction reduces the number of channels from 64 dimensions to 32 dimensions to simplify the model and reduce computational complexity. The channel attention mechanism automatically learns and emphasizes the channels that contribute more to the blood oxygen signal through the following formula, while suppressing the channels that are disturbed by noise or ambient light: CA( x )=Sigmoid( g ( x ))⊙Rex( x ) (14) Among them, Rex( x ) indicates that the result of feature extraction of the residual module of the backbone network is used as the input feature signal, Sigmoid indicates the Sigmoid activation function, ⊙ indicates the element-by-element multiplication of each channel, g ( x ) represents the features after being processed by global pooling and convolutional layers, and its formula is expressed as: g( x )=Conv(ReLU(Conv ( avg / max pool (Rex( x ))))) (15) in, avg / maxpool (Rex( x )) is a reference to Rex( x ) performs global average pooling or maximum pooling, Conv is a one-dimensional convolutional layer for dimensionality reduction and dimensionality increase, The spatial attention mechanism is introduced. The spatial attention mechanism dynamically identifies and emphasizes the key segments in the time series for blood oxygen value prediction through the following formula: SA(x)= Sigmoid ( s ( x )) ⊙Rex( x ) (16) here, s ( x ) is the feature processed by the operation on the time series and the convolution layer, and its formula is expressed as: s ( x )=Conv( concat ( avg pool (x ), max pool (Rex( x )))) (17) avg pool ( x )and max pool ( x ) are respectively for Rex( x ) for average pooling and maximum pooling, concat It connects the two pooling results in the channel dimension. Conv is a one-dimensional convolutional layer used to learn spatial weights. The output CSA combining the spatial attention mechanism and the channel attention mechanism is expressed as: CSA( x )=SA(CA( x )) (18) Finally, the feature maps are flattened and converted into one-dimensional signals for further processing in the fully connected layer, where they are used to generate the final prediction results. SpO2 pre , its formula is expressed as: SpO2 pre =FC3(View(CSA( x ))) (19) In this process, the first operation is the View operation, which expands the input data into a one-dimensional signal. This operation is to convert the original data into a form suitable for the fully connected layer to ensure that the data can be correctly processed by the fully connected layer. Then, the data will be processed by the FC3 operation, that is, the three-layer fully connected layer, and finally the blood oxygen prediction result will be output. SpO2 pre .

[0018] Furthermore, the step 3.2 includes the following specific steps: The sampler is used to load data, and a group resampling strategy is proposed. The strategy resamples data based on the proportion of the total sample amount, and performs multiple sampling on categories with less data. The specific steps of the group resampling are as follows: First, the blood oxygen value is divided into several intervals. The blood oxygen value is subdivided into 20 different intervals from 81 to 100; Secondly, the number of samples in each interval is counted. Before model training, the number of samples in each interval in the training set is counted in detail to identify the intervals with fewer and more samples, and record these intervals in the resampling grouping, that is, oversampling and undersampling operations are performed on the intervals with extreme sample numbers; Next, the resampling ratio is set. This ratio is dynamically calculated based on the number of samples in each interval. Finally, the number of samples in each interval with a small number of samples is increased to 2% of the total number of samples, and the number of samples in the interval with a large number of samples is reduced to 5% of the total number of samples. Finally, according to the actual situation of the data set, choose the appropriate oversampling or undersampling method; Oversampling strategy: For intervals with a small number of samples, the number of samples is expanded by copying existing samples or using SMOTE technology to generate new samples; Undersampling strategy: For intervals with a large number of samples, some samples are randomly removed to reduce the number of samples; A dynamic adjustment mechanism is introduced to appropriately perform oversampling or undersampling according to the actual situation of each grouped sample.

[0019] Furthermore, the step 3.3 includes the following specific steps: Two widely used loss functions are used: mean absolute error (MAE) and root mean square error (RMSE), and the two loss functions are weighted summed to obtain the loss : (20) (twenty one) Here SPO2 pre,j Represents the model j Predicted blood oxygen saturation values, SPO2 gt,j Indicates j A reference blood oxygen saturation value, n is the total number of samples; (twenty two) , Represent the loss coefficients of MAE and RMSE respectively, The larger the weight coefficient, the , ; Using weighted loss method: First, the blood oxygen values ​​were divided into 20 different groups ranging from 81 to 100. Before training the model, the data in the training set were grouped and counted, and the number of samples in each group was recorded; Then, during model training, the loss weight for each group is calculated: (twenty three)

[0020] w i Indicates i The weight of each group,N represents the total number of samples in the training set, and N i It means the i The number of samples in each group; Finally, this weight is applied to the i The loss of the classification LOSS pre，i The final loss LOSS pre , the calculation method is as follows: (twenty four)

[0021] Ensure that the model gives different weights to samples from different groups during training.

[0022] Furthermore, the step 4 includes the following specific steps: The prediction results of the blood oxygen estimation model based on mathematics and the blood oxygen estimation model based on data-driven methods are comprehensively processed. According to the blood oxygen output of the two models, the weight is dynamically calculated using the following formula w 1 and w 2 ,satisfy w 1 + w 2 =1: (25) SPO2 phy Represents the blood oxygen saturation value output by the mathematical blood oxygen estimation model, SPO2 pre Represents the blood oxygen saturation value output by the data-driven blood oxygen estimation model, applying these weights w 1 and w 2 The prediction results of the two models are weighted fused to output the final blood oxygen concentration estimate.

[0023] The multi-feature fusion blood oxygen estimation method based on visible light video of the present invention has the following advantages: The present invention combines rPPG signal acquisition, data preprocessing, feature extraction and deep learning models, and constructs a blood oxygen estimation model based on mathematics and a blood oxygen estimation model based on a data-driven method. It adopts innovative spatial and channel attention mechanisms, explores the manual extraction method of model input features, and manually extracts blood oxygen-related features from the signals of the three channels of visible light RGB, and uses them in the blood oxygen prediction process. This method complements the deep neural network model and improves the accuracy and reliability of blood oxygen detection. When calculating the loss, different weights are added to blood oxygen of different values, and classes with less data are resampled to achieve the same probability of sampling different blood oxygen values ​​for each batch, so that the model will not be biased towards predicting high blood oxygen during training, and the model's generalization ability is improved. Finally, the results predicted by the two models are fused to achieve higher prediction accuracy and stronger robustness. The present invention verifies the generalization ability and practicality of the model through testing of self-built data sets and public data set PURE. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is the relationship between blood oxygen and Red / IR pulse wave; Figure 2 This is a framework diagram of the blood oxygen estimation model of the present invention; Figure 3a Schematic diagram of signal quality before preprocessing; Figure 3b It is a schematic diagram of signal quality after preprocessing; Figure 4 This is a data-driven blood oxygen estimation model framework diagram of the present invention; Figure 5 This is a schematic diagram of multi-layer cascade of single-layer residual blocks of the present invention; Figure 6a This is a schematic diagram of data distribution before resampling of the present invention; Figure 6b This is a schematic diagram of data distribution after resampling of the present invention; Figure 7 Schematic diagram of blood oxygen distribution of the PURE data set of the present invention; Figure 8a This is a schematic diagram of the 5-fold cross-validation results of the training set; Figure 8b This is a schematic diagram of the 5-fold cross validation results of the test set; Fig. 9 It is a schematic diagram of the results of the hypoxemia prediction model of the present invention. DETAILED DESCRIPTION

[0025] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a multi-feature fusion blood oxygen estimation method based on visible light video of the present invention in conjunction with the accompanying drawings.

[0026] The model proposed in this invention is based on visible light video and estimates blood oxygen by extracting multiple features and fusing them. Figure 2 As shown in the figure, it includes rPPG signal acquisition and preprocessing module, mathematical-based blood oxygen estimation model, data-driven blood oxygen estimation model, and the final output dual-model fusion model. The specific implementation steps are as follows: Step 1: rPPG signal acquisition and preprocessing; Step 1.1: rPPG signal acquisition: extract remote photoplethysmography (rPPG) signals by monitoring the value changes of average pixels in different areas of the face; In this process, we used the facial key point recognition technology provided by Mediapipe. Mediapipe is an open source cross-platform framework developed by Google, which is widely used to build multimodal real-time applications, including human pose estimation, face recognition, and gesture recognition. This technology relies on advanced machine learning models to detect and track the key points of faces in videos in real time with high accuracy and low latency. This is crucial for us to obtain remote photoplethysmography (rPPG) signals.

[0027] According to research, specific areas of the human face have a high correlation with the acquisition of remote photoplethysmography (rPPG) signals. The rPPG signals in these areas are not only of higher quality, but also have a better signal-to-noise ratio. The human face has dense blood vessels, especially in the cheeks and forehead. Taking this part as the region of interest (ROI) can obtain more accurate rPPG signals. Therefore, the present invention selects 12 facial areas such as the left and right cheeks, forehead, nose, chin, philtrum and the entire face as regions of interest (ROIs) to obtain higher quality rPPG signals. Thereby improving the reliability of blood oxygen saturation monitoring and providing more accurate data support for non-contact physiological monitoring.

[0028] Step 1.2: Data preprocessing; In order to reduce the noise in the raw remote photoplethysmography (rPPG) signal and enhance the signal generated by the heart pulsation, we implemented a series of data preprocessing strategies. These strategies are designed to reduce the noise intensity in the training data while improving the quality of the rPPG signal. We used the fast Fourier transform (FFT) combined with a bandpass filter and designed the filter with a cutoff frequency range of 0.5 to 4Hz, which perfectly covers the spectrum of human heart rate (0.5 to 2.5Hz) and its harmonic frequencies. Through this frequency domain filtering method, we are able to retain key information in the rPPG signal, such as the second harmonic and core signal details, while effectively suppressing irrelevant noise. This process not only improves the signal-to-noise ratio of the signal, but also makes the signal curve smoother, providing a more accurate and reliable data basis for subsequent blood oxygen saturation estimation.

[0029] Fast Fourier filtering is divided into 4 steps: (1) Fast Fourier Transform: (1) in x ( t ) represents the input signal in the time domain, FFT represents the fast Fourier transform, Represents the input signal in the frequency domain; (2) Bandpass filter settings: (2) Indicates the mask setting of the bandpass filter, indicating the frequency The signals between are retained, and the signals in the rest range are discarded; (3) Frequency domain filtering: (3) Will The mask setting is applied to the input signal in the frequency domain to obtain a signal that only retains the frequency range we need. ; (4) Inverse Fast Fourier Transform: (4) Will Converted into a signal in the time domain through inverse Fourier transform y ( t ).

[0030] In order to further improve the efficiency of the training process, we normalized, interpolated and mean-filtered these filtered rPPG signals.

[0031] (5) The above formula represents the normalized formula, where Represents the original signal y ( t ), Represents the original signal y The variance in (t).

[0032] (6) The above formula represents the interpolation formula, where t Indicates the signal position that needs to be interpolated, t 0. t 1 means that the known signal positions on both sides need to be interpolated. y ( t 0), y ( t 1) Indicates t 0. t Signal strength at position 1.

[0033] (7) The above formula is a formula for average filtering. In range I ( t ) to find the average, Generally, it is 50. I’ ( t ) represents the average filtering t Position signal, I ( t+u ) indicates that t Location distance u signal strength.

[0034] Step 1.3: Enhance the face video data related to blood oxygen; In order to solve the problem that the small amount of data cannot be generalized to scenes with intense movements and large ambient lighting effects, we proposed a method for blood oxygen-related face video data enhancement. By collecting face videos under different blood oxygen levels and applying image processing technology (such as color adjustment to simulate changes in ambient light, blurring of faces to simulate people shaking their heads, left and right up and down scaling to simulate people nodding and shaking their heads, etc.) to simulate the visual effects under different blood oxygen conditions, we can expand the training data set and improve the generalization ability of the model. These include enhancement methods such as color adjustment, brightness change, horizontal scaling, multi-angle rotation, flipping, image blur, noise, geometric distortion and optical noise. On this basis, we use a combination of multiple data enhancement methods on the data set, such as flip and rotation combination, translation and scaling combination, etc., to achieve the expansion of the data set. The following introduces the implementation method of our face data enhancement method: (1) Color adjustment. By analyzing the changes in blood oxygen saturation, the color of the video frame can be appropriately adjusted, thereby simulating the face video effect shot under different ambient lighting conditions. To achieve this goal, the present invention adopts the technical means of converting RGB color space to HSV color space. In the HSV color space, hue is a key parameter for describing color. By adjusting it, the visual effects under different color lights can be simulated. This color adjustment technology not only helps to deeply understand and analyze the color changes in the video, but also has significant application value in actual blood oxygen prediction.

[0035] (2) Brightness adjustment. The present invention simulates the impact of ambient light brightness changes on face videos by adjusting the brightness and contrast of video frames to enhance the model's learning ability for scenes with drastic ambient light changes. In the brightness adjustment process, the method of adjusting the hue after HSV color space conversion is used to achieve brightness changes. To this end, the present invention implements a "video parameter continuity algorithm" that can continuously adjust light changes based on the color of the previous frame rather than randomly generating, thereby more realistically simulating the continuous changes in ambient light. The video parameter continuity algorithm can adapt to blood oxygen estimation under different lighting conditions. In addition, the algorithm also takes into account the correlation between video frames, and dynamically adjusts the brightness and contrast by analyzing the color changes of consecutive frames to simulate scenes with gradually changing light in the real world. This continuity adjustment not only improves the model's adaptability to light changes, but also enhances the naturalness of video data, making the enhanced data set closer to actual application scenarios.

[0036] The pseudo code of the video parameter continuity algorithm is as follows: Define the video parameter continuity algorithm (index, parameters, speed): index = speed * index If parameter < 0: If the quotient of index divided by (-2*argument) modulo 2 is equal to 0: Returns parameter + index modulo (-2*parameter) otherwise: Returns -1*parameter - index modulo (-2*parameter) otherwise: If the quotient of index divided by (2*argument) modulo 2 is equal to 0: Returns parameter - (index) modulo (2*parameter) otherwise: Returns -1*parameter+(index) modulo(2*parameter) The parameter defines the maximum increase or decrease value of the brightness adjustment. In addition, the video parameter continuity algorithm calculates the parameter value that should be adjusted for the current frame based on the input index value (i.e. the sequence number of the current frame) and the speed parameter. For example, if the maximum brightness adjustment value is set to 50, the speed parameter is 1, and the current frame sequence number is 1 (i.e. the second frame), the brightness of the current frame is adjusted to increase by 49 units. The brightness adjustment value will decrease to -50 as the frame sequence number decreases, and then increase to 50, forming a reciprocating pattern until the video frame traversal is completed.

[0037] (3) Horizontal scaling. To simulate the dynamic effect of a face swaying left and right, the present invention achieves this by adjusting the size of the video frame. The cv2.resize function in the OpenCV library is used to select appropriate scaling parameters, and then the image is scaled to achieve the desired scaling effect. However, similar to brightness adjustment, in order to more realistically simulate the continuity of face swaying during data enhancement, the continuity of scaling between video frames must be ensured. Therefore, the present invention extends the "video parameter continuity algorithm" used in brightness adjustment to the process of generating horizontal scaling.

[0038] (4) Multi-angle rotation. When simulating the head shaking movement, the implementation of the rotation operation is involved. By calling the cv2.getRotationMatrix2D(center, angle, 1.0) function, a rotation matrix can be constructed. Subsequently, the rotation matrix is ​​applied to the original image using the cv2.warpAffine function to perform an affine transformation to obtain the rotated image. In this process, the key lies in the precise setting of the rotation angle. Therefore, this process also needs to be combined with the "video parameter continuity algorithm" to ensure the consistency and accuracy of image processing.

[0039] (5) Flip. Simulate the left-right flipping of video recording. Use the transforms.RandomHorizontalFlip method.

[0040] (6) Image blur: In the field of image processing, for visual simulation under object occlusion, low resolution and bad weather conditions, it is usually necessary to use specific technical means to reproduce the impact of these factors on the visual quality of the image. Gaussian blur technology, as a widely used method, can effectively simulate the image visual effects under the above conditions. To achieve the Gaussian blur effect, the present invention uses the GaussianBlur function in the transforms library. By accurately setting the kernel size (kernel_size) and the standard deviation (sigma), the intensity and range of the blur effect can be accurately controlled, thereby simulating the image visual effects under object occlusion, low pixel resolution and bad weather conditions. The kernel size parameter determines the range of the blurred area, while the standard deviation parameter affects the intensity and distribution characteristics of the blur effect. By adjusting these two parameters, researchers can flexibly simulate a variety of different visual effects to meet the needs of different application scenarios.

[0041] (7) Adding noise: When studying the field of image processing, the impact of image noise caused by environmental simulation or camera imaging defects on portrait recognition cannot be ignored. To simulate the noise effect, image processing technology can be used to artificially add random noise to the image. In specific operations, the ColorJitter method in the transforms module can be used to adjust parameters such as brightness and contrast to achieve the purpose of simulating noise.

[0042] (8) Geometric distortion: simulates the dynamic changes of the face during oral expression, so that the model can learn the deformation rules of the facial skin during speech expression. This process can be achieved with the help of the transforms.RandomAffine method, where the shear parameter is used to adjust the strength of the shear transformation.

[0043] Step 2: Construct a mathematical blood oxygen estimation model; The traditional mathematical model-based method has proved that there is a strong correlation between the DC component (DC) and the AC component (AC) in the rPPG signal and the SPO2 value. We use the mean of a 10-second rPPG signal as the DC component (DC) and the variance as the AC component (AC). In the present invention, we designed a mathematical model-based method to manually extract the features of this part and use it in the blood oxygen detection process. Although the traditional mathematical model-based method has the support of the interpretable Beer-Lambert theory, the method of predicting blood oxygen through rPPG signals has obvious external influence factors. Therefore, according to the method of evaluating rPPG signals, we selected 6 features of rPPG signals for each channel, which are: ① Waveform S+ from trough to trough; ② The peak-to-peak waveform S-; ③The height difference H+ between two consecutive troughs; ④The height difference between two consecutive peaks H-; ⑤The time difference between the peak and the trough T1; ⑥The height difference between the peak and the trough is T2.

[0044] And use the rPPG signal quality assessment algorithm on the data preprocessed signal to judge the signal quality, and the score is between 0-1, where 0 means very poor and 1 means very good. Figure 3a , 3b Shown is the comparison of signal quality before and after preprocessing.

[0045] We will select rPPG signals with better quality for feature extraction. The selected signals with better quality must meet the following requirements: (1) there are two obvious peaks and troughs within the duration; (2) the rPPG signal quality score must be above 0.6.

[0046] Finally, add the DC (mean) and AC ( ), different channels are marked with subscripts r, g, and b, for example: mean r The mean value of the rPPG signal of the red channel is taken as DC, with a total of 24 features, as shown in formula (8): (8) And the physiological signal features obtained after multiple linear layers and activation layers f phy , the formula can be expressed as: f phy =ReLU( W 2×ReLU( W 1× C + b 1)+ b 2) (9) in, W 1, W 2, b 1, b 2 represent the weights and biases of different linear layers, ReLU is the activation function, and C is the manually extracted feature signal of the input.

[0047] Step 3: Build a data-driven blood oxygen estimation model; Step 3.1: Construct a data-driven blood oxygen estimation model structure framework; The data-driven blood oxygen estimation model Residual Attentional Oxygen Estimation Network (RAOEN) is used to extract characteristic signals of physiological signals from the cleaned rPPG signals and use them to calculate blood oxygen. Figure 4 The detailed information of the network is shown, including the residual module and two attention modules, namely the channel attention module and the spatial attention module. Compared with the traditional CNN network model, the residual connection solves the problem of gradient disappearance in the multi-layer network model and plays a key role in the extraction of complex physiological signal features. And through the attention mechanism, the physiological signal features related to blood oxygen can be extracted more accurately, making the results more accurate.

[0048] The reflected photoplethysmography (rPPG) signals obtained from the face, after being aligned through a preprocessing step, are then passed through a residual module backbone network for feature extraction. The design of the residual module backbone network introduces a regularization mechanism that allows the network to learn a residual function, that is, the residual between the input and the output. This mechanism makes it easier for the network to learn the identity mapping, thereby improving the generalization ability of the model. The residual connection ensures that the integrity of the signal and the effective propagation of the gradient can be maintained even in complex network structures with multiple levels. This design not only improves the model's ability to extract physiological signal features, but also enhances the model's adaptability to changes in different individuals and physiological states. In this way, the model can more accurately capture subtle changes in physiological signals, thereby more effectively extracting blood oxygen-related feature information.

[0049] In the present invention, each residual block realizes the dimension reduction, extraction and dimension increase of features through three one-dimensional convolutional layers. The specific formula is expressed as follows: RB( x )= W 3*ReLU( W 2*ReLU( W 1* x )) (10) in, W 1, W 2, W 3 represent the weights of different convolutional layers, * represents the convolution operation, ReLU is the activation function, and x Is the input feature signal. The first convolutional layer W 1 is responsible for dimensionality reduction to reduce the amount of calculation; the second convolution layer W 2 Perform feature extraction; the third convolution layer W 3 is responsible for dimensionality increase and restoring the dimension of the feature map. The sizes of the three convolution kernels are 1, 3, and 1 respectively.

[0050] When residual blocks are used in cascade, we distinguish two different configurations. The first case requires adjusting the number of channels, such as Figure 5 The dotted line connecting the adjusted layers in the formula F1 can be expressed as: F1=RB( x )+ W s * x (11) Among them, RB represents the single residual block operation process, see formula (10), and W s is the weight matrix used to adjust the residual connection to match the number of channels. At the same time, the residual connection ensures that the input feature signal x This helps alleviate the gradient vanishing problem in deep networks.

[0051] In the second case, there is no need to adjust the number of channels, such as Figure 5 The solid line connection part that has not been adjusted in the layer can be expressed as: F2=RB( x )+ x (12) That is, directly in the original input feature map x We apply residual connection on the whole residual module to extract feature maps Res( x ) is expressed by the following formula: (13) Res( x ) represents the result of feature extraction of the residual module as the backbone network, It does not mean to sum, it means to repeat the process seven times.

[0052] With this design, our model can not only effectively extract key physiological features from rPPG signals, but also adapt to different computing environments, ensuring fast and accurate blood oxygen saturation estimation even on resource-constrained terminal devices.

[0053] In this study, we first used the backbone network to extract image features. Subsequently, these features were input into a specially designed module that integrates the channel attention mechanism (CA) and the spatial attention mechanism (SA). The purpose of the channel attention mechanism is to strengthen the attention to the correlation between different channels in the feature map to more effectively capture the interaction between features. The spatial attention mechanism focuses on the local area of ​​the image, especially the distribution and change of the rPPG signal in the spatial dimension. These two points are highly consistent with the application law of the Beer-Lambert principle in blood oxygen calculation, that is, oxygenated hemoglobin and deoxygenated hemoglobin have different absorbances in different channels (wavelengths), resulting in the amplitude of the rPPG signal in different channels directly reflecting the concentration of oxygenated hemoglobin and deoxygenated hemoglobin. The spatial attention mechanism just complements the shortcomings of the channel attention mechanism in extracting local spatial amplitude information. By combining the two methods, the model can analyze and understand the key information in the image more finely, thereby improving the detection and analysis accuracy of rPPG signals. This method of combining channel and spatial attention mechanisms enables the model to extract key features more efficiently and accurately when processing complex image data, thereby improving overall performance.

[0054] In the present invention, the channel attention mechanism and the spatial attention mechanism are closely connected in a front-to-back manner. Subsequently, the combination is connected to a residual network consisting of three layers, whose main function is to increase or reduce the dimension of the channel to achieve effective feature extraction. This process is called a primary attention extraction. In subsequent steps, the attention extraction process is repeated twice. The first attention extraction increases the number of channels from 32 dimensions to 64 dimensions to enhance the model's ability to capture features. Immediately afterwards, the second attention extraction reduces the number of channels from 64 dimensions to 32 dimensions to simplify the model and reduce computational complexity. Through this repeated dimensionality increase and dimensionality reduction operation, the model can more accurately capture the key information in the image, thereby showing excellent performance in various visual tasks.

[0055] The channel attention mechanism automatically learns and emphasizes the channels that contribute more to the blood oxygen signal through the following formula, while suppressing the channels that are disturbed by noise or ambient light: CA( x )=Sigmoid( g ( x ))⊙Rex( x ) (14) Among them, Rex( x) indicates that the result of feature extraction of the residual module of the backbone network is used as the input feature signal, Sigmoid indicates the Sigmoid activation function, ⊙ indicates the element-by-element multiplication of each channel, g ( x ) represents the features after being processed by global pooling and convolutional layers, and its formula is expressed as: g( x )=Conv(ReLU(Conv ( avg / max pool (Rex( x ))))) (15) in, avg / maxpool (Rex( x )) is a reference to Rex( x ) performs global average pooling or maximum pooling, Conv is a one-dimensional convolutional layer used for dimensionality reduction and dimensionality increase, see Figure 4 Channel attention module part.

[0056] Furthermore, since most studies on rPPG signals focus on the analysis of time dimension features, the present invention also introduces a spatial attention mechanism. The spatial attention mechanism dynamically identifies and emphasizes the key segments in the time series for blood oxygen value prediction through the following formula: SA(x)= Sigmoid ( s ( x )) ⊙Rex( x ) (16) here, s ( x ) is the feature processed by time series operations (such as average pooling and maximum pooling) and convolutional layers, and its formula is expressed as: s ( x )=Conv( concat ( avg pool ( x ), max pool (Rex( x )))) (17) avg pool ( x )and max pool ( x ) are respectively for Rex( x ) for average pooling and maximum pooling. concat is to connect the two pooling results in the channel dimension. Conv is a one-dimensional convolution layer used to learn spatial weights. Figure 4 Spatial attention module part.

[0057] By combining channel attention and spatial attention mechanisms, our model is able to understand and analyze rPPG signals more comprehensively. Specifically, the output CSA of combining spatial attention mechanism and channel attention mechanism can be expressed as: CSA( x )=SA(CA( x )) (18) This enables our model to provide more stable and accurate blood oxygen saturation estimates under various conditions.

[0058] Finally, the feature maps are flattened and converted into one-dimensional signals to be fed into the fully connected layer for further processing. In the fully connected layer, these one-dimensional signals are used to generate the final prediction results. SpO2 pre , its formula is expressed as: SpO2 pre =FC3(View(CSA( x ))) (19) In this process, the first operation is the View operation, which expands the input data into a one-dimensional signal. This operation is to convert the original data into a form suitable for the fully connected layer to ensure that the data can be correctly processed by the fully connected layer. Then, the data will go through the FC3 operation, that is, the processing of three layers of fully connected layers. The final output is the blood oxygen prediction result SpO2 pre .

[0059] Step 3.2: Group and resample the collected data; The present invention uses a sampler to load data to ensure the diversity and representativeness of the data. The sampler can extract samples from a large-scale data set according to preset rules, which helps to improve the generalization ability of the model.

[0060] In response to the scarcity of abnormal blood oxygen data, the present invention proposes a specific group resampling strategy. This strategy performs data resampling based on the proportion of the total sample volume, with the aim of increasing the proportion of abnormal blood oxygen data, and performing multiple sampling on categories with less data (such as low blood oxygen value samples) to optimize the model's learning effect on facial features under low blood oxygen conditions, and to maintain the rarity of abnormal data in the entire data distribution as much as possible.

[0061] The group resampling strategy aims to solve the problem of class imbalance in the data set. This method adjusts the number of samples in the minority class and the majority class to achieve a balanced state of the data set. The following are the specific steps of group resampling: First, the target variable (blood oxygen value) is divided into several intervals. The blood oxygen value is subdivided into 20 different intervals from 81 to 100.

[0062] Secondly, the number of samples in each interval is counted. Before model training, the number of samples in each interval in the training set is counted in detail. The intervals with fewer and more samples are identified and recorded in the resampling grouping, that is, oversampling and undersampling operations are performed on the intervals with extreme sample numbers.

[0063] Next, the resampling ratio is set, which is dynamically calculated based on the number of samples in each interval. Finally, the number of samples in each interval with a small number of samples is increased to 2% of the total number of samples, and the number of samples in the interval with a large number of samples is reduced to 5% of the total number of samples.

[0064] Finally, according to the actual situation of the data set, choose the appropriate oversampling or undersampling method.

[0065] Oversampling strategy: For intervals with a small number of samples, the number of samples can be expanded by copying existing samples or using techniques such as SMOTE to generate new samples.

[0066] Undersampling strategy: For intervals with a large number of samples, some samples are randomly removed to reduce the number of samples.

[0067] Through the above resampling process, the number of samples in each interval will tend to be consistent, making the distribution of the entire data set more balanced. The pseudo code of the entire process is shown below.

[0068] Group resampling (training set, resampling group,): For each group in the group boundary: Calculate the number of samples for this group If the sample size of this group is <2% * total sample size: Oversample this group to 2% * Total sample size Otherwise if the sample size of this group > 5% * total sample size: Undersample this group to 5% * Total sample size Return the balanced training set In order to deal with the situation where oversampling may introduce duplicate samples, leading to model overfitting, and undersampling may cause information loss, resulting in poor model performance, we did not strictly force the samples of each group to be flattened, but adopted a more flexible strategy. Specifically, we introduced a dynamic adjustment mechanism to moderately oversample or undersample according to the actual situation of each group sample. This not only ensures the diversity of the data set, but also avoids the negative impact on model performance due to sample duplication or information loss. In this way, we can better balance the data set, thereby improving the generalization ability and prediction accuracy of the model. Figure 6a , Figure 6bAs shown, the data distribution before and after resampling is more uniform.

[0069] Step 3.3: Propose a targeted loss calculation method, namely weighted loss; In order to evaluate the prediction performance of the model, the present invention adopts two widely used loss functions: mean absolute error (MAE) and root mean square error (RMSE), and performs a weighted summation on these two loss functions to obtain the loss : (20) (twenty one) Here SPO2 pre,j Represents the model j Predicted blood oxygen saturation values, SPO2 gt,j Indicates j A reference (true) blood oxygen saturation value, n is the total number of samples.

[0070] (twenty two) , Represent the loss coefficients of MAE and RMSE respectively. Since RMSE works better when the loss is large and MAE works better when the loss is small, we give The larger the weight coefficient, the , By calculating the values ​​of these loss functions, this study can quantify the difference between the model prediction results and the true values, and then evaluate and optimize the performance of the model.

[0071] In order to solve the problem of data balance, we have made some special improvements based on the above loss calculation. Although we have taken data enhancement measures to improve the model's ability to cope with unprecedented situations, such as changes in ambient light and distortion of artifact images, and applied resampling strategies to resample smaller data and larger data groups, the overall data set is still unbalanced. We have performed corresponding data enhancement processing on each sample, but these efforts cannot completely solve the problem of unbalanced data distribution. Therefore, we proposed a targeted loss calculation method, that is, the weighted loss method.

[0072] First, we grouped the labels carefully. Specifically, we divided the blood oxygen values ​​into 20 different groups ranging from 81 to 100. Before training the model, we grouped and counted the data in the training set and recorded the number of samples in each group.

[0073] Then, during model training, we calculate the loss weight for each group according to a specific formula.

[0074] (twenty three) In this formula, w i Indicates i The weight of each group, N represents the total number of samples in the training set, and N i It means the i The number of samples in a group.

[0075] Finally, this weight is applied to the i The loss of the classification LOSS pre，i The final loss LOSS pre , the calculation method is as follows: (twenty four)

[0076] In this way, we can ensure that the model gives different emphasis to samples in different groups during training, thereby alleviating the impact of unbalanced data distribution to a certain extent.

[0077] Step 4: Comprehensively process the prediction results of the blood oxygen concentration estimation model based on mathematical principles and the blood oxygen concentration estimation model based on data-driven methods, and output the dual model fusion result.

[0078] The design concept of the blood oxygen fusion module is to combine the prediction results of the mathematical blood oxygen estimation model and the data-driven blood oxygen estimation model to achieve higher prediction accuracy and stronger robustness. Specifically, the module will dynamically calculate the weights based on the blood oxygen output of the two models using the following formula: w 1 and w 2 ,satisfy w 1 + w 2 =1: (25) SPO2 phy Represents the blood oxygen saturation value output by the mathematical blood oxygen estimation model, SPO2 pre Represents the blood oxygen saturation value output by the data-driven blood oxygen estimation model, applying these weights w 1and w 2 The prediction results of the two models are weighted and fused to output the final blood oxygen concentration estimate. This weight distribution mechanism ensures that the more accurate and reliable model can occupy a larger proportion in different scenarios, thereby improving the overall prediction performance.

[0079] In addition, the blood oxygen fusion module also has an adaptive adjustment function. The module can dynamically adjust the weight distribution according to the changes in real-time monitoring data. This function enables the blood oxygen fusion module to flexibly respond to changes in blood oxygen under different individuals and environmental conditions, further improving the adaptability and flexibility of the system. Through this fusion mechanism, the blood oxygen fusion module not only significantly improves the accuracy of blood oxygen monitoring, but also ensures that the system can still maintain an efficient and stable working state in the face of various complex situations. This comprehensive processing method not only optimizes the limitations of a single model, but also provides new ideas and directions for the development of blood oxygen monitoring technology.

[0080] Experimental methods, experimental results and analysis: The dataset we used for training was composed of videos and blood oxygen data collected by ourselves, including 9 boys and 1 girl. Each of them collected two videos, one for free breathing and the other for holding their breath. We required the test subjects to remain still during both collections. This part of the dataset is not public, but in order to compare with other researchers' methods, we also used the public dataset PURE for testing. The public dataset is purely for testing and is not used for training. The relevant information of the PURE dataset is shown in Table 1: Table 1. Relevant information of PURE dataset

[0081]

[0082] The blood oxygen distribution of the PURE data set is between 89-99. We made a histogram of different blood oxygen levels to facilitate the observation of the specific blood oxygen data distribution. The specific distribution histogram is as follows: Figure 7 shown.

[0083] 1. Experimental setup This experiment is divided into three parts: rPPG signal acquisition, preprocessing, and blood oxygen prediction. The rPPG signal acquisition process takes a long time, which forces us to think about how to reduce the time occupancy. We experimented with the method of reducing the resolution of the input face data. We input the processed videos of different resolutions into the rPPG signal acquisition module, and the performance of the trained model on the test set is shown in Table 2. It is found that when the input is 224*224, the processing speed can reach the input speed, and the accuracy loss is not much, which is the most suitable choice.

[0084] Table 2 Comparison experiment of input size

[0085] In this experiment, we used 5-fold cross validation, dividing 10 people into 5 groups, four groups for training, and the remaining group for testing. The experimental results are as follows: Figure 8a , Figure 8b shown.

[0086] After resampling the data set and enhancing the data, it was achieved Fig. 9 As shown in the figure, the drop in blood oxygen at 100 seconds, 150 seconds, and 270 seconds is predictable, which is more generalized than the previous results. However, there is still a situation where the abnormal drop in blood oxygen at 50 seconds and 225 seconds is not perceived, which is also an area that needs to be improved in future work.

[0087] 2. Evaluation Metrics In order to verify the effectiveness of the model, the mean absolute error (MAE) and root mean square error (RMSE) are used to represent the experimental results. The mean absolute error (MAE) represents the average of the absolute value of the difference between the predicted value and the actual value. The formula is as follows: (20) The root mean square error (RMSE) is the square root of the average of the squared prediction errors and is given by the following formula: (twenty one) Here SPO2 pre,j Indicates j Predicted blood oxygen saturation values, SPO2 gt,j Indicates j A reference (true) blood oxygen saturation value, n is the total number of samples.

[0088] MAE gives the average size of the prediction error, while RMSE gives the standard deviation of the prediction error. RMSE gives greater weight to large prediction errors, so RMSE is generally considered to be a better indicator of the severity of model errors than MAE.

[0089] 3. Ablation experiment In order to evaluate the application of our proposed channel attention and temporal attention on 1DCNN, which saves time cost while retaining the accuracy improvement, we conducted an ablation experiment. We compared four cases, namely: (1) Using 2DCNN’s 7-layer residual block alone (2) Using 2DCNN’s 7-layer residual block + channel-wise attention (3) Using 1DCNN’s 7-layer residual block alone (4) Using 1DCNN’s 7-layer residual block + channel and temporal attention The results show that the blood oxygen estimation effect is best when using 1DCNN's 7-layer residual block + channel and temporal attention on the test set.

[0090] Table 3 Ablation experiment results

[0091] 4. Comparison with other models We collected representative paper results from recent years for comparison, as shown in Figure 3. Due to the particularity of video datasets, which contain a lot of sensitive facial information, most researchers conduct experiments on self-built datasets. Due to the differences in datasets, it is difficult to compare model results. The * in Table 4 is the result of other researchers reproducing the code of these authors and running it on PURE, and - indicates experiments not conducted by the authors.

[0092] According to the comparison in Table 4, we can find that the result obtained by our model on the self-built dataset has the smallest mean absolute error among all models. However, because we did not use some data in PURE as part of the training set, our result on PURE is not the best. However, we can also make the model obtain relatively stable results in the cases of stillness, conversation and slow movement.

[0093] Table 4 Comparison with other models

[0094] 5. Summary This paper successfully developed and verified a blood oxygen saturation (SpO2) monitoring method based on non-contact video analysis. By adopting advanced remote photoplethysmography (rPPG) technology and deep learning models, we have overcome the limitations of traditional contact blood oxygen monitoring in specific populations and improved the accuracy and convenience of monitoring.

[0095] The present invention combines rPPG signal acquisition, data preprocessing, feature extraction and deep learning models to form a complete blood oxygen monitoring process. The proposed RAOEN network structure, including residual blocks and attention mechanisms, effectively improves the prediction accuracy of blood oxygen saturation. The generalization ability and practicality of the model are verified by testing the self-built dataset and the public dataset PURE. The model performance is comprehensively evaluated by indicators such as MAE and RMSE, showing that the model has a small mean absolute error on the self-built dataset.

[0096] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A multi-feature fusion blood oxygen estimation method based on visible light video, characterized in that: The steps include: Step 1: rPPG signal acquisition and preprocessing: extract the remote photoplethysmography (rPPG) signal by monitoring the numerical changes of the average pixels in different areas of the face, and perform data preprocessing to enhance the face video data related to blood oxygen; Step 2: Construct a mathematical-based blood oxygen estimation model: Based on the method of evaluating rPPG signals, the features of 6 rPPG signals of each channel were selected, and a mathematical-based method was designed to manually extract these features and use them in the blood oxygen detection process; Step 3: Build a data-driven blood oxygen estimation model: extract the characteristic signals of physiological signals from the cleaned rPPG signal and use them to calculate blood oxygen; Step 4: Comprehensively process the prediction results of the blood oxygen concentration estimation model based on mathematical principles and the blood oxygen concentration estimation model based on data-driven methods, and output the dual model fusion result.

2. The multi-feature fusion blood oxygen estimation method based on visible light video according to claim 1 is characterized in that: The step 1 comprises the following specific steps: Step 1.1: rPPG signal acquisition: 12 facial regions were selected as regions of interest (ROIs); Step 1.2: Data preprocessing; Step 1.3: Enhance the blood oxygen-related face video data.

3. The multi-feature fusion blood oxygen estimation method based on visible light video according to claim 2 is characterized in that: The step 1.2 includes the following specific steps: By using the fast Fourier transform (FFT) combined with a bandpass filter, the filter cutoff frequency range is designed to be 0.5 to 4 Hz, which covers the spectrum of human heart rate and its harmonic frequencies. Fast Fourier filtering is divided into 4 steps: Fast Fourier Transform: （1） in x ( t ) represents the input signal in the time domain, FFT represents the fast Fourier transform, Represents the input signal in the frequency domain; Bandpass filter settings: （2） Indicates the mask setting of the bandpass filter, indicating the frequency The signals between are retained, and the signals in the rest range are discarded; Frequency domain filtering: （3） Will The mask setting is applied to the input signal in the frequency domain to obtain a signal that only retains the frequency range we need. ; Inverse Fast Fourier Transform: （4） Will Converted into a signal in the time domain through inverse Fourier transform y ( t ); These filtered rPPG signals were normalized, interpolated, and mean filtered: （5） The above formula represents the normalized formula, where Represents the original signal y ( t ), Represents the original signal y The variance in (t); （6） The above formula represents the interpolation formula, where t Indicates the signal position that needs to be interpolated, t 0. t 1 means that the known signal positions on both sides need to be interpolated. y ( t 0), y ( t 1) Indicates t 0. t 1. Signal strength at location; （7） The above formula is a formula for average filtering. In range I ( t ) to find the average, where I’ ( t ) represents the average filtering t Position signal, I ( t+u ) indicates that t Location distance u signal strength.

4. The multi-feature fusion blood oxygen estimation method based on visible light video according to claim 2 is characterized in that: The step 1.3 includes the following specific steps: By collecting face videos under different blood oxygen levels, and applying image processing technology to simulate the visual effects under different blood oxygen conditions, including color adjustment, brightness change, horizontal scaling, multi-angle rotation, flipping, image blur, noise, geometric distortion and optical noise enhancement, and on this basis, a combination of multiple data enhancement methods is used to expand the data set: Color adjustment: By analyzing the changes in blood oxygen saturation, the color of the video frame is adjusted to simulate the face video effect shot under different ambient lighting conditions. The technical means of converting RGB color space to HSV color space is adopted: in the HSV color space, the color tone is adjusted to simulate the visual effects under different color lights; Brightness adjustment: By adjusting the brightness and contrast of the video frame, the effect of changes in ambient light brightness on the face video is simulated to enhance the model's learning ability for scenes with drastic changes in ambient light. In the brightness adjustment process, the method of adjusting the hue after HSV color space conversion is used to achieve brightness changes; Horizontal scaling: To simulate the dynamic effect of a face swaying left and right, the size of the video frame is adjusted. The cv2.resize function in the OpenCV library is used to select the scaling parameters and then scale the image to achieve the desired scaling effect. Multi-angle rotation: When simulating the head shaking movement, the implementation of the rotation operation is involved. By calling the cv2.getRotationMatrix2D(center, angle, 1.0) function, a rotation matrix is ​​constructed. Then, the cv2.warpAffine function is used to apply the rotation matrix to the original image to perform an affine transformation to obtain the rotated image. Flip: simulate the left-right reversal during video recording, using the transforms.RandomHorizontalFlip method; Image blur: use the Gaussian blur effect, using the GaussianBlur function in the transforms library, and accurately control the intensity and range of the blur effect by accurately setting the kernel size and standard deviation, thereby simulating the image visual effects under object occlusion, low pixel resolution, and severe weather conditions; Adding noise: Using image processing technology, add random noise to the image artificially. Use the ColorJitter method in the transforms module to adjust the brightness and contrast parameters to achieve the purpose of simulating noise. Geometric distortion: simulates the dynamic changes of the face during oral expression, so that the model can learn the deformation rules of the facial skin during speech expression. This process is achieved with the help of the transforms.RandomAffine method, where the shear parameter is used to adjust the strength of the shear transformation.

5. The multi-feature fusion blood oxygen estimation method based on visible light video according to claim 1, characterized in that: The step 2 comprises the following specific steps: Using the mean of a 10-second rPPG signal as the DC component and the variance as the AC component, a mathematical model-based method is designed to manually extract the features of the DC component and the AC component and use them in the blood oxygen detection process. According to the method of evaluating rPPG signals, 6 features of rPPG signals of each channel are selected, which are: ① Waveform S+ from trough to trough; ② The peak-to-peak waveform S-; ③The height difference H+ between two consecutive troughs; ④The height difference between two consecutive peaks H-; ⑤The time difference between the peak and the trough T1; ⑥ The height difference between the peak and the trough T2; And use the rPPG signal quality assessment algorithm on the data preprocessed signal to judge the signal quality, with a score between 0 and 1, where 0 means very poor and 1 means very good; Select rPPG signals with better quality for feature extraction. The selected signals with better quality must meet the following requirements:

1. There are two obvious peaks and troughs within the duration; 2. The rPPG signal quality is above 0.6 points; Finally, add the DC and AC features of each channel: mean and , different channels are marked with subscripts of r, g, and b, with a total of 24 features, as shown in formula (8): （8） And the physiological signal features obtained after multiple linear layers and activation layers f phy , its formula is expressed as: f phy =ReLU( W 2×ReLU( W 1× C + b 1)+ b 2) (9) in, W 1, W 2, b 1, b 2 represent the weights and biases of different linear layers, ReLU is the activation function, and C is the manually extracted feature signal of the input.

6. The multi-feature fusion blood oxygen estimation method based on visible light video according to claim 1, characterized in that: The step 3 comprises the following specific steps: Step 3.1: Construct a data-driven blood oxygen estimation model structure framework: used to extract the characteristic signals of physiological signals from the cleaned rPPG signals and use them to calculate blood oxygen; Step 3.2: Group and resample the collected data: adjust the number of samples in the minority category and the majority category to achieve a balanced state of the data set; Step 3.3: Propose a targeted loss calculation method, namely weighted loss.

7. The multi-feature fusion blood oxygen estimation method based on visible light video according to claim 6, characterized in that: The step 3.1 includes the following specific steps: The data-driven blood oxygen estimation model includes a residual module and two attention modules, wherein the two attention modules are a channel attention module and a spatial attention module; The reflected photoplethysmogram (rPPG) signals obtained from the face, after being aligned in a preprocessing step, are then passed through a residual module backbone network for feature extraction. The design of the residual module backbone network introduces a regularization mechanism that allows the network to learn a residual function, i.e., the residual between the input and the output; Each residual block uses three one-dimensional convolutional layers to achieve feature dimensionality reduction, extraction, and dimensionality increase. The specific formula is expressed as: RB( x )= W 3*ReLU( W 2*ReLU( W 1* x )) (10) in, W 1, W 2, W 3 represent the weights of different convolutional layers, * represents the convolution operation, ReLU is the activation function, and x Is the input feature signal, the first convolutional layer W 1 is responsible for dimensionality reduction to reduce the amount of calculation; the second convolution layer W 2 Perform feature extraction; the third convolution layer W 3 is responsible for dimensionality increase and restoring the dimension of the feature map. The sizes of the three convolution kernels are 1, 3, and 1 respectively; When the residual blocks are used in cascade, two different configurations are distinguished. In the first case, the number of channels needs to be adjusted. Formula F1 is expressed as: F1=RB( x )+ W s * x (11) Among them, RB represents the single residual block operation process, see formula (10), and W s is the weight matrix used to adjust the residual connection to match the number of channels. At the same time, the residual connection ensures that the input feature signal x Flow; In the second case, there is no need to adjust the number of channels, and the formula F2 is expressed as: <h2 style=";text-align:left;direction:ltr">F2=RB(<h2 style=";text-align:left;direction:ltr"> x <h2 style=";text-align:left;direction:ltr"> )+<h2 style=";text-align:left;direction:ltr"> x <h2 style=";text-align:left;direction:ltr"> (12) That is, the residual connection is directly applied to the original input feature map x, and the whole residual module extracts the feature map process Res( x ) is expressed by the following formula: （13） Res( x ) represents the result of feature extraction of the residual module as the backbone network, It does not mean to sum, but to repeat the process seven times; The channel attention mechanism and the spatial attention mechanism are closely connected in a front-to-back manner. Subsequently, the combination is connected to a three-layer residual network, which is used to increase or reduce the dimension of the channel to achieve effective feature extraction. This process is called a single attention extraction. In the subsequent steps, the attention extraction process is repeated twice. The first attention extraction increases the number of channels from 32 dimensions to 64 dimensions to enhance the model's ability to capture features; then, the second attention extraction reduces the number of channels from 64 dimensions to 32 dimensions to simplify the model and reduce computational complexity. The channel attention mechanism automatically learns and emphasizes the channels that contribute more to the blood oxygen signal through the following formula, while suppressing the channels that are disturbed by noise or ambient light: CA( x )=Sigmoid( g ( x ))⊙Rex( x ) (14) Among them, Rex( x ) indicates that the result of feature extraction of the residual module of the backbone network is used as the input feature signal, Sigmoid indicates the Sigmoid activation function, ⊙ indicates the element-by-element multiplication of each channel, g ( x ) represents the features after being processed by global pooling and convolutional layers, and its formula is expressed as: g( x )=Conv(ReLU(Conv ( avg / max pool (Rex ( x ))))) (15) in, avg / maxpool (Rex ( x )) is a reference to Rex ( x ) performs global average pooling or maximum pooling, Conv is a one-dimensional convolutional layer for dimensionality reduction and dimensionality increase, The spatial attention mechanism is introduced. The spatial attention mechanism dynamically identifies and emphasizes the key segments in the time series for blood oxygen value prediction through the following formula: SA(x)= Sigmoid ( s ( x )) ⊙Rex ( x ) (16) here, s ( x ) is the feature processed by the operation on the time series and the convolution layer, and its formula is expressed as: s ( x )=Conv( concat ( avg pool ( x ), max pool (Rex ( x )))) (17) avg pool ( x )and max pool ( x ) are respectively for Rex ( x ) for average pooling and maximum pooling, concat It connects the two pooling results in the channel dimension. Conv is a one-dimensional convolutional layer used to learn spatial weights. The output CSA combining the spatial attention mechanism and the channel attention mechanism is expressed as: CSA( x )=SA(CA( x )) (18) Finally, the feature maps are flattened and converted into one-dimensional signals for further processing in the fully connected layer, where they are used to generate the final prediction results. SpO2 pre , its formula is expressed as: SpO2 pre =FC3(View(CSA( x ))) (19) In this process, the first operation is the View operation, which expands the input data into a one-dimensional signal. This operation is to convert the original data into a form suitable for the fully connected layer to ensure that the data can be correctly processed by the fully connected layer. Then, the data will be processed by the FC3 operation, that is, the three-layer fully connected layer, and finally the blood oxygen prediction result will be output. SpO2 pre .

8. The multi-feature fusion blood oxygen estimation method based on visible light video according to claim 6, characterized in that: The step 3.2 includes the following specific steps: The sampler is used to load data, and a group resampling strategy is proposed. The strategy resamples data based on the proportion of the total sample amount, and performs multiple sampling on categories with less data. The specific steps of the group resampling are as follows: First, the blood oxygen value is divided into several intervals. The blood oxygen value is subdivided into 20 different intervals from 81 to 100; Secondly, the number of samples in each interval is counted. Before model training, the number of samples in each interval in the training set is counted in detail to identify the intervals with fewer and more samples, and record these intervals in the resampling grouping, that is, oversampling and undersampling operations are performed on the intervals with extreme sample numbers; Next, the resampling ratio is set. This ratio is dynamically calculated based on the number of samples in each interval. Finally, the number of samples in each interval with a small number of samples is increased to 2% of the total number of samples, and the number of samples in the interval with a large number of samples is reduced to 5% of the total number of samples. Finally, according to the actual situation of the data set, choose the appropriate oversampling or undersampling method; Oversampling strategy: For intervals with a small number of samples, the number of samples is expanded by copying existing samples or using SMOTE technology to generate new samples; Undersampling strategy: For intervals with a large number of samples, some samples are randomly removed to reduce the number of samples; A dynamic adjustment mechanism is introduced to appropriately perform oversampling or undersampling according to the actual situation of each grouped sample.

9. The multi-feature fusion blood oxygen estimation method based on visible light video according to claim 6, characterized in that: The step 3.3 includes the following specific steps: Two widely used loss functions are used: mean absolute error (MAE) and root mean square error (RMSE), and the two loss functions are weighted summed to obtain the loss : (20) (21) Here SPO2 pre,j Represents the model j Predicted blood oxygen saturation values, SPO2 gt,j Indicates j A reference blood oxygen saturation value, n is the total number of samples; （22） , Represent the loss coefficients of MAE and RMSE respectively, The larger the weight coefficient, the , ; Using weighted loss method: First, the blood oxygen values ​​were divided into 20 different groups ranging from 81 to 100. Before training the model, the data in the training set were grouped and counted, and the number of samples in each group was recorded; Then, during model training, the loss weight for each group is calculated: （23） w i Indicates i The weight of each group, N represents the total number of samples in the training set, and N i It means the i The number of samples in each group; Finally, this weight is applied to the i The loss of the classification LOSS pre，i The final loss LOSS pre , the calculation method is as follows: （24） Ensure that the model gives different weights to samples from different groups during training.

10. The multi-feature fusion blood oxygen estimation method based on visible light video according to claim 1, characterized in that: The step 4 comprises the following specific steps: The prediction results of the blood oxygen estimation model based on mathematics and the blood oxygen estimation model based on data-driven methods are comprehensively processed. According to the blood oxygen output of the two models, the weight is dynamically calculated using the following formula w 1 and w 2 ,satisfy w 1 + w 2 =1: （25） SPO2 phy Represents the blood oxygen saturation value output by the mathematical blood oxygen estimation model, SPO2 pre Represents the blood oxygen saturation value output by the data-driven blood oxygen estimation model, applying these weights w 1 and w 2 The prediction results of the two models are weighted fused to output the final blood oxygen concentration estimate.

Images