Blood oxygen saturation measuring method based on smart phone camera

Abstract

The invention belongs to the field of artificial intelligence and contact type measurement, and discloses a blood oxygen saturation degree measuring method based on a smart phone camera. The method comprises the following steps: acquiring fingertip video data by using a smart phone under the condition of no external equipment, extracting a pulse photoplethysmography (PPG) signal from the fingertip video data, and constructing a multi-dimensional feature set through feature extraction and calculation; furthermore, the invention provides a two-channel deep learning model fusing a convolutional neural network and a long-short term memory network, the multi-dimensional feature set and the preprocessed PPG signal are processed in parallel, the advantages of the two networks in the aspects of feature extraction and time sequence modeling are fully exerted, and the accuracy of feature extraction and time sequence modeling is improved. And the sufficiency of the extracted features and the generalization ability of the model are ensured, so that the accurate estimation of the oxyhemoglobin saturation is realized. The research provides a feasible scheme for health self-examination in a family environment.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and contact measurement, and relates to a method for measuring blood oxygen saturation based on a smartphone camera. Background Technology

[0002] Blood oxygen saturation refers to the percentage of oxyhemoglobin in the total hemoglobin in peripheral capillary blood, reflecting the blood's oxygen-carrying capacity. Low blood oxygen saturation leads to hypoxemia, a common complication of various clinical diseases. Hypoxemia can cause a range of serious health problems and damage multiple organ systems, particularly the brain and heart. During the COVID-19 pandemic, it was observed that some patients had significantly reduced blood oxygen levels before experiencing obvious shortness of breath (a phenomenon known as "silent hypoxia"). For high-risk individuals, continuous monitoring of oxygen levels helps in the timely detection of lung abnormalities, allowing for early treatment. Although this phenomenon has now ended, it highlights the important value of regular blood oxygen saturation monitoring. Regularly monitoring blood oxygen saturation in daily life helps in understanding one's health status and avoiding delays in treatment. For patients with lung and heart diseases, close monitoring of blood oxygen saturation is a crucial means of managing the disease and ensuring treatment effectiveness.

[0003] The widespread adoption of smartphones and advancements in sensor technology have provided new avenues for real-time blood oxygen saturation monitoring. Traditionally, gas chromatography has been the gold standard for measuring blood oxygen levels, requiring puncture blood collection and relying on specialized clinicians and medical equipment. In contrast, pulse oximeters have become a widely adopted non-invasive measurement tool, and many wearable devices (such as smartwatches) have integrated blood oxygen detection functionality. Smartphone-based blood oxygen detection solutions utilize the phone's built-in camera and flash to capture changes in light signals reflected from the finger, which are correlated with blood oxygen levels. By analyzing the signals using specific algorithms, blood oxygen saturation values ​​can be estimated. This method not only provides users with a convenient and safe measurement experience but also significantly improves comfort.

[0004] Smartphones have become necessities in daily life. In contrast, pulse oximeters and wearable devices need to be purchased separately, and the latter are updated more frequently. The deep integration of smartphones into daily life allows users to take measurements anytime, anywhere, effectively protecting user privacy and supporting instant detection in various indoor and outdoor scenarios. These characteristics demonstrate the development potential of smartphone-based pulse oximetry.

[0005] Smartphone-based blood oxygen monitoring solutions hold promise as a beneficial health monitoring tool in daily life. They help users more easily monitor their health indicators, alleviating strain on medical resources and reducing the burden of medical care to some extent. In particular, they provide a convenient health monitoring method for areas with relatively scarce medical resources, which has positive implications for improving public health. Summary of the Invention

[0006] This invention proposes a method for measuring blood oxygen saturation based on a smartphone camera, aiming to create a convenient and universally applicable new way to monitor blood oxygen levels in daily environments without the need for wearing any other devices. First, the user's fingertip light reflection signal is acquired using the smartphone camera with the aid of a flash. After processing, a photoplethysmography (PPG) signal with three RGB channels is obtained. In the signal processing stage, motion artifact removal and filtering methods are used to preprocess the raw PPG signal to improve signal quality. Subsequently, feature points and biomarkers are extracted from the PPG signal, and a feature set is constructed by combining the calculated AC and DC components and their ratio features. To improve the accuracy of blood oxygen estimation, this invention employs a dual-channel deep learning architecture based on a convolutional neural network (CNN) and a long short-term memory network (LSTM) to combine the advantages of both types of networks and reduce measurement errors. This method enables high-precision blood oxygen measurement in a low-cost and convenient manner, providing users with an economical and easy-to-use option.

[0007] The technical solution of this invention is as follows:

[0008] (1) Using the built-in camera and flash of a smartphone, without wearing any additional equipment, the user's finger completely covers the camera and flash to capture and obtain fingertip video data and obtain photoplethysmography (PPG) signals.

[0009] (2) Motion artifact removal and filtering preprocessing are performed on the original PPG signal using the SVD algorithm and bandpass filter. Relevant features are extracted and calculated, a multidimensional feature set is constructed, and it is divided into a training set and a test set. The preprocessed PPG signal and the multidimensional feature set are used as the input of the model, respectively.

[0010] (3) Based on different data input modalities, a dual-channel deep learning model was constructed. The first channel uses a convolutional neural network to process multi-dimensional feature sets. Its structure includes a 2D convolutional layer and two 1D convolutional layers. Each convolutional layer is followed by a batch normalization layer and a dropout layer, and the ReLU activation function is used. The last layer of the convolutional neural network is a fully connected layer. The second channel uses a long short-term memory network (LSTM) to process the preprocessed PPG signal. It contains two LSTM layers, each of which is followed by a dropout layer. The ReLU activation function is also used. The last layer of the long short-term memory network is a fully connected layer. The outputs of the two channels are finally integrated through a fusion layer. This fusion layer contains two linear fully connected layers, which use the ReLU and Softmax activation functions, respectively.

[0011] (4) Use leave-one-out cross-validation to train and test the dual-channel model built in step (3), perform hyperparameter tuning and iterative optimization, and save the optimal model that minimizes the mean absolute error (MAE).

[0012] The present invention has the following beneficial effects:

[0013] (1) In this invention, the built-in camera and flash of a smartphone are used to measure blood oxygen levels anytime and anywhere without wearing any external devices.

[0014] (2) The present invention constructs a multidimensional feature set, which not only includes the AC and DC components of the three-color channels that reflect the light absorption differences of hemoglobin, but also incorporates important feature points and biomarkers in the PPG signal; the construction of this multidimensional feature set can better adapt to individual differences of users and improve the accuracy of blood oxygen estimation.

[0015] (3) By deeply exploring and utilizing the fingertip video signal, this invention achieves accurate prediction of blood oxygen saturation in the range of 70%-100%, providing users with a convenient means of self-monitoring. Attached Figure Description

[0016] Figure 1 This is a model structure diagram of the present invention;

[0017] Figure 2 This is the model network structure diagram for feature sets in this invention, corresponding to... Figure 1 The channel model based on multidimensional feature sets in [the text].

[0018] Figure 3 This is a model network structure diagram of the preprocessed PPG signal used in this invention, corresponding to... Figure 1 The channel model is based on the preprocessed PPG signal; Detailed Implementation

[0019] To make the objectives, technical solutions, and beneficial effects of the present invention clearer, the implementation methods of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0020] A method for estimating blood oxygen saturation based on a smartphone camera, such as Figure 1 As shown. The specific implementation process is as follows:

[0021] Step 1: The user first enters gender and age information. Using the built-in camera and flash of the smartphone, without wearing any additional equipment, the user's finger completely covers the camera and flash to capture fingertip video data. The RGB three-channel photoplethysmography (PPG) signal of each frame of the video is acquired in real time through the CameraX interface. The raw data acquired through the smartphone camera is preprocessed to remove motion artifacts and filter the data. PPG features are then extracted and calculated to construct a multi-dimensional feature set and divide the data into training and test sets.

[0022] For the original PPG signals of the three RGB channels, we used the SVD algorithm to remove motion artifacts and a fourth-order Butterworth bandpass filter (0.5-12Hz) for noise reduction. Subsequently, we used the pyPPG library to extract feature points and biomarkers from the PPG signals and calculated the AC and DC components of each channel. The AC component is represented by the average amplitude within the signal window, and the DC component is the average signal within the window. We calculated the AC / DC ratio (ratio R) of each color channel and the pairwise ratios of R values ​​between different channels. These were combined with the feature points and biomarkers to form a multidimensional feature set. The preprocessed signals and feature sets were then divided into training and test sets. Data from one subject was retained for testing, and the remaining data were used for training.

[0023] Step 2: Build a deep learning blood oxygen saturation estimation model based on fingertip video signals, such as... Figure 2 As shown in Figure 3;

[0024] We designed a dual-channel parallel processing network structure for different input data modalities, including a convolutional neural network based on preprocessed PPG signals and a long short-term memory network for processing multi-dimensional feature sets.

[0025] The channel model network structure based on the preprocessed PPG signal is as follows: Figure 2As shown, the network mainly consists of one 2D convolutional layer and two 1D convolutional layers. Each convolutional layer is followed by a batch normalization layer and a dropout layer. The number of convolutional kernels is 32, 64, and 64 respectively, and the stride of each kernel is set to 1. Each convolutional layer uses the ReLU activation function. The last layer of the convolutional neural network is a fully connected layer. In order to suppress overfitting and reduce computational complexity, the parameter of the Dropout layer in the network is set to 0.4.

[0026] Long Short-Term Memory (LSTM) network structures based on multidimensional feature sets, such as... Figure 3 As shown, it mainly consists of two LSTM layers, each containing 128 units. Each LSTM layer is connected to a dropout layer and uses ReLU as the activation function. The last layer of the Long Short-Term Memory network is a fully connected layer. The parameters of the Dropout layer introduced in this network are set to 0.2.

[0027] The outputs of the two channels are finally integrated in a fusion layer; this fusion layer contains two linear fully connected layers, using ReLU and Sigmoid as activation functions respectively; the final output is constrained to the range of 60-105 (the true range is 70-100) to help alleviate the gradient vanishing problem.

[0028] Step 3: Train and test the model using leave-one-out cross-validation. During model training, MSELoss was used as the loss function. The optimizer chosen was Adam, with an initial learning rate of 0.005. The Adam optimizer can adaptively update the learning rate. The training epochs were set to 100, and the batch size to 32. The model performance was evaluated using the average mean absolute error (MAE), and the model with the lowest average MAE was selected as the optimal model.

Claims

1. A method for measuring blood oxygen saturation based on a smartphone camera, characterized in that, Includes the following steps: (1) Using the built-in camera and flash of a smartphone, without wearing any additional equipment, the user’s finger completely covers the camera and flash to capture and obtain fingertip video data and obtain photoplethysmography (PPG) signal. (2) Motion artifact removal and filtering preprocessing are performed on the original PPG signal using the SVD algorithm and bandpass filter. Relevant features are extracted and calculated, a multidimensional feature set is constructed, and it is divided into a training set and a test set. The preprocessed PPG signal and the multidimensional feature set are used as the input of the model, respectively. (3) Based on different data input modalities, a dual-channel deep learning model was constructed. The first channel uses a convolutional neural network to process multi-dimensional feature sets. Its structure includes a 2D convolutional layer and two 1D convolutional layers. Each convolutional layer is followed by a batch normalization layer and a dropout layer, and the ReLU activation function is used. The last layer of the convolutional neural network is a fully connected layer. The second channel uses a long short-term memory network (LSTM) to process the preprocessed PPG signal. It includes two LSTM layers, each of which is followed by a dropout layer. The ReLU activation function is also used. The last layer of the long short-term memory network is a fully connected layer. The outputs of the two channels are finally integrated through a fusion layer. This fusion layer includes two linear fully connected layers, which use the ReLU and Softmax activation functions, respectively. (4) Use leave-one-out cross-validation to train and test the dual-channel model built in step (3), perform hyperparameter adjustment and iterative optimization, and save the optimal model that minimizes the mean absolute error (MAE).

2. The method for measuring blood oxygen saturation based on a smartphone camera according to claim 1, characterized in that, In step (2), the specific steps for constructing the multidimensional feature set are as follows: use the pyPPG library to extract feature points and biomarkers from the preprocessed PPG signal; at the same time, calculate the AC component and DC component of each color channel: the AC component is represented by the average amplitude within the signal window, and the DC component is taken as the average value of the signal within the same window; further calculate the AC / DC ratio of each color channel, i.e., the ratio R, and the pairwise ratio of R values ​​between different channels; finally, combine the feature points, biomarkers, and all calculated ratios into a multidimensional feature set.

Images