Deep learning based contactless atrial fibrillation recognition and multi-physiological index monitoring system and method

Abstract

This application relates to a contactless atrial fibrillation identification and multi-physiological indicator monitoring system and method based on deep learning. The system includes: extracting pulse wave signals from a user's facial video; extracting facial temporal feature sequences from the user's facial video; assigning attention weights to the pulse wave signals and the facial temporal feature sequences and performing weighted processing; and predicting the probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level based on the weighted pulse wave signals and facial temporal features. This solution can effectively improve the robustness and accuracy of atrial fibrillation detection, while simultaneously monitoring multiple physiological indicators of the user.

Description

Technical Field

[0001] This application relates to computer-aided medical data processing, and more particularly to a deep learning-based system and method for atrial fibrillation identification and multi-physiological indicator monitoring. Background Technology

[0002] Atrial fibrillation (AF) is the most common cardiac arrhythmia. When it progresses to a certain stage, it poses a serious risk of complications such as stroke, heart failure, and death. However, atrial fibrillation (hereinafter referred to as AF) is usually asymptomatic and occurs in paroxysmal episodes, often being overlooked clinically. Therefore, early screening and detection of AF is of great significance, as it facilitates timely drug treatment, interventional therapy, and other treatments, thereby effectively controlling the condition and reducing the occurrence of complications.

[0003] Traditional atrial fibrillation detection typically requires multiple specialized medical instruments for electrocardiogram (ECG) monitoring, blood pressure measurement, and blood oxygenation. Each instrument is costly and has limited accessibility, making them unsuitable for long-term continuous monitoring. Furthermore, they are expensive, and some methods are invasive (e.g., implantable circulatory recorders and Holter monitor patches). While handheld ECG devices offer convenience for intermittent screening, they remain impractical for long-term monitoring.

[0004] In recent years, atrial fibrillation detection technology based on remote photoplethysmography (rPPG) has attracted widespread attention. rPPG is a non-contact monitoring technology that uses a regular camera to capture changes in light reflected from the skin to monitor vital signs. Its principle is based on a model of the reflection and absorption of natural light on the skin. When natural light shines on the skin, some light is directly reflected by the epidermal tissue, while the rest is absorbed by hemoglobin in the capillaries and diffusely reflected. Each contraction and relaxation of the heart causes a change in the concentration of hemoglobin in the blood, resulting in a subtle change in skin color. This faint change in skin color can be captured by a high-resolution camera (such as a mobile phone or computer webcam), thereby obtaining the pulse wave signal (hereinafter referred to as the rPPG signal). Then, physiological information is extracted by analyzing the morphology and characteristics of the rPPG signal. Therefore, the rPPG-based atrial fibrillation detection method does not require contact with the human body or specialized medical instruments. It can be monitored using a regular camera. It extracts rPPG signals from a facial video uploaded by the user, performs feature extraction and classification on the rPPG signals, and thus achieves atrial fibrillation detection.

[0005] It should be noted that the above content is only used to help understand the technical solution of this application and should not be used as a basis for evaluating the prior art of this application. Summary of the Invention

[0006] However, the inventors found in practice that this rPPG-based atrial fibrillation and physiological signal monitoring is significantly affected by the environment and the monitored subject, and its accuracy and robustness need improvement. This is because this monitoring method typically requires the monitored subject to remain still and the ambient lighting to be stable, thus recording a relatively static facial video of the subject over a period of time. However, in practical applications, slight movements due to changes in ambient light or decreased patience and emotional fluctuations of the monitored subject often result in short-term movements or changes in facial expressions in the recorded facial video. This introduces unwanted noise into the captured rPPG signal, potentially causing confusion when distinguishing abnormal segments from genuine atrial fibrillation segments during rPPG signal processing, thus affecting detection accuracy. Furthermore, this method is clearly unsuitable for monitoring atrial fibrillation in moving subjects.

[0007] The purpose of this application is to provide a more accurate and robust rPPG-based atrial fibrillation and physiological signal monitoring scheme.

[0008] The above objective is achieved through the following technical solution:

[0009] According to a first aspect of the embodiments of this application, a deep learning-based physiological monitoring system is provided. The system includes one or more recognition models, each comprising: a signal extraction module for extracting pulse wave signals from a user's facial video; a facial feature extraction module for extracting facial temporal feature sequences from the user's facial video; a feature-level attention mechanism module for assigning attention weights to the pulse wave signals and the facial temporal feature sequences and performing weighted processing; and a classification module for outputting physiological monitoring results based on the weighted pulse wave signals and facial temporal features, wherein the physiological monitoring results include predictions for one of the following: the probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level.

[0010] In some embodiments, the one or more recognition models may be trained independently and used to output predictions of the probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level based on the input user's facial video; and the system may also include an integration module for receiving the user's facial video and simultaneously feeding it to each recognition model, as well as summarizing and outputting the prediction results of each recognition model.

[0011] In some embodiments, the signal extraction module may be configured to extract corresponding pulse wave signals from one or more of the following facial regions of interest: forehead, left cheek, and right cheek.

[0012] In some embodiments, the signal extraction module may be configured to determine the region of interest (ROI) of a face by extracting key points characterizing the facial shape from the detected facial region in each video frame; and determining the ROI based on the location of these key points in the facial region and the size of the facial region.

[0013] In some embodiments, the signal extraction module may also be configured to use a fourth-order Chebyshev type II bandpass filter to filter the extracted pulse wave signal.

[0014] In some embodiments, the facial feature extraction module may include a multi-scale convolutional feature extraction and multi-scale feature transformation module, wherein for each video frame from the input facial video data, the multi-scale convolutional feature extraction module extracts convolutional features from the video frame using a 2D convolutional neural network and three sets of different convolutional kernel sizes, and concatenates these three sets of convolutional features into a feature map in the channel dimension; the feature transformation module continues to extract facial features from the feature map using three sets of different convolutional kernel sizes, and further transforms the extracted three sets of facial features into a one-dimensional feature vector after concatenating them in the channel dimension, and uses this feature vector as the facial temporal feature corresponding to the video frame.

[0015] In some embodiments, the feature-level attention mechanism module can be configured to take the pulse wave signal and the facial temporal feature sequence together as multi-channel input, calculate the attention score of each channel through a fully connected layer, and use a normalized exponential function to weight the features of each channel.

[0016] In some embodiments, the classification module can be configured to: use a bidirectional long short-term memory network (LSTM) to perform temporal modeling on the feature sequence data after weighting by the feature-level attention mechanism module; concatenate the outputs of the forward and backward LSTMs to obtain a temporal output; process the temporal output through a multi-head self-attention mechanism to obtain attention-weighted temporal features; perform temporal mean pooling on the temporal features, and map them to the category space through a fully connected layer to output the physiological monitoring results.

[0017] In some embodiments, the signal extraction module may be configured to extract pulse wave signals from pre-specified frames in an input facial video.

[0018] In some embodiments, the one or more recognition models may be pre-trained on a training sample set, wherein the training sample set may include pre-collected facial video data of subjects with atrial fibrillation and subjects without atrial fibrillation, as well as corresponding monitoring values ​​of heart rate, blood oxygen saturation, and blood pressure data, wherein each facial video data is labeled with whether atrial fibrillation is present and the corresponding monitoring values ​​of heart rate, blood oxygen saturation, and blood pressure data.

[0019] According to a second aspect of the embodiments of this application, a physiological monitoring method based on deep learning is also provided, comprising: providing a facial video of a user to be detected as input to one or more pre-trained recognition models; each of the one or more recognition models outputting physiological monitoring results, wherein the physiological monitoring results include prediction results for one of the following: probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level; wherein the recognition model is the recognition model in the system described in the first aspect of the embodiments of this application.

[0020] Compared with the prior art, the technical solutions provided in this application embodiments may include the following beneficial effects:

[0021] A deep learning approach was adopted for atrial fibrillation identification and monitoring of multiple physiological indicators. A temporal adaptive attention mechanism architecture was introduced, which dynamically assigned weights to abnormal segments in the video, enabling the deep learning model to more effectively distinguish interference signals from key cardiac rhythm information. In addition, an adaptive weight mechanism was set for combining the rPPG signal generated by combining the region of interest in the video with the features extracted by the convolution module, so as to fully capture the importance of features in different regions and time periods, thereby improving the robustness and accuracy of atrial fibrillation detection and physiological indicator monitoring.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0024] Figure 1 A schematic diagram of the architecture of a deep learning-based physiological monitoring system according to an embodiment of this application is shown.

[0025] Figure 2A schematic diagram of the structure of an identification model according to an embodiment of this application is shown.

[0026] Figure 3 A schematic diagram of the facial shape extracted from each video frame according to an embodiment of this application is shown.

[0027] Figure 4 A schematic diagram of the ROI region extracted from each video frame according to an embodiment of this application is shown.

[0028] Figure 5a and Figure 5b This diagram illustrates a comparison of the rPPG signal extracted from a facial video before and after filtering. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description, in conjunction with the accompanying drawings, provides specific embodiments. It should be understood that the described embodiments are only a portion, not all, of the embodiments described herein. All other embodiments obtained by those skilled in the art based on the embodiments described herein without creative effort are within the scope of protection of this application.

[0030] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0033] To facilitate understanding, let's first give a brief introduction to rPPG technology. rPPG is a non-contact monitoring technology that uses a regular camera to capture changes in light reflected from the skin to monitor vital signs. In the case of the human heart rate, the blood flow with each heartbeat creates periodic changes in the microvessels of the skin tissue, resulting in a periodic signal from the absorption or reflection of light by the skin. This periodic signal is invisible to the human eye but can be captured by a high-definition camera. rPPG does not require the subject to wear any sensors, making it applicable to a wider range of subjects, such as newborns, burn patients, or those requiring long-term monitoring in hospital wards. Furthermore, the affordability, convenience, simplicity, and widespread availability of ordinary high-definition cameras give rPPG a very broad application prospect.

[0034] The basic process of remote heart rate measurement based on rPPG is as follows: First, under certain light sources or natural ambient light conditions, a camera captures skin data of a specific area on the face; this area is called the Region of Interest (ROI). The ROI can be manually selected or automatically detected and tracked using certain algorithms. Second, the spatial mean of the RGB colors is calculated on the acquired ROI data; this can be done for a single channel or multiple channels. Next, signal processing methods (low-pass filtering, blind source separation, etc.) are applied to the calculated spatial mean to obtain the component containing heart rate information. Then, a Fast Fourier Transform (FFT) or peak detection algorithm is typically applied to this component to estimate the corresponding frequency Fs of the heart rate signal or the number of peaks Ns during the processing duration T(s), etc. The resulting signal can also be called an rPPG signal.

[0035] However, such rPPG-based remote detection typically requires the object being detected to remain stationary and the ambient lighting to remain stable. Changes in lighting or slight movements of the object being detected during video recording can affect the accuracy of the detection results.

[0036] Figure 1 A schematic diagram of the architecture of a deep learning-based physiological monitoring system according to an embodiment of this application is shown. The system includes one or more recognition models. Each recognition model is pre-trained and is used to output corresponding physiological monitoring results based on an input sequence of facial video frames of a user. Such physiological monitoring results may include predictions of one of the following: the probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level.

[0037] In some embodiments, the various recognition models may have the same model structure, but may be trained independently. At least one recognition model may be trained to output the probability of atrial fibrillation based on an input sequence of facial video frames of a user. Other recognition models may be trained to output predictions of corresponding physiological indicators based on an input sequence of facial video frames of a user, such as predictions of heart rate zones, blood oxygen saturation levels, blood pressure levels, etc. Each recognition model corresponds to a prediction output of only one physiological indicator, such as heart rate zone (bpm): (0,60], (60,100],(100,∞), blood oxygen saturation (%): (0,95], (95,100], or blood pressure (mmhg): (0, 90 / 60], (90 / 60,120 / 80], (120 / 80, 140 / 90], (140 / 90,∞), etc.

[0038] In some embodiments, one or more of the recognition models described above are pre-trained on a training sample set. The training sample set includes pre-collected facial video data of subjects with and without atrial fibrillation, along with corresponding monitoring values ​​of heart rate, blood oxygen saturation, and blood pressure. Each facial video data point used as a sample is labeled with whether atrial fibrillation is present and the corresponding monitoring values ​​of heart rate, blood oxygen saturation, and blood pressure.

[0039] like Figure 1 As shown, the system may also include an integration module for outputting the prediction results of each recognition model. The output recognition results may indicate whether the user has atrial fibrillation, and the corresponding ranges for atrial fibrillation, heart rate, blood oxygen saturation, and blood pressure. In other embodiments, the integration module may also be used for unified management of these recognition models. For example, the integration module may act as a unified entry point to receive the facial video data stream input by the user, then simultaneously distribute the received data to each recognition model for prediction, and summarize the output results from each model, thereby integrating them into a structured comprehensive assessment report containing multi-dimensional physiological indicators, and presenting it to the user or downstream applications. It should be understood that this integration module is optional, and the physiological monitoring results of each recognition model can also be directly presented to the user as output.

[0040] Figure 2 A schematic diagram of the structure of an identification model according to an embodiment of this application is shown. Figure 2 As shown, the recognition model mainly includes a signal extraction module, a facial feature extraction module, a feature-level attention mechanism module, and a classification module. The input of the recognition model is a series of video frames of the facial video of the user to be identified, and the output recognition result (i.e., physiological monitoring result) may include, for example, a prediction result of one of the following: the probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level.

[0041] The signal extraction module is used to extract the corresponding rPPG signal from the user's facial video. In some embodiments, the signal extraction module can also be used to filter the extracted raw rPPG signal to improve its quality. For example, to ensure high-quality physiological information is extracted from the raw rPPG signal, a fourth-order Chebyshev type II bandpass filter can be used to filter the raw rPPG signals from the three regions of interest (ROIs). In some embodiments, for the input user facial video, the signal processing module can extract the rPPG signal from a fixed frame segment (e.g., frames 200 to 1200, approximately 30 seconds of video duration), thereby ensuring that the extracted signal is effective and has a high signal-to-noise ratio.

[0042] The facial feature extraction module is used to extract a temporal sequence of facial features from a user's facial video. In some embodiments, this facial feature extraction module employs multi-scale convolutional feature extraction and transformation to extract features at different scales from each frame of the input video, thereby transforming the input facial video into a temporal feature sequence. For example, in some embodiments, multi-scale feature extraction of each frame of the input image can be achieved by using convolutional kernels of different sizes (3×3, 5×5, 7×7) in parallel, and the extracted features can be directly transformed using convolution, pooling, and fully connected layers to convert them into a temporal feature sequence. This sequence not only preserves the multi-scale fusion effect of the image's spatial information but also effectively captures the dynamic changes of facial physiological signals.

[0043] The feature-level attention mechanism module assigns attention weights to the rPPG signal from the signal extraction module and the temporal feature sequence from the facial feature extraction module. This module first uses the rPPG signal and the facial temporal feature sequence together as multi-channel input. It then calculates the attention score for each channel through a fully connected layer and uses a normalized exponential function to weight the features of each channel. This highlights key signal features and suppresses redundant or interfering information, providing a more discriminative input for the subsequent classification module.

[0044] The classification module utilizes weighted feature sequence data from the feature-level attention mechanism module for classification. In some embodiments, this classification module employs an attention-enhanced bidirectional Long Short-Term Memory (LSTM) network. First, a bidirectional LSTM is used to perform temporal modeling on the weighted feature sequence data from the feature-level attention mechanism module, thereby capturing contextual relationships. Next, a multi-head self-attention mechanism is introduced to further capture global dependencies and improve the sensitivity of the recognition model to local anomalies. Finally, classification is performed through fully connected layers to output the recognition model's identification result for heart rhythm status.

[0045] To verify the performance of the recognition model used in the recognition system of this application, the inventors conducted relevant experiments, employing 5-fold cross-validation to evaluate the model's performance. The collected sample dataset was randomly divided into five roughly equal-sized portions (called "folds"). Each time, four folds were used as the training set to train the model, and the remaining fold was used as the test set to evaluate it. This process was repeated five times, each time selecting a different fold as the test set and the other four folds as the training set. During training, the recognition model performed forward propagation, loss calculation, backpropagation, and weight updates on each batch of data. Finally, the average of the five evaluation results was used as the final evaluation metric for the model. The metrics for each evaluation included accuracy, sensitivity, specificity, F1 score, and the area under the receiver operating characteristic (ROC) curve (AUC). Experimental results show that the recognition model has good accuracy and robustness in identifying aberrations (AF) and monitoring physiological signals.

[0046] The following will discuss data collection and processing, as well as... Figure 2 The training process, implementation method, and experimental process of the recognition model, which consists of a signal extraction module, a facial feature extraction module, a feature-level attention mechanism module, and a classification module, will be described in more detail.

[0047] 1. Sample Data Collection

[0048] The inventors collected facial video data from 500 adult patients with atrial fibrillation (AF) and 500 non-AF patients, along with corresponding data on atrial fibrillation, heart rate, blood oxygen saturation, and blood pressure. All AF patients underwent routine electrocardiogram (ECG) confirmation of AF episodes during data recording. In the examination room, each participant received a standard 12-lead ECG and immediately sat in front of a digital camera 1 meter away for facial video recording. Each subject recorded two facial videos (one taken with a mobile phone and one with a webcam) along with the corresponding ECG signal. One minute of facial video was recorded using a mobile phone (iPhone 14 and above) at 30 frames per second in 1080p MOV format; another minute was recorded using a webcam (Logitech C930) at 30 frames per second in 1080p MP4 format. Participants were instructed to maintain a stable posture as much as possible and minimize movement during recording. The ambient light source used for video recording was a fluorescent lamp, with an illuminance of approximately 200–400 lx measured in front of the participants. During the video recording, all participants were without makeup, and the recording ended after one minute. The diagnosis of atrial fibrillation (AF) was based on a detailed interpretation of the one-minute signal from a 12-lead ECG by a cardiologist, using the 12-lead ECG results as the label for AF classification. Based on the 12-lead ECG results, participants were divided into two groups: the "AF group": with or without other abnormal ECG patterns; and the "non-AF group": no AF was detected on the ECG.

[0049] 2. Signal Extraction Module

[0050] The signal extraction module is used to extract the rPPG signal from the acquired facial video data. For example, as mentioned above, for each frame of the facial image in the video sequence, a face detection algorithm can be used to locate the region of interest (ROI) on the face. The spatial mean of the RGB colors of the acquired ROI data is calculated, and then signal processing methods are applied to the calculated spatial mean to obtain the component containing heart rate information. Then, a Fast Fourier Transform (FFT) or peak detection algorithm is typically applied to this component to obtain the rPPG signal.

[0051] The quality of the acquired rPPG signal directly affects the accuracy of subsequent atrial fibrillation identification. To improve the quality of the rPPG signal extracted from facial video data, in some embodiments of this application, the signal extraction module can be configured to perform the following rPPG signal extraction processing on the facial video data.

[0052] First, to avoid environmental noise at the beginning and end of video recording (such as the patient not being fully prepared at the beginning of recording or the patient losing patience at the end of recording) affecting the accuracy of rPPG signal extraction, in the embodiments of this application, the first 200 frames (about 6.7s) and the last 400 frames (about 13.4s) of each video are discarded. For the remaining video, 1000 frames (about 33.3s) are extracted in frame order for subsequent processing, thus avoiding significant environmental noise interference.

[0053] Next, for each video frame, face tracking is performed to identify facial regions. For example, OpenCV software is used to read the video frame by frame and count the total number of frames. Then, the color image is converted to grayscale to simplify subsequent calculations and extract image structure information more efficiently. Next, a frontal face detector from the dlib library is used to process each grayscale image frame. This detector can quickly detect facial regions in an image and return a set of bounding boxes containing candidate face regions. If no face is detected in a frame, that frame is skipped; if multiple candidate regions are detected, the first one is selected as the target face region. For detected face regions, the corresponding region is first cropped from the original color image based on the bounding box coordinates returned by the detector. Then, the region is scaled to a predefined size (e.g., 64×64 pixels) using OpenCV software to facilitate subsequent feature extraction or as input to a neural network.

[0054] Then, a Region of Interest (ROI) is selected from the face region of each frame of video to exclude irrelevant information from the background. Existing methods typically set the ROI to a fixed-size region, directly selecting a fixed-size ROI region (e.g., the forehead) from the face region. However, the inventors found in practice that this method does not accurately differentiate ROIs because the face size varies between individuals, and the size of the ROI regions on the faces of different individuals also varies. Therefore, in the embodiments of this application, a dynamic ROI region determination method is provided instead of using a fixed-size ROI.

[0055] Specifically, the process begins by extracting key points that can characterize or represent the facial shape from the detected facial regions in each frame. For example, 68 facial key points can be extracted by calling the facial shape predictor from the dlib library, such as... Figure 3As shown. Based on these keypoints, dynamically sized Regions of Interest (ROIs) were set for the forehead, left cheek, and right cheek. The width of the ROI on the forehead was set to the length of keypoints 19 to 24, and the height to 1 / 4 of the total face size; the width of the ROI on the left cheek was set to the length of keypoints 4 to 48, and the height to the length of keypoint 1; the width of the ROI on the right cheek was set to the length of keypoints 54 to 12, and the height to the length of keypoint 12 to 15. The selected ROI areas are shown below. Figure 4 As shown.

[0056] Figure 4 The three ROI regions shown are not of fixed size, but are determined by the location of key points extracted from the facial regions of different individuals and the size of the specific facial regions. Therefore, the size of the ROI regions can be dynamically adjusted based on the location of 68 key points, which not only adaptively reflects the differences in individual facial structures, but also provides more accurate region division for subsequent extraction and analysis of facial temporal feature signals.

[0057] After determining the three ROI regions—forehead, left cheek, and right cheek—the average RGB value of the green channel of each ROI region in each frame is calculated as the rPPG signal intensity point for that frame.

[0058] In another embodiment of this application, the recognition model further includes a signal filtering module for filtering the rPPG signal from the signal extraction module. To reduce noise caused by motion or changes in ambient light intensity, the signal filtering module uses a fourth-order Chebyshev Type II bandpass filter (cutoff frequency: 0.5-3Hz) to filter the obtained rPPG signal. The signal comparison before and after filtering is as follows: Figure 5a and Figure 5b As shown.

[0059] After the signal extraction and signal filtering modules have performed the above processing, three sets of high-quality rPPG signals from different facial regions have been extracted as features for subsequent classification.

[0060] It should be noted that the above embodiments using three ROI regions as examples are merely illustrative and not intended to impose any limitations. Other embodiments may use one, two, or more ROI regions, and are not limited to the forehead, left cheek, and right cheek. Furthermore, it should be understood that the signal filtering module is an optional module and only a preferred implementation. In other embodiments, the signal filtering module may be omitted, which does not affect the implementation of the solution of this application. In some other embodiments, the filtering process may also be performed jointly by the signal extraction module.

[0061] 3. Facial Feature Extraction Module

[0062] In the embodiments of this application, in addition to using the rPPG signal extracted from facial video data as a feature for subsequent classification, facial temporal features are also introduced for subsequent classification. The inventors discovered in their research that skin color changes observed in facial videos depend not only on underlying blood flow pulses (which can be characterized by rPPG signals) but are also influenced by other factors, such as ambient lighting and facial movement. Therefore, in the embodiments of this application, the facial feature extraction module learns the interrelationships between various factors through multi-scale convolutional feature extraction and multi-scale feature transformation modules, and extracts facial temporal feature sequences from them. Considering that facial temporal feature signals are associated with video features (such as skin color and hair), the multi-scale convolutional feature extraction module uses 2D convolutional neural networks (CNNs) for convolutional feature extraction, extracting feature signals rich in facial features from the RGB three channels of each video frame in the input facial video frame sequence.

[0063] Assuming there are T video frames to be processed, the input to this multi-scale convolutional feature extraction module can be represented as:

[0064]

[0065] in Indicates the first One video frame image, and These represent the height and width of the video frame image, respectively, with 3 indicating the RGB three channels. The output of this multi-scale convolutional feature extraction module is a temporal feature signal sequence rich in facial features, i.e., the feature transformation value corresponding to each frame. For each frame... This module employs three sets of 2D convolutional operations with different kernel sizes of 3×3, 5×5, and 7×7 to extract multi-scale spatial features. Specifically, let:

[0066]

[0067] in, , and Representing respectively to Perform 3×3, 5×5, and 7×7 2D convolution operations. , and Indicates to , , The output is batch normalized. This indicates that the result is applied after batch normalization. The activation function (Rectified Linear Unit) sets all negative values ​​to 0 and retains positive values, introducing a non-linear transformation. Taking a 3×3 2D convolution operation as an example, the result is denoted as... This represents the feature map obtained after a 3×3 convolution. Then, these three sets of features are concatenated along the channel dimension:

[0068]

[0069] Next, a 2D average pooling layer (AvgPool) and a dropout layer (with a dropout probability of 0.5) are applied to the stitched result:

[0070]

[0071] The average pooling layer applies a two-dimensional average pooling operation to the input feature map. Its main purpose is to reduce the spatial size of the feature map while preserving the main information, thereby reducing the number of parameters and improving the robustness of the model. The dropout layer is mainly used to prevent the model from overfitting. During training, it randomly sets the output of some neurons to zero with a certain probability (e.g., a dropout probability of 0.5), thus forcing the 2D convolutional neural network to avoid over-reliance on certain local features.

[0072] The feature representation output by the multi-scale convolutional feature extraction module described above is the feature representation of each frame of the video. This provides input for the subsequent multi-scale feature transformation module. For ease of description, the feature representation of each video frame output by the multi-scale convolutional feature extraction module will be used. Using feature maps (in and The feature map F is represented by its spatial dimensions after pooling, and is provided as input to the feature transformation module. The feature transformation module extracts facial feature-related information from the feature map F through further convolution operations.

[0073] Specifically, the features of the input The feature transformation module also uses three different convolution kernel sizes: 3×3, 5×5, and 7×7 convolution operations.

[0074]

[0075] Next, the features obtained from the three convolutional branches are concatenated along the channel dimension:

[0076]

[0077] And the dimension is compressed to 1×1 using Adaptive Global Average Pooling (AdaptiveAvgPool2d):

[0078]

[0079] Furthermore, applying 1×1 convolution to From 24 channels to 1 channel:

[0080]

[0081] And further flatten the result into a one-dimensional vector:

[0082]

[0083] Then it is mapped to the hidden layer through a fully connected layer (FC):

[0084]

[0085] Finally, the signal after feature transformation is obtained through a second fully connected layer:

[0086]

[0087] In general, the processing steps of the facial feature extraction module described above can be understood as follows: for the input RGB channels Frame video, each frame The video frame features are obtained after the multi-scale convolution feature extraction module:

[0088]

[0089] Then, the image features of each frame are input into the multi-scale feature transformation module:

[0090]

[0091] The final obtained temporal feature sequence P of the entire video is:

[0092]

[0093] Therefore, the facial feature extraction module converts the video into a temporal feature sequence, which is then used as the temporal features of the face for subsequent classification.

[0094] 4. Feature-level attention mechanism module

[0095] This module aims to effectively fuse rPPG signals extracted from three different regions (forehead, left cheek, and right cheek) in the video by the signal extraction module with facial temporal feature signals obtained by the facial feature extraction module. This adaptively adjusts the importance of each feature, providing a more discriminative input for the subsequent classification module. This processing balances the complementarity and differences between rPPG signals from different facial regions and facial temporal feature information in autofocus detection (AF), thereby effectively improving the accuracy and robustness of AF recognition.

[0096] Specifically, the input data Z of this feature-level attention mechanism module can be represented as:

[0097]

[0098] in This indicates the number of frames in the video, specifically T video frames. `batch` represents the batch size, and each frame has four feature channels (three rPPG signal channels corresponding to the ROI regions and one facial temporal feature signal channel). This feature-level attention mechanism module uses a single-layer fully connected network to process the input... A linear transformation is performed to calculate attention weights, which are then used to weight the input. This linear transformation can be expressed, for example, as:

[0099]

[0100] Where z t Given the t-th element in input Z, , t=1,…,T; This is the weight matrix; This is the bias vector. Next, the attention weights are obtained by normalizing using the normalized exponential function (Softmax function):

[0101]

[0102] Finally, the weighted input features are used to obtain the attention-required features:

[0103]

[0104] in This represents element-wise multiplication. Therefore, for input data Z, the temporal feature data ultimately output by this feature-level attention mechanism module can be denoted as temporal data. The feature-level attention mechanism highlights important feature information, thus providing more discriminative input for subsequent classification modules.

[0105] 5. Classification Module

[0106] In one embodiment of this application, an attention-enhanced bidirectional long short-term memory network is used as a classification module, which processes the temporal data output by the feature-level attention mechanism module. The model is then built and classification results are output. In this embodiment, the attention-enhanced bidirectional long short-term memory network first processes the input data using a single-layer bidirectional LSTM with 32 hidden units. The outputs of the forward and backward LSTMs are as follows:

[0107]

[0108] After concatenating the forward and reverse outputs, the resulting timing output is:

[0109]

[0110] Therefore, the output sequence of the entire LSTM layer is as follows:

[0111]

[0112] Based on the above description of the output of a single-layer bidirectional LSTM, the computation process of the LSTM unit is further detailed. Specifically, at each time step... The LSTM unit then performs the following operations in sequence:

[0113]

[0114] in , , , , and These represent the forget gate, input gate, candidate cell state, cell state update, output gate, and hidden state, respectively. For a moment The input vector, This is the hidden state from the previous moment. This represents the sigmoid activation function. It is the hyperbolic tangent function. Represents element-wise product. This is the weight matrix. This is the bias vector.

[0115] To further capture the local and global dependencies in the temporal sequence, this attention-enhanced bidirectional long short-term memory network further employs a multi-head attention mechanism to process the output H of the single-layer bidirectional LSTM. The embedding dimension is 64, and the number of heads is 2. The processing procedure of the multi-head attention mechanism can be described as follows:

[0116]

[0117] The formula for calculating each attention point is as follows:

[0118]

[0119] And for each head, there are:

[0120]

[0121] in This is the weight matrix.

[0122] The temporal features after the above multi-head attention processing are denoted as follows: Next, the feature sequence is further subjected to time-series mean pooling to obtain:

[0123]

[0124] Then, the data is mapped to the category space through a fully connected layer. The number of categories varies across different tasks, therefore the parameter settings also differ. Here, N represents the number of categories for the task (atrial fibrillation prediction is binary, heart rate detection is tri-class, blood oxygen saturation detection is binary, and blood pressure detection is quadruple). Let the parameters of the fully connected layer be... and bias The original output score for the classification is:

[0125]

[0126] Therefore, the classification score vector obtained after the fully connected layer can be denoted as:

[0127]

[0128] in arrive These represent the unnormalized scores for categories 1, 2, and N, respectively. Finally, the predicted probabilities are calculated using the normalized exponential function based on the classification results:

[0129]

[0130] here Indicate category Predicted probability

[0131] The above text combined Figure 1This paper describes the working process of each component of the recognition model in this embodiment. The recognition model can be trained based on AF patients and their facial video data, which are labeled with 12-lead ECG results for AF classification. For training on physiological signal monitoring, the corresponding physiological signal values ​​monitored by the patients during video recording are used. In each training iteration, the facial video data is input into the recognition model. The signal extraction module extracts the corresponding rPPG signal from the input facial video data, the facial feature extraction module obtains the temporal feature sequence of the face from the input facial video data, the feature-level attention mechanism module fuses and weights the rPPG signal and the facial temporal feature sequence, and then the fused feature sequence data is classified using an attention-enhanced bidirectional long short-term memory network to output a predicted probability. The predicted result is then compared with the corresponding label, a loss function is calculated, and backpropagation is performed to adjust the relevant parameters of the recognition model.

[0132] During training, the recognition model employed the cross-entropy loss function and the Adam optimizer (learning rate 0.001), inputting training samples in batches of 2 to reduce computational and storage burden. The training epoch was set to 100. All experiments were conducted using Python 3.9, PyTorch 2.0, and PyCharm 2024. The cross-entropy loss function was used to measure the difference between the predicted results and the true sample labels. The true label for each training sample... It can be represented in one-hot encoding form, that is:

[0133]

[0134] The cross-entropy loss function is then defined as:

[0135]

[0136] in, For category The predicted probability is the classification result output by the recognition model mentioned above. Substituting the normalized exponential function form, it can be expressed as:

[0137]

[0138] in This represents the loss value of the overall recognition model.

[0139] In yet another embodiment of this application, a deep learning-based contactless physiological monitoring method is also provided. The method includes inputting a facial video of a user to be monitored into one or more pre-trained recognition models. Each recognition model outputs different physiological monitoring results. The physiological monitoring results output by each recognition model may include a prediction of one of the following: the probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level. The structure of each recognition model is as described above. Figure 2 As shown, it is used to extract rPPG signals and facial temporal feature sequences from input facial video data; assign attention weights to the rPPG signals and facial temporal feature sequences to obtain weighted features; and input the weighted features into a classification module to obtain predictions of atrial fibrillation probability, heart rate, blood oxygen saturation, or blood pressure. The classification module is a combination of the above... Figure 2 This paper introduces an attention-enhanced bidirectional long short-term memory network. This recognition method incorporates rPPG signals reflecting heart rate status as well as temporal features capturing dynamic facial changes, thus improving the accuracy and robustness of recognition.

[0140] In another embodiment of this application, a computer-readable storage medium is provided, on which a computer program or executable instructions are stored. When the computer program or executable instructions are executed by a processor or other computing unit, the technical solution described in the foregoing embodiments is implemented. The implementation principle is similar and will not be repeated here. In the embodiments of this application, the computer-readable storage medium can be any tangible medium capable of storing data and readable by a computing device. Examples of computer-readable storage media include hard disk drives, network attached storage (NAS), read-only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical or non-optical data storage devices. The computer-readable storage medium may also include computer-readable media distributed across a network-coupled computer system so that computer programs or instructions can be stored and executed in a distributed manner.

[0141] References to "various embodiments," "some embodiments," "one embodiment," or "embodiment," etc., in this specification refer to a specific feature, structure, or property described in connection with the said embodiment, included in at least one embodiment. Therefore, the appearance of the phrases "in various embodiments," "in some embodiments," "in one embodiment," or "in an embodiment," etc., throughout this specification does not necessarily refer to the same embodiment. Furthermore, specific features, structures, or properties can be combined in any suitable manner in one or more embodiments. Therefore, a specific feature, structure, or property shown or described in connection with one embodiment can be combined, in whole or in part, with features, structures, or properties of one or more other embodiments without limitation, provided that the combination is not illogical or inoperable.

[0142] The terms "comprising," "having," and similar expressions used in this specification are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus. "A" or "an" does not exclude multiple instances. Furthermore, the elements in the accompanying drawings are for illustrative purposes only and are not drawn to scale.

[0143] Although this application has been described through the above embodiments, this application is not limited to the embodiments described herein, and includes various changes and variations without departing from the scope of this application.

Claims

1. A deep learning-based physiological monitoring system, the system comprising one or more recognition models, the recognition models comprising: The signal extraction module is used to extract pulse wave signals from the user's facial video; A facial feature extraction module is used to extract a sequence of facial temporal features from a user's facial video. The sequence of facial temporal features includes facial temporal features corresponding to each video frame in the facial video. The facial feature extraction module includes a multi-scale convolutional feature extraction and multi-scale feature transformation module, wherein for each video frame in the input facial video: The multi-scale convolutional feature extraction module uses a 2D convolutional neural network and three sets of different convolutional kernel sizes to extract convolutional features from the video frame, and then concatenates these three sets of convolutional features into a feature map along the channel dimension. The feature transformation module uses three sets of different convolution kernel sizes to continue extracting facial features from the feature map, and then concatenates the three sets of facial features in the channel dimension to further transform them into a one-dimensional feature vector, and uses this feature vector as the facial temporal feature corresponding to the video frame. The feature-level attention mechanism module is used to assign attention weights to the pulse wave signal and the facial temporal feature sequence and perform weighted processing. The feature-level attention mechanism module is configured to take the pulse wave signal and the facial temporal feature sequence together as multi-channel input, perform linear transformation on the multi-channel input through a fully connected layer, normalize the output of the fully connected layer through a normalized exponential function to obtain attention weights, and use the obtained attention weights to perform element-wise multiplication on the multi-channel input to obtain the weighted pulse wave signal and facial temporal features. The classification module is used to output physiological monitoring results based on weighted pulse wave signals and facial temporal features, wherein the physiological monitoring results include predictions for one of the following: probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level.

2. The system according to claim 1, wherein, The one or more recognition models are trained independently and are used to output predicted results for the probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level based on the input user's facial video; and The system also includes an integration module, which receives the user's facial video and feeds it to each recognition model simultaneously, as well as summarizing and outputting the prediction results of each recognition model.

3. The system according to claim 1, wherein the signal extraction module is configured to extract corresponding pulse wave signals from one or more of the following facial regions of interest: forehead, left cheek, and right cheek.

4. The system of claim 1, wherein the signal extraction module is configured to determine the region of interest of the face in the following manner: Extract key points to characterize the facial shape from the detected facial regions in each video frame; The region of interest is determined based on the location of these key points in the facial region and the size of the facial region.

5. The system according to claim 1, wherein the signal extraction module is further configured to perform filtering processing on the extracted pulse wave signal using a fourth-order Chebyshev type II bandpass filter.

6. The system according to claim 1, wherein the classification module is configured as follows: A bidirectional long short-term memory network (LSTM) is used to perform temporal modeling on the feature sequence data after weighted processing by the feature-level attention mechanism module. The outputs of the forward and backward LSTMs are obtained, and the temporal output is obtained by concatenating the forward and backward outputs. The temporal output is processed by a multi-head self-attention mechanism to obtain attention-weighted temporal features; After performing temporal mean pooling on the temporal features, the data is mapped to the category space through a fully connected layer, thereby outputting the physiological monitoring results.

7. The system according to claim 1, wherein the one or more recognition models are pre-trained on a training sample set, wherein the training sample set includes pre-collected facial video data of subjects with and without atrial fibrillation, and corresponding monitoring values ​​of heart rate, blood oxygen saturation, and blood pressure. Each facial video data point is labeled with whether atrial fibrillation is present, along with corresponding monitoring values ​​for heart rate, blood oxygen saturation, and blood pressure.

8. A physiological monitoring method based on deep learning, comprising: A facial video of the user to be detected is provided as input to one or more pre-trained recognition models; Each of the one or more recognition models outputs physiological monitoring results, wherein the physiological monitoring results include predictions for one of the following: probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level; The recognition model described therein is the recognition model in the system according to claim 1.

Images