Deep learning-based systems and methods for contactless atrial fibrillation recognition and multi-physiological parameter monitoring

Abstract

The invention relates to a non-contact atrial fibrillation recognition and multi-physiological index monitoring system and method based on deep learning, and the method comprises the steps: extracting a pulse wave signal from a face video of a user, and extracting a face time sequence feature sequence from the face video of the user, and distributing attention weights to the pulse wave signals and the face time sequence feature sequence, performing weighting processing, and predicting the probability of atrial fibrillation, the heart rate interval, the blood oxygen saturation level and the blood pressure level based on the weighted pulse wave signals and the face time sequence features. According to the solution, the robustness and accuracy of atrial fibrillation detection can be effectively improved, and meanwhile, multiple physiological indexes of the user are synchronously monitored.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202511547601.9 (22) Application Date 2025.10.28 (71) Applicant: University of Hong Kong Address: Pok Fu Lam Road, Hong Kong, China Applicant: University of Hong Kong-Shenzhen Hospital (72) Inventors: Yao Qiheng, Huang Jiayi, Wang Chenxu (74) Patent Agency: Beijing Panhua Weiye Intellectual Property Agency Co., Ltd. 11280 Patent Attorney: Wang Bo (51) Int.Cl. A61B 5 / 024 (2006.01) A61B 5 / 00 (2006.01) G06V 40 / 16 (2022.01) G06V 10 / 82 (2022.01) (54) Invention Title: Deep Learning-Based Contactless Atrial Fibrillation Identification and Multi-Physiological Indicator Monitoring System and Method (57) Abstract: This application relates to a deep learning-based contactless atrial fibrillation identification and multi-physiological indicator monitoring system and method, comprising: extracting pulse wave signals from a user's facial video; extracting facial temporal feature sequences from the user's facial video; assigning attention weights to the pulse wave signals and the facial temporal feature sequences and performing weighted processing; and predicting the probability of atrial fibrillation and heart rate zones, blood oxygen saturation levels, and blood pressure grades based on the weighted pulse wave signals and facial temporal features. This solution can effectively improve the robustness and accuracy of atrial fibrillation detection, while simultaneously monitoring multiple physiological indicators of the user. Claims (2 pages), Description (13 pages), Drawings (3 pages), CN 121400797 A 2026.01.27 CN 1 21 40 07 97 A 1. A physiological monitoring system based on deep learning, the system comprising one or more recognition models, the recognition model comprising: a signal extraction module for extracting pulse wave signals from a user's facial video; a facial feature extraction module for extracting facial temporal feature sequences from the user's facial video; a feature-level attention mechanism module for assigning attention weights to the pulse wave signals and the facial temporal feature sequences and performing weighted processing; a classification module for outputting physiological monitoring results based on the weighted pulse wave signals and facial temporal features, wherein the physiological monitoring results include predictions of one of the following: probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level. 2. The system according to claim 1, wherein the one or more recognition models are trained independently and are respectively used to output predicted results of the probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level based on the input user's facial video; and the system further includes an integration module for receiving the user's facial video and simultaneously feeding it to each3. The system of claim 1, wherein the signal extraction module is configured to extract corresponding pulse wave signals from one or more of the following facial regions of interest: forehead, left cheek, right cheek. 4. The system of claim 1, wherein the signal extraction module is configured to determine the facial regions of interest by: extracting key points characterizing the facial shape from the detected facial regions in each video frame; determining the regions of interest based on the position of these key points in the facial regions and the size of the facial regions. 5. The system of claim 1, wherein the signal extraction module is further configured to filter the extracted pulse wave signals using a fourth-order Chebyshev type II bandpass filter. 6. The system according to claim 1, wherein the facial feature extraction module includes a multi-scale convolutional feature extraction and multi-scale feature transformation module, wherein for each video frame from the input facial video data: the multi-scale convolutional feature extraction module extracts convolutional features from the video frame using a 2D convolutional neural network and three sets of different convolutional kernel sizes, and concatenates these three sets of convolutional features into a feature map in the channel dimension; the feature transformation module continues to extract facial features from the feature map using three sets of different convolutional kernel sizes, and further transforms the extracted three sets of facial features into a one-dimensional feature vector after concatenating them in the channel dimension, and uses this feature vector as the facial temporal feature corresponding to the video frame. 7. The system according to claim 1, wherein the feature-level attention mechanism module is configured to take the pulse wave signal and the facial temporal feature sequence together as multi-channel input, calculate the attention score of each channel through a fully connected layer, and use a normalized exponential function to weight the features of each channel. 8. The system according to claim 1, wherein the classification module is configured to: perform temporal modeling on the feature sequence data after weighting processing by the feature-level attention mechanism module using a bidirectional long short-term memory network LSTM, and concatenate the outputs of the forward and backward LSTMs to obtain a temporal output; process the temporal output through a multi-head self-attention mechanism to obtain attention-weighted temporal features; perform temporal mean pooling on the temporal features, and map them to the category space through a fully connected layer, thereby outputting the physiological monitoring results of claim 1 / 2 page 2 CN 121400797 A. 9. The system according to claim 1, wherein the one or more recognition models are pre-trained on a training sample set, wherein the training sample set includes pre-collected facial video data of subjects with atrial fibrillation and subjects without atrial fibrillation, as well as corresponding monitoring values ​​of heart rate, blood oxygen saturation, and blood pressure data, wherein each facial videoThe data is labeled with the presence or absence of atrial fibrillation and the corresponding monitoring values ​​of heart rate, blood oxygen saturation, and blood pressure. 10. A physiological monitoring method based on deep learning, comprising: providing a facial video of a user to be detected as input to one or more pre-trained recognition models; and outputting physiological monitoring results by each of the one or more recognition models, wherein the physiological monitoring results include prediction results for one of the following: probability of atrial fibrillation, heart rate range, blood oxygen saturation level, and blood pressure level; wherein the recognition model is the recognition model in the system according to claim 1. Claims 2 / 2 Page 3 CN 121400797 A Deep Learning-Based Non-Contact Atrial Fibrillation Recognition and Multi-Physiological Indicator Monitoring System and Method Technical Field

[0001] This application relates to computer-aided medical data processing, and particularly to a deep learning-based atrial fibrillation recognition and multi-physiological indicator monitoring system and method. Background Art

[0002] Atrial fibrillation (AF) is the most common arrhythmia. When it develops to a certain extent, it poses a serious risk of complications, such as stroke, heart failure, and death. However, atrial fibrillation (hereinafter referred to as AF) is usually asymptomatic and occurs in paroxysmal episodes, often being overlooked clinically. Therefore, early screening and detection of AF is of great significance, which is conducive to timely drug treatment, interventional treatment, etc., thereby effectively controlling the condition and reducing the occurrence of complications.

[0003] Traditional AF detection usually requires the use of a variety of specialized medical instruments for electrocardiogram monitoring, blood pressure measurement, and blood oxygenation, etc. Each instrument has high cost and accessibility barriers, which are not only unsuitable for long-term continuous monitoring, but also expensive, and some detection methods are even invasive (such as implantable circulatory recorders and dynamic electrocardiogram patches). Although handheld electrocardiogram devices provide convenience for intermittent screening, they are still not practical for long-term monitoring.

[0004] In recent years, atrial fibrillation detection technology based on remote photoplethysmography (rPPG) has received widespread attention. rPPG is a non-contact monitoring technology that uses a regular camera to capture changes in light reflected from the skin to monitor vital signs. Its principle is based on a model of reflection and absorption of natural light on the skin. When natural light shines on the skin, part of the light is directly reflected by the epidermal tissue, and the other part is diffusely reflected after being absorbed by hemoglobin in the capillaries. Each contraction and relaxation of the heart causes a change in the concentration of hemoglobin in the blood, resulting in a slight change in skin color. This subtle change in skin color can be captured by a high-resolution camera device (such as a mobile phone or computer camera), thereby obtaining the pulse wave signal (hereinafter referred to as the rPPG signal). Then, the morphology and characteristics of the rPPG signal are analyzed...The physiological information is extracted from the rPPG signal. Therefore, the atrial fibrillation detection method based on rPPG does not require contact with the human body or professional medical instruments. It can be monitored using a regular camera. It extracts rPPG signals from a facial video uploaded by the user, performs feature extraction and classification on the rPPG signals, and thus realizes atrial fibrillation detection.

[0005] It should be noted that the above content is only used to help understand the technical solution of this application and should not be used as a basis for evaluating the prior art of this application. Summary of the Invention

[0006] However, the inventors found in practice that this atrial fibrillation and physiological signal monitoring based on rPPG is greatly affected by the environment and the monitored object, and its accuracy and robustness need to be improved. Because this monitoring method usually requires the monitored object to remain still and the ambient lighting to remain stable, so as to record a relatively still facial video of the monitored object for a period of time. However, in practical applications, due to changes in ambient light or slight movements caused by decreased patience and emotional fluctuations of the monitored object, short-term movements or changes in facial expressions often appear in the recorded facial video. This will cause unwanted noise in the captured rPPG signal, which may lead to confusion when distinguishing abnormal segments from real atrial fibrillation segments during rPPG signal processing, thus affecting the detection accuracy. Moreover, such a method is obviously not suitable for atrial fibrillation monitoring of objects in motion.

[0007] Therefore, the purpose of this application is to provide a more accurate and robust rPPG-based atrial fibrillation and physiological signal monitoring scheme.

[0008] The above objective is achieved through the following technical solutions:

[0009] According to a first aspect of the embodiments of this application, a physiological monitoring system based on deep learning is provided. The system includes one or more recognition models, the recognition model including: a signal extraction module for extracting pulse wave signals from a user's facial video; a facial feature extraction module for extracting facial temporal feature sequences from the user's facial video; a feature-level attention mechanism module for assigning attention weights to the pulse wave signals and the facial temporal feature sequences and performing weighted processing; and a classification module for outputting physiological monitoring results based on the weighted pulse wave signals and facial temporal features, wherein the physiological monitoring results include prediction results for one of the following: the probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level.

[0010] In some embodiments, the one or more recognition models may be trained independently and used to output prediction results of the probability of atrial fibrillation, heart rate zone, blood oxygen saturation level and blood pressure level based on the input user's facial video; and the system may also include an integration module, which is used to receive the user's facial video and feed it to each recognition model at the same time, and to summarize and output the prediction results of each recognition model.

[0011] In some embodiments, the signal extraction module may be configured to extract corresponding pulse wave signals from one or more of the following regions of interest on the face: forehead, left cheek, and right cheek.

[0012] In some embodiments, the signal extraction module may be configured to determine the regions of interest on the face in the following manner: extracting key points for characterizing the facial shape from the facial regions detected in each video frame; determining the regions of interest based on the position of these key points in the facial regions and the size of the facial regions.

[0013] In some embodiments, the signal extraction module may also be configured to filter the extracted pulse wave signals using a fourth-order Chebyshev type II bandpass filter.

[0014] In some embodiments, the facial feature extraction module may include a multi-scale convolutional feature extraction and multi-scale feature transformation module, wherein for each video frame from the input facial video data, the multi-scale convolutional feature extraction module extracts convolutional features from the video frame using a 2D convolutional neural network and three sets of different convolutional kernel sizes, and concatenates these three sets of convolutional features into a feature map in the channel dimension; the feature transformation module continues to extract facial features from the feature map using three sets of different convolutional kernel sizes, and further transforms the extracted three sets of facial features into a one-dimensional feature vector after concatenating them in the channel dimension, and uses this feature vector as the facial temporal feature corresponding to the video frame.

[0015] In some embodiments, the feature-level attention mechanism module may be configured to take the pulse wave signal and the facial temporal feature sequence together as multi-channel input, calculate the attention score of each channel through a fully connected layer, and use a normalized exponential function to weight the features of each channel.

[0016] In some embodiments, the classification module may be configured to: use a bidirectional long short-term memory network (LSTM) to perform temporal modeling on the feature sequence data after weighting by the feature-level attention mechanism module; obtain the outputs of the forward and backward LSTMs and concatenate the forward and backward outputs to obtain a temporal output; process the temporal output through a multi-head self-attention mechanism to obtain attention-weighted temporal features; perform temporal mean pooling on the temporal features and map them to the category space through a fully connected layer to output the physiological monitoring results.

[0017] In some embodiments, the signal extraction module may be configured to extract pulse wave signals from pre-specified frames in the input facial video.

[0018] In some embodiments, the one or more recognition models may be pre-trained on a training sample set to obtain the specification page 2 / 13 5 CN 121400797 A, wherein the training sample set may include pre-collected facial video data of subjects with atrial fibrillation and subjects without atrial fibrillation, as well as the corresponding monitoring values ​​of heart rate, blood oxygen saturation, and blood pressure data, wherein each facial video data isThe label indicates whether atrial fibrillation is present, along with the corresponding monitoring values ​​for heart rate, blood oxygen saturation, and blood pressure.

[0019] According to a second aspect of the embodiments of this application, a physiological monitoring method based on deep learning is also provided, which includes: providing a facial video of a user to be detected as input to one or more pre-trained recognition models; each of the one or more recognition models outputs a physiological monitoring result, wherein the physiological monitoring result includes a prediction result for one of the following: the probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level; wherein the recognition model is the recognition model in the system according to the first aspect of the embodiments of this application.

[0020] Compared with the prior art, the technical solution provided by the embodiments of this application may include the following beneficial effects:

[0021] Atrial fibrillation recognition and multi-physiological indicator monitoring are performed using a deep learning method, and a temporal adaptive attention mechanism architecture is introduced. By dynamically assigning weights to abnormal segments in the video, the deep learning model can more effectively distinguish interference signals from key heart rhythm information; and a weight adaptive mechanism is set for the rPPG signal generated by combining the region of interest in the video with the features extracted by the convolution module, so as to fully capture the importance of features in different regions and time periods, thereby improving the robustness and accuracy of atrial fibrillation detection physiological indicator monitoring.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit the present application. Brief Description of the Drawings

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. Obviously, the drawings described below are merely some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative effort. In the drawings:

[0024] FIG1 shows a schematic diagram of the architecture of a deep learning-based physiological monitoring system according to an embodiment of the present application.

[0025] FIG2 shows a schematic diagram of the structure of a recognition model according to an embodiment of the present application.

[0026] FIG3 shows a schematic diagram of the facial shape extracted from each video frame according to an embodiment of the present application.

[0027] FIG4 shows a schematic diagram of the ROI region extracted from each video frame according to an embodiment of the present application.

[0028] FIG5a and FIG5b show comparative schematic diagrams of the rPPG signal extracted from the facial video before and after filtering. Detailed Description of Embodiments

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the described embodiments are only a part of the embodiments of this application, and not all of them.These are all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0030] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of the embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be used. In other cases, known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of this application.

[0031] The block diagrams shown in the drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined, so the actual execution order may change depending on the actual situation.

[0033] For ease of understanding, a brief introduction to rPPG technology is given first. rPPG is a non-contact monitoring technology that monitors vital signs by capturing changes in light reflected from the skin using a regular camera. Regarding human heart rate, the blood flow brought about by each heartbeat creates periodic changes in the microvessels of the skin tissue, resulting in a periodic signal for the absorption or reflection of light by the skin. This periodic signal is invisible to the human eye but can be captured by a high-definition camera. rPPG does not require the subject to wear any sensors, making it applicable to a wider range of subjects, such as newborns, burn patients, or long-term ward monitoring. Furthermore, because regular high-definition cameras are inexpensive, convenient, simple to use, and widely distributed, rPPG has a very broad application prospect.

[0034] The basic process of remote heart rate measurement based on rPPG is as follows: First, under certain light source or natural ambient light conditions, a camera is used to capture skin data of a certain area on the face. This area is called the Region of Interest (ROI). The ROI can be manually selected or automatically detected and tracked using certain algorithms. Second, the spatial mean of the RGB color values ​​is calculated on the collected ROI data. Only one channel can be calculated, or multiple channels can be calculated. Next, the calculated spatial mean is...The mean value is obtained by applying signal processing methods (low-pass filtering, blind source separation, etc.) to obtain the component containing heart rate information. Then, a fast Fourier transform (FFT) or peak detection algorithm is usually applied to this component to estimate the corresponding frequency Fs of the heart rate signal or the number of peaks Ns during the processing duration T(s), etc. The signal obtained can also be called an rPPG signal.

[0035] However, such rPPG-based remote detection usually requires the object being detected to remain stationary and the ambient lighting to remain stable. If the lighting changes or the object being detected moves slightly during video recording, the accuracy of the detection results will be affected.

[0036] Figure 1 shows a schematic diagram of the architecture of a deep learning-based physiological monitoring system according to an embodiment of this application. The system includes one or more recognition models. Each recognition model is pre-trained to output corresponding physiological monitoring results based on the input sequence of facial video frames of the user. Such physiological monitoring results may include predictions of one of the following: the probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level.

[0037] In some embodiments, the various recognition models may have the same model structure, but may be trained independently. At least one of the recognition models may be trained to output the probability of atrial fibrillation based on an input sequence of facial video frames of a user. Other recognition models may be trained to output predictions of corresponding physiological indicators, such as predictions of heart rate zones, blood oxygen saturation levels, blood pressure levels, etc., based on an input sequence of facial video frames of a user. Each recognition model corresponds to only one physiological indicator prediction output, such as heart rate zone (bpm): (0, 60], (60, 100], (100, ∞), blood oxygen saturation (%): (0, 95], (95, 100], or blood pressure (mmHg): (0, 90 / 60], (90 / 60, 120 / 80], (120 / 80, 140 / 90], (140 / 90, ∞), etc.

[0038] In some embodiments, the above-mentioned one or more recognition models are pre-trained on a training sample set. The training sample set includes pre-collected facial video data of subjects with atrial fibrillation and subjects without atrial fibrillation, as well as the corresponding monitoring values ​​of heart rate, blood oxygen saturation, and blood pressure. Each facial video data as a sample is labeled with whether there is atrial fibrillation and the corresponding monitoring values ​​of heart rate, blood oxygen saturation, and blood pressure. Specification 4 / 13 pages 7 CN 121400797 A

[0039] As shown in Figure 1, the system may further include an integration module for outputting the prediction results of each recognition model. The output recognition results may indicate whether the user has atrial fibrillation, and the corresponding atrial fibrillation, heart rate, and blood oxygen saturation.And what is the blood pressure range? In some other embodiments, the integration module can also be used to manage these recognition models in a unified manner. For example, the integration module can serve as a unified entry point to receive the facial video data stream input by the user, and then distribute the received data to each recognition model in parallel for prediction, and summarize the output results from each model, and then integrate them into a structured comprehensive evaluation report containing multi-dimensional physiological indicators, and present it to the user or downstream application. It should be understood that the integration module is optional, and the physiological monitoring results of each recognition model can also be directly presented to the user as output.

[0040] Figure 2 shows a schematic diagram of the structure of a recognition model according to an embodiment of the present application. As shown in Figure 2, the recognition model mainly includes a signal extraction module, a facial feature extraction module, a feature-level attention mechanism module and a classification module. The input of the recognition model is a series of video frames of the facial video of the user to be recognized, and the output recognition result (i.e., physiological monitoring result) may include, for example, a prediction result for one of the following: the probability of atrial fibrillation, heart rate interval, blood oxygen saturation level, and blood pressure level.

[0041] The signal extraction module is used to extract the corresponding rPPG signal from the user's facial video. In some embodiments, the signal extraction module can also be used to filter the extracted raw rPPG signal to improve the quality of the extracted rPPG signal. For example, to ensure high-quality physiological information is extracted from the raw rPPG signal, a fourth-order Chebyshev type II bandpass filter can be used to filter the raw rPPG signal from the three regions of interest (ROI). In some embodiments, for the input user facial video, the signal processing module can extract the rPPG signal from the video within a fixed frame segment (e.g., frame 200 to frame 1200, approximately 30 seconds of video duration), thereby ensuring that the extracted signal is effective and has a high signal-to-noise ratio.

[0042] The facial feature extraction module is used to extract a facial temporal feature sequence from the user's facial video. In some embodiments, the facial feature extraction module uses multi-scale convolutional feature extraction and transformation to extract features of different scales from each frame of the input video, thereby transforming the input facial video into a temporal feature sequence. For example, in some embodiments, multi-scale feature extraction of each frame of the input image can be achieved by using convolutional kernels of different sizes (3×3, 5×5, 7×7) in parallel, and the extracted features can be directly transformed into a temporal feature sequence using convolution, pooling and fully connected layers. This sequence not only retains the multi-scale fusion effect of the image spatial information, but also effectively captures the dynamic changes of facial physiological signals.

[0043] The feature-level attention mechanism module is used to assign attention weights to the rPPG signal from the signal extraction module and the temporal feature sequence from the facial feature extraction module. The feature-level attention mechanism module first compares the rPPG signal and the facial time sequence.The sequence of features is used as a multi-channel input. The attention score of each channel is calculated through a fully connected layer, and the features of each channel are weighted using a normalized exponential function. This highlights key signal features and suppresses redundant or interfering information, providing a more discriminative input for the subsequent classification module.

[0044] The classification module uses the weighted feature sequence data from the feature-level attention mechanism module for classification. In some embodiments, the classification module uses an attention-enhanced bidirectional long short-term memory (LSTM) network. First, a bidirectional LSTM is used to perform temporal modeling on the feature sequence data weighted by the feature-level attention mechanism module to capture contextual relationships. Then, by introducing a multi-head self-attention mechanism, global dependencies are further captured, and the sensitivity of the recognition model to local abnormal changes is improved. Then, classification is performed through a fully connected layer to output the recognition result of the recognition model for heart rhythm status.

[0045] In order to verify the performance of the recognition model used in the recognition system of this application, the inventors conducted relevant experiments and used 5-fold cross-validation to evaluate the performance of the recognition model. The collected sample dataset was randomly divided into five equal parts (called "folds"). Four folds were used as the training set to train the model each time, and the remaining fold was used as the test set to evaluate the model. This process was repeated five times, each time selecting a different fold as the test set and the other four folds as the training set. During training, the recognition model performed forward propagation, loss calculation, backpropagation, and weight updates on each batch of data. Finally, the average of the five evaluation results was used as the final evaluation metric for the model. The metrics for each evaluation included accuracy, sensitivity, specificity, F1 score, and the area under the receiver operating characteristic curve (ROC) and the coordinate axes (AUC). Experimental results show that the recognition model has good accuracy and robustness in identifying autoanalytical reactions (AF) and monitoring physiological signals.

[0046] The following will provide a more detailed description of the data acquisition and processing, as well as the training process, implementation method, and experimental process of the recognition model shown in Figure 2, which consists of a signal extraction module, a facial feature extraction module, a feature-level attention mechanism module, and a classification module.

[0047] 1. Data Acquisition

[0048] The inventors collected facial video data from 500 adult AF patients and 500 non-AF patients, along with corresponding atrial fibrillation, heart rate, blood oxygen saturation, and blood pressure data. All AF patients underwent routine electrocardiogram (ECG) during data recording.ECG confirmed the presence of atrial fibrillation (AF). In the examination room, each participant underwent a standard 12-lead ECG and immediately sat in front of a digital camera 1 meter away for facial video recording. Two facial videos were recorded for each subject (one by a mobile phone and one by a webcam) along with the corresponding ECG signal. One minute of facial video was recorded using a mobile phone (iPhone 14 and above) at 30 frames per second in 1080p MOV format; another minute was recorded using a computer webcam (Logitech C930) at 30 frames per second in 1080p MP4 format. Participants were instructed to maintain a stable posture as much as possible and minimize movement during recording. The ambient light source for video recording was fluorescent lighting, with an illuminance of approximately 200–400 lx measured in front of the participants. All subjects were not wearing makeup during video recording, and recording ended after one minute. AF was determined based on a detailed interpretation of the one-minute signal from the 12-lead ECG by a cardiologist, using the 12-lead ECG result as the label for AF classification. Based on the 12-lead ECG results, participants were divided into two groups: "AF group": with or without other abnormal ECG patterns; and "non-AF group": no AF was detected on the ECG.

[0049] 2. Signal Extraction Module

[0050] The signal extraction module is used to extract rPPG signals from the acquired facial video data. For example, as mentioned above, for each frame of the facial image in the video sequence, a facial detection algorithm can be used to locate the region of interest (ROI) on the face, calculate the spatial mean of the RGB color of the acquired ROI data, and then apply signal processing methods to the calculated spatial mean to obtain a component containing heart rate information. Then, a fast Fourier transform (FFT) or peak detection algorithm is usually applied to this component to obtain the rPPG signal.

[0051] The quality of the acquired rPPG signal directly affects the accuracy of subsequent atrial fibrillation identification. In order to improve the quality of the rPPG signal extracted from the facial video data, in some embodiments of this application, the signal extraction module can be configured to perform the following rPPG signal extraction processing on the facial video data.

[0052] First, to avoid environmental noise at the beginning and end of video recording (such as the patient not being fully prepared at the beginning of recording or the patient losing patience at the end) affecting the accuracy of rPPG signal extraction, in the embodiments of this application, the first 200 frames (approximately 6.7s) and the last 400 frames (approximately 13.4s) of each video are discarded. For the remaining video, 1000 frames (approximately 33.3s) are extracted sequentially for subsequent processing, thus avoiding significant environmental noise interference.

[0053] Next, for each video frame data, face tracking is performed to identify the face region. For example, firstly...The video was read frame by frame using OpenCV software, and the total number of frames was counted. Then, the color image was converted to grayscale image to simplify subsequent calculations and extract image structure information more efficiently. Next, the frontal face detector in the dlib library was used to process each grayscale image. This detector can quickly detect facial regions in the image and return a set of bounding boxes containing candidate face regions. If no face is detected in a frame, the frame is skipped; if multiple candidate regions are detected, the first one is selected as the target face region. For the detected face region, the corresponding region is first cropped from the original color image according to the bounding box coordinates returned by the detector. Then, the region is scaled to a predefined size (e.g., 64×64 pixels) using OpenCV software to facilitate subsequent feature extraction or as input to a neural network.

[0054] Then, a region of interest (ROI) is selected from the face region of each frame of video to exclude irrelevant information in the background. Existing methods typically set the ROI to a fixed-size region, directly selecting a fixed-size ROI region (e.g., forehead) from the face region. However, the inventors found in practice that the ROI extracted in this way is not accurate, because the face size of different individuals varies, and the size of the ROI region on the face of different individuals will also vary. Therefore, in the embodiments of this application, a dynamic ROI region determination method is provided instead of using a fixed-size ROI.

[0055] Specifically, firstly, key points that can characterize or represent the facial shape are extracted from the detected facial region in each frame. For example, 68 facial key points can be extracted by calling the facial shape predictor of the dlib library, as shown in Figure 3. Based on these key points, dynamically sized ROIs are set for the forehead, left cheek, and right cheek respectively. The width of the ROI region on the forehead was set to the length of key points 19 to 24, and the height was set to 1 / 4 of the total face size; the width of the ROI region on the left cheek was set to the length of key points 4 to 48, and the height was set to the length of key point 4 to 1; the width of the ROI region on the right cheek was set to the length of key points 54 to 12, and the height was set to the length of key points 12 to 15. The selected ROI regions are shown in Figure 4.

[0056] The three ROI regions shown in Figure 4 are not of fixed size, but are determined based on the key point positions extracted from the facial regions of different individuals and the size of the specific facial regions. Thus, the size of the ROI region can be dynamically adjusted according to the position of 68 key points, which can not only adaptively reflect the differences in individual facial structures, but also provide a more accurate region division for the subsequent extraction and analysis of facial temporal feature signals.

[0057] After determining the three ROI regions of the forehead, left cheek, and right cheek, the green color of the ROI region in each frame is calculated.The average RGB value of the channel is used as the rPPG signal intensity point in that frame.

[0058] In another embodiment of this application, the recognition model further includes a signal filtering module for filtering the rPPG signal from the signal extraction module. In order to reduce noise caused by motion or changes in ambient light intensity, the signal filtering module uses a fourth-order Chebyshev type II bandpass filter (cutoff frequency: 0.5-3Hz) to filter the obtained rPPG signal. The signal comparison before and after filtering is shown in Figure 5a and Figure 5b.

[0059] After the above processing by the signal extraction module and the signal filtering module, three sets of high-quality rPPG signals from different facial regions are extracted as features for subsequent classification.

[0060] It should be noted that the above embodiment uses three ROI regions as examples only, and is not intended to impose any limitations. In other embodiments, one, two or more ROI regions may be used, and are not limited to the forehead, left cheek, and right cheek. Furthermore, it should be understood that the signal filtering module is an optional module and is only a preferred implementation method. In other embodiments, the signal filtering module for filtering can also be omitted, which does not affect the implementation of the solution of this application. In some other embodiments, the filtering process can also be performed by the signal extraction module together.

[0061] 3. Facial feature extraction module

[0062] In the embodiments of this application, in addition to using the rPPG signal extracted from the facial video data as a feature for subsequent classification, facial temporal features are also introduced for subsequent classification. The inventors found in their research that the skin color changes observed in facial videos not only depend on the underlying blood flow pulse (which can be characterized by the rPPG signal), but are also affected by other factors, such as ambient light and facial movement, which can also cause skin color changes in facial videos. Therefore, in the embodiment on page 7 / 13 of the specification of this application, 10 CN 121400797 A, the facial feature extraction module learns the interrelationships between various factors through multi-scale convolution feature extraction and multi-scale feature transformation modules, and extracts facial temporal feature sequences from them. Considering that facial temporal feature signals are related to video features (such as skin color and hair), the multi-scale convolutional feature extraction module uses 2D convolutional neural networks (CNN) to extract convolutional features, extracting feature signals rich in facial features from the RGB three channels of each video frame in the input facial video frame sequence.

[0063] Assuming that there are T video frames to be processed, the input of the multi-scale convolutional feature extraction module can be expressed as:

[0064]

[0065] where represents the i-th video frame image, and and are the height and width of the video frame image respectively, and 3 representsRGB three channels. The output of this multi-scale convolutional feature extraction module is a temporal feature signal sequence rich in facial features, that is, the feature transformation value corresponding to each frame. For each frame, the module uses three sets of 2D convolution operations with different kernel sizes, namely 3×3, 5×5, and 7×7, to extract multi-scale spatial features. Specifically, let:

[0066]

[0067] where, , and represent 3×3, 5×5, and 7×7 2D convolution operations respectively, and , and represent batch normalization processing of the outputs of , , and. indicates that the activation function (Rectified Linear Unit) is applied to the batch normalized result, that is, all negative values ​​are set to 0, positive values ​​are retained, and nonlinear transformation is introduced. Taking the 3×3 2D convolution operation as an example, the result is denoted as, which represents the feature mapping obtained after 3×3 convolution. Subsequently, these three sets of features are concatenated along the channel dimension:

[0068]

[0069] Next, a 2D average pooling layer (AvgPool) and a dropout layer (dropout probability of 0.5) are applied to the concatenated result:

[0070]

[0071] The average pooling layer represents the application of two-dimensional average pooling operation to the input feature map. Its main purpose is to reduce the spatial size of the feature map while maintaining the main information, thereby reducing the number of parameters and improving the robustness of the model. The dropout layer is mainly used to prevent the model from overfitting. During training, it randomly sets the output of some neurons to zero with a certain probability (e.g., dropout probability of 0.5), thereby forcing the 2D convolutional neural network not to rely too much on certain local features.

[0072] The feature representation obtained by the above multi-scale convolutional feature extraction module is the feature expression of each frame of video, providing input for the subsequent multi-scale feature transformation module. For ease of subsequent description, the feature representation of each video frame output by the multi-scale convolutional feature extraction module is represented by a feature map (where and are the spatial dimensions of the feature map after pooling, respectively), which is provided as input to the feature transformation module. The feature transformation module extracts facial feature-related information from the feature map F through further convolutional operations.

[0073] Specifically, for the input features, the feature transformation module also uses three sets of different convolution kernel sizes, namely 3 × 3, 5 × 5, and 7 × 7 convolution operations:

[0074]

[0075] Next, the features obtained from the three convolution branches are concatenated along the channel dimension:

[0076]

[0077] And the dimension is compressed to 1 × 1 using Adaptive Global Average Pooling (AdaptiveAvgPool2d):

[0078]

[0079] Furthermore, a 1 × 1 convolution is applied to fuse 24 channels into 1 channel:

[0080]

[0081] And the result is further flattened into a one-dimensional vector:

[0082]

[0083] Then, it is mapped to the hidden layer through a fully connected layer (FC):

[0084]

[0085] Finally, the signal after feature transformation is obtained through a second fully connected layer:

[0086]

[0087] In general, the above-mentioned processing of the facial feature extraction module can also be understood as follows: For the input RGB channel frame video, each frame is processed by the multi-scale convolution feature extraction module to obtain the image features of the video frame:

[0088]

[0089] Then, the image features of each frame are input into the multi-scale feature transformation module:

[0090]

[0091] The final obtained temporal feature sequence P of the entire video is:

[0092]

[0093] Thus, through the facial feature extraction module, the video is transformed into a temporal feature sequence, which is used as the temporal features of the face for subsequent classification.

[0094] 4. Feature-level attention mechanism module

[0095] This module aims to effectively fuse the rPPG signals extracted by the signal extraction module from three different regions (forehead, left cheek, and right cheek) in the video with the facial temporal feature signals obtained by the facial feature extraction module, thereby adaptively adjusting the importance of each feature and providing a more discriminative input for the subsequent classification module. This processing takes into account the complementarity and difference between the rPPG signals of different facial regions and facial temporal feature information in AF detection, thereby effectively improving the accuracy and robustness of AF recognition.

[0096] Specifically, the input data Z of this feature-level attention mechanism module can be expressed as:

[0097]

[0098] where represents the number of frames contained in the video, that is, a total of T video frames, batch represents the batch size, and each frame has 4 feature channels (3 rPPG signal channels corresponding to ROI regions and 1 facial temporal feature signal channel). The feature-level attention mechanism module performs a linear transformation on the input through a single-layer fully connected network to calculate attention weights, which are then used to weight the input. This linear transformation can be expressed as:

[0099]

[0100] where zt is the t-th element in the input Z, , t=1 ,… ,T; is the weight matrix; is the bias vector. Next, the attention weights are obtained by normalization using a normalized exponential function (Softmax function):

[0101]

[0102] Finally, the weighted input features are used to obtain the attention-enhanced features:

[0103]

[0104] where represents element-wise multiplication. Therefore, for the input data Z, the final output temporal feature data of this feature-level attention mechanism module can be denoted as temporal data. The operation of the feature-level attention mechanism module highlights important feature information, thus providing a more discriminative input for the subsequent classification module.

[0105] 5. Classification Module

[0106] In one embodiment of this application, an attention-enhanced bidirectional long short-term memory network is used as the classification module, which models the temporal data output by the feature-level attention mechanism module and outputs the classification result. In this embodiment, the attention-enhanced bidirectional long short-term memory network first uses a single-layer bidirectional LSTM to process the input data. The number of hidden units used in the LSTM is 32. The outputs of the forward and backward LSTMs are as follows:

[0107]

[0108] After concatenating the forward and backward outputs, the temporal output is as follows:

[0109]

[0110] Thus, the output sequence of the entire LSTM layer is as follows:

[0111]

[0112] Based on the above description of the output of the single-layer bidirectional LSTM, the calculation process of the LSTM unit is further detailed. Specifically, at each time step, the LSTM unit performs the following operations in sequence:

[0113]

[0114] where , , , , and represent the forget gate, input gate, candidate unit state, unit state update, output gate, and hidden state, respectively; is the input vector at time step ; is the hidden state at the previous time step; represents the sigmoid activation function (page 13 of the specification, CN 121400797); is the hyperbolic tangent function; represents the element-wise product; is the weight matrix; and is the bias vector.

[0115] In order to further capture the local and global dependencies in the time series, this attention-enhanced bidirectional long short-term memory network further adopts a multi-head attention mechanism to process the output H of the single-layer bidirectional LSTM. The embedding dimension used is 64, and the number of heads is 2. The processing of the multi-head attention mechanism can be described as follows:

[0116]

[0117] The calculation formula for each attention head is:

[0118]

[0119] And for each head, we have:

[0120]

[0121] where is the weight matrix.

[0122] The temporal features after the above multi-head attention processing are denoted as. Then, the feature sequence is further subjected to temporal mean pooling to obtain:

[0123]

[0124] Then, it is mapped to the category space through a fully connected layer. The number of categories is inconsistent under different tasks, so the parameter settings are also different. Here, N represents the number of categories of the task (atrial fibrillation prediction is binary, heart rate detection is tri-class, blood oxygen saturation detection is binary, and blood pressure detection is quadruple). Let the parameters of the fully connected layer be and the bias be , then the original classification output score is:

[0125]

[0126] Therefore, the classification score vector obtained after the fully connected layer can be denoted as:

[0127]

[0128] where and represent the unnormalized scores of category 1, category 2 to category N, respectively. Finally, the above classification results are used to calculate the predicted probability through a normalized exponential function:

[0129]

[0130] Here, the predicted probability of the category is represented.

[0131] The working process of each component of the recognition model of this application embodiment has been introduced above with reference to Figure 1. The recognition model can be trained based on the facial video data of AF patients and AF patients collected above with 12-lead ECG results as the label for AF classification. For training of physiological signal monitoring, the corresponding physiological signal values ​​monitored by the recorded patients during video recording are used. In each training iteration, the facial video data is input into the recognition model. The signal extraction module is used to extract the corresponding rPPG signal from the input facial video data. The facial feature extraction module obtains the temporal feature sequence of the face from the input facial video data. The feature-level attention mechanism module fuses and weights the rPPG signal and the temporal feature sequence of the face. The fused feature sequence data is then classified using the attention-enhanced bidirectional long short-term memory network to output the predicted probability. The predicted result is then compared with the corresponding label, the loss function is calculated, and backpropagation is then performed to adjust the relevant parameters of the recognition model.

[0132] During training, the recognition model uses the cross-entropy loss function and the Adam optimizer (learning rate 0.001), with a batch size of 2 training samples input each time to reduce computational and storage burden. The training cycle is set to 100. All experiments were completed in Python 3.9, PyTorch 2.0 and PyCharm 2024 environments. The cross-entropy loss function is used to measure the difference between the predicted result and the real sample label. The real label of each training sample can be represented in one-hot encoding form, i.e.:

[0133]

[0134] The cross-entropy loss function is defined as:

[0135]

[0136] where is the predicted probability of the category. Substituting the normalized exponential function form of the classification result output by the recognition model mentioned above, it can be expressed as:

[0137]

[0138] where represents the loss value of the overall recognition model.

[0139] In another embodiment of this application, a contactless physiological monitoring method based on deep learning is also provided. This method involves inputting a facial video of a user to be detected into one or more pre-trained recognition models. Each recognition model outputs different physiological monitoring results. The physiological monitoring results output by each recognition model may include a prediction of one of the following: the probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level. The structure of each recognition model is shown in Figure 2 above, which is used to extract rPPG signals and facial temporal feature sequences from the input facial video data; assign attention weights to the rPPG signals and facial temporal feature sequences to obtain weighted features; and...The weighted features are input to the classification module to obtain predictions of atrial fibrillation probability, heart rate, blood oxygen saturation, or blood pressure. The classification module is the attention-enhanced bidirectional long short-term memory network described above in conjunction with Figure 2. This recognition method takes into account both the rPPG signal reflecting heart rate status and the temporal features reflecting captured facial dynamic changes, improving the accuracy and robustness of the recognition.

[0140] In another embodiment of this application, a computer-readable storage medium is also provided, on which a computer program or executable instructions are stored. When the computer program or executable instructions are executed by a processor or other computing unit, the technical solution described in the foregoing embodiments is implemented. The implementation principle is similar and will not be repeated here. In the embodiments of this application, the computer-readable storage medium can be any tangible medium capable of storing data and readable by a computing device. Examples of computer-readable storage media include hard disk drives, network attached storage (NAS), read-only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical or non-optical data storage devices. Computer-readable storage media may also include computer-readable media distributed across a network-coupled computer system so that computer programs or instructions can be stored and executed in a distributed manner.

[0141] References to “various embodiments,” “some embodiments,” “one embodiment,” or “embodiment,” etc., in this specification refer to a particular feature, structure, or property described in connection with the embodiment, which is included in at least one embodiment. Therefore, the appearance of the phrases “in various embodiments,” “in some embodiments,” “in one embodiment,” or “in an embodiment,” etc., throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or property may be combined in any suitable manner in one or more embodiments. Thus, a particular feature, structure, or property shown or described in connection with one embodiment may be combined, in whole or in part, with features, structures, or properties of one or more other embodiments without limitation, as long as the combination is not illogical or inoperable.

[0142] The terms "comprising," "having," and similar expressions used in this specification are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. "A" or "an" does not exclude multiple instances. Furthermore, the various elements in the accompanying drawings are for illustrative purposes only and are not drawn to scale.

[0143] Although this application has been described through the above embodiments, it is not limited to what is described herein.The embodiments also include various changes and modifications made without departing from the scope of this application. Instruction sheet 13 / 13 page 16 CN 121400797 A Figure 1 Figure 2 Figure 3 Instruction sheet drawing 1 / 3 page 17 CN 121400797 A Figure 4 Figure 5a Instruction sheet drawing 2 / 3 page 18 CN 121400797 A Figure 5b Instruction sheet drawing 3 / 3 page 19 CN 121400797 A The present application relates to a Abstract deep learning-based contactless recognition of atrial fibrillation and multi-physiological-parameter monitoring system and method, consisting of: extracting a pulse wave signal from a facial video of a user; extracting a facial time-series feature sequence from the facial video of the user; assigning attention weights to the pulse wave signal and the facial time-series feature sequence and performing weighting processing; and predicting a probability of atrial fibrillation, a heart rate range, a blood oxygen saturation level, and a blood grade pressure based on the weighted pulse wave signal and facial time-series features. This solution can effectively improve the robustness and accuracy of atrial fibrillation detection, and simultaneously monitor multiplephysiological parameters of the user synchronously. Abstract

Claims

1. A deep learning-based physiological monitoring system, the system comprising one or more recognition models, the recognition models comprising: The signal extraction module is used to extract pulse wave signals from the user's facial video; The facial feature extraction module is used to extract facial temporal feature sequences from the user's facial video; A feature-level attention mechanism module is used to assign attention weights to the pulse wave signal and the facial temporal feature sequence and perform weighted processing; The classification module is used to output physiological monitoring results based on weighted pulse wave signals and facial temporal features, wherein the physiological monitoring results include predictions for one of the following: probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level.

2. The system according to claim 1, wherein, The one or more recognition models are trained independently and are used to output predicted results for the probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level based on the input user's facial video; and The system also includes an integration module, which receives the user's facial video and feeds it to each recognition model simultaneously, as well as summarizing and outputting the prediction results of each recognition model.

3. The system according to claim 1, wherein the signal extraction module is configured to extract corresponding pulse wave signals from one or more of the following facial regions of interest: forehead, left cheek, and right cheek.

4. The system of claim 1, wherein the signal extraction module is configured to determine the region of interest of the face in the following manner: Extract key points to characterize the facial shape from the detected facial regions in each video frame; The region of interest is determined based on the location of these key points in the facial region and the size of the facial region.

5. The system according to claim 1, wherein the signal extraction module is further configured to perform filtering processing on the extracted pulse wave signal using a fourth-order Chebyshev type II bandpass filter.

6. The system according to claim 1, wherein the facial feature extraction module includes a multi-scale convolutional feature extraction and multi-scale feature transformation module, wherein for each video frame from the input facial video data: The multi-scale convolutional feature extraction module uses a 2D convolutional neural network and three sets of different convolutional kernel sizes to extract convolutional features from the video frame, and then concatenates these three sets of convolutional features into a feature map along the channel dimension. The feature transformation module uses three sets of different convolution kernel sizes to continue extracting facial features from the feature map, and then concatenates the three sets of facial features in the channel dimension to further transform them into a one-dimensional feature vector, and uses this feature vector as the facial temporal feature corresponding to the video frame.

7. The system according to claim 1, wherein the feature-level attention mechanism module is configured to take the pulse wave signal and the facial temporal feature sequence together as multi-channel input, calculate the attention score of each channel through a fully connected layer, and use a normalized exponential function to weight the features of each channel.

8. The system according to claim 1, wherein the classification module is configured as follows: A bidirectional long short-term memory network (LSTM) is used to perform temporal modeling on the feature sequence data after weighted processing by the feature-level attention mechanism module. The outputs of the forward and backward LSTMs are obtained, and the temporal output is obtained by concatenating the forward and backward outputs. The temporal output is processed by a multi-head self-attention mechanism to obtain attention-weighted temporal features; After performing temporal mean pooling on the temporal features, the data is mapped to the category space through a fully connected layer, thereby outputting the physiological monitoring results.

9. The system according to claim 1, wherein the one or more recognition models are pre-trained on a training sample set, wherein the training sample set includes pre-collected facial video data of subjects with and without atrial fibrillation, and corresponding monitoring values ​​of heart rate, blood oxygen saturation, and blood pressure. Each facial video data point is labeled with whether atrial fibrillation is present, along with corresponding monitoring values ​​for heart rate, blood oxygen saturation, and blood pressure.

10. A physiological monitoring method based on deep learning, comprising: A facial video of the user to be detected is provided as input to one or more pre-trained recognition models; Each of the one or more recognition models outputs physiological monitoring results, wherein the physiological monitoring results include predictions for one of the following: probability of atrial fibrillation, heart rate zone, blood oxygen saturation level, and blood pressure level; The recognition model described therein is the recognition model in the system according to claim 1.

Images