Atrial fibrillation detection method based on face video

Abstract

The invention relates to the technical field of atrial fibrillation detection, in particular to an atrial fibrillation detection method based on a face video, and the method comprises the steps: extracting a rough rPPG signal and a motion level from the face video, and carrying out the multi-task learning through combining a GAN network and an LSTM module with an atrial fibrillation recognition network of an attention mechanism, thereby achieving the recognition of the class of atrial fibrillation. According to the method, non-inductive detection of atrial fibrillation can be realized only by acquiring the face video, so that the user experience is good, and the safety coefficient is high; besides, motion level analysis is introduced, multi-task learning is carried out in combination with a GAN network and an atrial fibrillation recognition network with an LSTM module and an attention module, and higher robustness is achieved for noise caused by face motion.

Description

Technical Field

[0001] The invention belongs to the technical field of atrial fibrillation detection, and in particular relates to an atrial fibrillation detection method based on facial video. Background Art

[0002] Atrial fibrillation (hereinafter referred to as AF) is the most common arrhythmia, with an incidence rate of 6% to 8% in people over 65 years old. It is estimated that by 2050, the number of patients with AF will reach 66 million, becoming a major public health problem. Early screening for AF is crucial, and traditional electrocardiogram (ECG) diagnostic methods have problems such as cumbersome operation and high cost. Photoplethysmography (PPG) is recommended as an alternative, while remote photoplethysmography (rPPG) technology can capture pulse signals from facial videos and is suitable for special populations such as newborns and burn patients. However, rPPG signals are easily disturbed by ambient light and motion, requiring efficient denoising technology. The accuracy of AF detection can be improved by optimizing signal processing algorithms and applying deep learning methods.

[0003] However, despite these advantages, rPPG technology faces many challenges in its application in atrial fibrillation detection. First, the rPPG signal is very weak and easily affected by ambient lighting and motion interference. Since the skin's reflection and absorption characteristics of light vary with the intensity and direction of the light source, the light intensity distribution on the face is often uneven. This spatial distribution inhomogeneity, especially the difference in light intensity in different skin areas, can easily lead to signal contamination by ambient light noise. Secondly, motion noise during facial video acquisition is another major challenge. Even slight facial shaking or expression changes can significantly interfere with the rPPG signal. In addition, the intensity of reflected light in facial videos may vary with factors such as distance and angle, further increasing the difficulty of signal extraction. Therefore, atrial fibrillation detection for facial videos places extremely high demands on denoising technology. Summary of the invention

[0004] In view of this, the present invention aims to provide an atrial fibrillation detection method based on facial video, extract facial landmarks and motion levels from facial videos, and use GAN (generative adversarial network) and LSTM (long short-term memory network, Long Short-Term Memory) modules combined with an atrial fibrillation recognition network with an attention mechanism to perform multi-task learning, thereby achieving non-sensing detection of atrial fibrillation only by acquiring facial videos; at the same time, multi-task learning is combined with a GAN network and an atrial fibrillation recognition network with an LSTM module and an attention module, which has stronger robustness to noise caused by facial movement.

[0005] A method for detecting atrial fibrillation based on facial video comprises the following steps: S1: Obtain an atrial fibrillation dataset, which includes facial videos of multiple samples and corresponding atrial fibrillation categories; extract facial landmarks from the facial videos, and then obtain the motion level corresponding to the facial videos according to the facial landmarks; S2: Obtain a rough rPPG signal from the facial landmarks in step S1, and then input the rough rPPG signal into the trained GAN network to obtain a rPPG signal; then extract a feature vector from the rPPG signal; S3: construct an atrial fibrillation recognition network with an LSTM module and an attention module; use the feature vector obtained in step S2, and the atrial fibrillation category and movement level in step S1 to train the atrial fibrillation recognition network to obtain an atrial fibrillation recognition model; S4: Repeat the operations of steps S1 to S2 for the facial video to be detected to obtain the rPPG signal to be detected, and input the rPPG signal to be detected into the atrial fibrillation recognition model obtained in step S3 to obtain the corresponding atrial fibrillation category probability.

[0006] Furthermore, in step S1, facial landmarks are extracted from the facial video using the MediaPipe algorithm; and the motion level is calculated using the following formula in combination with the facial landmarks: ; Where m represents the motion level, t represents the time of the facial video, and T represents the time window moving along the time axis of the facial video. represents the jth facial landmark point in the facial video at time t, and J represents the total number of facial landmark points.

[0007] Furthermore, in step S2, a ROI region is determined among the facial landmarks, and a CHROM algorithm is performed on the ROI region to obtain a rough rPPG signal.

[0008] Furthermore, in the process of extracting features from the rPPG signal in step S2, peak point detection is performed on the rPPG signal to obtain the RR interval, HRV time domain analysis, HRV frequency domain analysis and HRV nonlinear analysis are performed based on the RR interval, and the three analysis results are combined to obtain a feature vector.

[0009] Furthermore, the atrial fibrillation recognition network in step S3 also includes a linear input module; after the feature vector is input into the linear input module, the extracted features are input into the attention module for feature enhancement; the output features of the attention module are then input into the LSTM module to capture the temporal features between the features, and the output features are respectively output through two linear layers to output the atrial fibrillation category probability and the motion level.

[0010] Furthermore, the linear input module includes a cascaded linear layer and a convolutional layer; wherein the input feature is input into the linear layer, activated by a ReLU activation function, and then input into the convolutional layer; the output feature of the convolutional layer is activated by a ReLU activation function and then input into the attention module.

[0011] Furthermore, the attention module includes a channel attention submodule and a feature processing submodule; wherein the channel attention submodule is an ECA-Net structure; the feature processing submodule includes two cascaded convolutional layers, in each convolutional layer, a convolution operation is performed on the input features and then a ReLU activation operation is performed to obtain the output features.

[0012] Furthermore, the loss function used in training the atrial fibrillation recognition network in step S3 is is the loss function for atrial fibrillation and motion level loss function The linear addition of is: ; in, Represents the balance parameter between the two losses; Atrial fibrillation class loss function As follows: ; in, represents the true atrial fibrillation category of the nth sample, represents the predicted atrial fibrillation category probability of the nth sample; Motion Level Loss Function As follows: ; in, represents the motion level of the nth sample, represents the predicted motion level of the nth sample.

[0013] Compared with the prior art, the invention can achieve the following beneficial effects: In the facial video-based atrial fibrillation detection method created by the present invention, facial features and movement patterns are identified by analyzing facial videos, so that the proposed method only needs facial videos as input to achieve non-invasive detection of atrial fibrillation; at the same time, the two are combined with a generative adversarial network (GAN), an atrial fibrillation recognition network with a long short-term memory network (LSTM) and an attention mechanism, thereby enhancing the robustness to noise caused by facial movement. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings: Figure 1 A schematic diagram of a flow chart of an atrial fibrillation detection method based on facial video according to an embodiment of the present invention; Figure 2 A schematic diagram of the process of step S2 described in the embodiment of the present invention; Figure 3 A schematic diagram of the overall network structure of the GAN described in the embodiment of the present invention; Figure 4 A schematic diagram of the network structure of the generator described in the embodiment of the present invention; Figure 5 A schematic diagram of the network structure of the identifier described in the embodiment of the present invention; Figure 6 A schematic diagram of the network structure of the atrial fibrillation recognition network described in an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0016] like Figures 1 to 6 As shown, the facial video-based atrial fibrillation detection method described in the embodiment of the present invention includes the following steps: S1: Obtain an atrial fibrillation dataset, which includes facial videos of multiple samples and corresponding atrial fibrillation categories; extract facial landmarks from the facial videos, and then obtain the motion level corresponding to the facial videos based on the facial landmarks.

[0017] In an embodiment of the present invention, the atrial fibrillation data set is a self-built data set, that is, the Logitech C920 PRO webcam is used for video acquisition. The data set contains 200 experimental subjects, among which the ratio of atrial fibrillation and normal people is 1:1, totaling 16.7 hours. The data set is divided into a training set and a validation set at a ratio of 80% and 20%, that is, 160 experimental subjects are selected as training data and 40 experimental subjects are selected as test data. The video acquisition process is carried out under ambient light without auxiliary light source. During the acquisition process, the experimental subject maintains a natural state and looks directly at the camera as much as possible. The shooting time is five minutes, and physiological data such as ECG and ppg need to be collected synchronously as a reference standard.

[0018] In step S1, facial landmarks are extracted from the facial video using the MediaPipe algorithm. MediaPipe is an open source face analysis tool that supports real-time detection and tracking of faces and can accurately locate 468 facial landmarks. These landmarks provide an important basis for evaluating the subject's head movement, facial expression changes, etc., and help determine motion interference in rPPG signals.

[0019] Use MediaPipe to detect the face in each frame of video and extract 468 facial landmarks. These facial landmarks can be used to calculate the amplitude of head movement and facial micro-expressions through their displacement changes between adjacent frames. On the timeline, a sliding window is used to evaluate the movement of the landmarks in each time period to measure the movement level and perform interference filtering during signal extraction. Specifically, after using MediaPipe to detect the facial landmarks, the movement level is calculated using the following formula: ; Where m represents the motion level, t represents the time of the facial video, and T represents the time window moving along the time axis of the facial video. represents the jth facial landmark point in the facial video at time t, and J represents the total number of facial landmark points.

[0020] S2: Obtain a rough rPPG signal from the facial landmarks in step S1, and then input the rough rPPG signal into the trained GAN network to obtain an rPPG signal; then extract a feature vector from the rPPG signal.

[0021] like Figure 2 As shown in the figure, 468 facial landmarks are defined as regions of interest (ROI) according to the positions of the left and right cheeks; secondly, the RGB channels are extracted for the part within the selected ROI; then, each color channel (R, G, B) is filtered and detrended separately. The filtering step uses a bandpass filter (0.8-2.5 Hz). After that, detrending is performed, and linear regression is used to remove long-term trends and low-frequency drifts in the signal; finally, the CHROM algorithm (published in the paper "Robust pulse rate fromchrominance-based rPPG" in "IEEE Transactions on Biomedical Engineering") is used to obtain a rough rPPG signal, which will be used as the input of the GAN network. Finally, the rough rPPG signal is denoised by the GAN network to obtain a high-quality rPPG signal.

[0022] Specifically, the structure of the GAN network is as follows Figure 3As shown, the GAN network consists of a generator and a discriminator. The generator is used to map the rough rPPG signal X to an rPPG signal G(X) close to the reference PPG signal Xc; the discriminator is used to distinguish the true value Xc from the rPPG signal G(X); in order to better pair the input and output, the embodiment of the present invention sets the rough rPPG signal X as a condition in the discriminator, so the input of the discriminator consists of two channels, namely (G(X), X) or (Xc, X). The discriminator outputs a lower score for the input (G(X), X) and a higher score for the input (Xc, X). Through adversarial learning between the generator and the discriminator, the characteristics of the PPG signal are continuously learned, so that the output rPPG signal has a distribution close to the reference PPG signal.

[0023] The structure of the generator is as follows Figure 4 As shown in the figure, the encoder and decoder include jump connections. Both the encoder and the decoder have six hidden layers. Specifically, the encoder consists of six one-dimensional convolutional layers (Conv1d), and the activation function corresponding to each one-dimensional convolutional layer is the parameter rectified linear activation function (Parametric Rectified Linear Unit, PReLU); the decoder consists of six deconvolutional layers (Deconv1d), and the activation function corresponding to the first 5 deconvolutional layers is the PReLU activation function. The activation function of the last deconvolutional layer is the hyperbolic tangent function (tanh). The jump connection between the encoder and the decoder is used to transfer fine-grained features from the encoder to the corresponding features in the decoder, which is very important for the generator to construct a high-quality waveform.

[0024] The structure of the discriminator is as follows Figure 5 As shown, it is composed of a plurality of one-dimensional convolutional layers (Conv1d) and a fully connected layer (Fullyconnection, Fc) cascaded. In the embodiment of the present invention, the discriminator is composed of 7 one-dimensional convolutional layers and a fully connected layer cascaded. In the first 6 one-dimensional convolutional layers (Conv1d), the leaky linear rectifier function (LeakyReLU) is selected to activate the output features, and batch normalization (BN) is used to accelerate convergence. In the seventh one-dimensional convolutional layer (Conv1d), only LeakyReLU is selected to activate the output features, and batch normalization of the features is not required.

[0025] In the embodiment of the present invention, the training data of the GAN network comes from UBFC-rPPG and UBFC-Phys. UBFC-rPPG (published at https: / / sites.google.com / view / ybenezeth / ubfcrppg) contains 42 RGB videos containing sunlight and indoor lighting. The video is recorded using a Logitech C920 HD Pro webcam with a resolution of 640×480 and a frame rate of 30fps. The real BVP (Blood Volume Pulse) signal and HR value (Heart Rate) are collected by CMS50E. UBFC-Phys (published at https: / / sites.google.com / view / ybenezeth / ubfc-phys) provides 168 videos, 64HZ BVP (blood volume pulse) and 4HZ EDA (skin galvanic response) signals and other information. There are 168 videos from 56 subjects (46 females and 10 males) in the dataset, each with a resolution of 1024x1024 pixels, a capture frame rate of 35fps, and a video length of 3 minutes. The BVP signal is reduced to 30fps by cubic spline interpolation to keep it consistent with the video. When training the GAN network provided in the embodiment of the present invention, the rough rPPG signal and BVP signal are extracted by the CHROM algorithm to train the GAN network.

[0026] In the process of extracting features from the rPPG signal in step S2, the peak point detection of the rPPG signal is performed to obtain the RR interval, and HRV (heart rate variability) time domain analysis, HRV frequency domain analysis and HRV nonlinear analysis are performed based on the RR interval, and the three analysis results are combined to obtain a feature vector. The RR interval refers to the time interval between two R waves in a continuous electrocardiogram (ECG), which is usually used to measure HRV. The R wave is the most significant part of the electrocardiogram and is usually used to determine each heart beat. Therefore, the RR interval reflects the time interval between each heart beat.

[0027] The results of HRV time domain analysis include: ; Among them, SDNN represents the standard deviation of all normal sinus beat intervals, in milliseconds, reflecting the degree to which heart rate variability deviates from the mean value. It is RR intervals, K represents the number of cycles of the RR interval, is the average value of the RR interval of N heart beats; ; Among them, RMSSD represents the root mean square of the difference between all adjacent RR intervals, in ms, reflecting the regulatory effect of the parasympathetic nerve on heart rate; ; Among them, SDSD reflects the difference between consecutive RR intervals and can provide sensitivity to dynamic changes in heart rate variability; ; in, It is the interval between the qth heartbeats, Q is the total number of heartbeats. Mean IBI reflects the average value of the heartbeat interval, usually in milliseconds; ; in, Indicates that in the sliding window The standard deviation of the IBI in the sliding window starts from the starting position of the IBI sequence and slides step by step. By calculating each sliding window IBI standard deviation within sliding window It can slide forward with a fixed step size 𝑠, usually 1 or 2 IBI points; Indicates the first heartbeat interval, Represents a sliding window The mean value of internal IBI; ; in, Represents the difference between adjacent RR intervals The standard deviation reflects the fluctuation degree of changes in the intervals between adjacent heartbeats. It represents the average value of the difference between adjacent RR intervals; ; Among them, CVSD is a standardized index that reflects the fluctuation of continuous changes in heartbeat (short-term heart rate variability). It emphasizes fast-changing components (such as parasympathetic nervous regulation); ; Among them, CVNN is an overall measure of heart rate variability, reflecting the variability of the time intervals between all normal heartbeats. It focuses on the overall heart rate variability (long-term regulation), and Mean NN represents the average value of normal heartbeat intervals. ; Among them, PNN50 represents the proportion of the difference between two consecutive NN intervals (i.e., RR intervals under normal sinus rhythm) greater than 50 milliseconds within a period of time. Among them, 50 milliseconds is used to define the interval with larger differences to evaluate the fluctuation of heart rate. A higher PNN50 value usually indicates better heart health and stronger autonomic nervous regulation function; K 50 is the number of times the RR interval difference is greater than 50 milliseconds; ; Among them, NN50 represents the number of differences between two normal sinus beats (i.e., normal RR intervals) greater than 50 milliseconds; ; Among them, PNN20 represents the proportion of the difference between two consecutive NN intervals greater than 20 milliseconds within a period of time; K 20 is the number of times the RR interval difference is greater than 20 milliseconds; ; NN20 represents the number of differences between two normal sinus beats (i.e., normal RR intervals) greater than 20 milliseconds.

[0028] The results of HRV frequency domain analysis include: ; Among them, LF represents the low-frequency energy of the pulse wave of the experimental subject in the frequency range [μ, ν], μ and ν are both positive decimals, and PSD represents the power spectral density, that is, the power within the unit frequency bandwidth. Specifically, PSD describes how the power of the signal is distributed as the frequency changes; ; Among them, HF represents the high-frequency energy of the pulse wave of the experimental subject in the frequency range [ν,ξ], and ν and ξ are both positive decimals; ; Among them, LF / HF represents the relative ratio and balance of sympathetic nerve activity and parasympathetic nerve activity.

[0029] For HRV nonlinear analysis, use the Poincare map. For a continuous heartbeat interval, use the rth heartbeat interval as the horizontal coordinate and the r+1th heartbeat interval as the vertical coordinate to draw an elliptical scatter plot on a two-dimensional plane. The formula used is as follows: ; Where SD1 is the semi-major axis of the ellipse, E is the length of the heartbeat interval sequence, and x r is the rth data point of the time series; ; where SD2 is the semi-minor axis of the ellipse.

[0030] S3: construct an atrial fibrillation recognition network with an LSTM module and an attention module; use the feature vector obtained in step S2, and the atrial fibrillation category and movement level in step S1 to train the atrial fibrillation recognition network to obtain an atrial fibrillation recognition model.

[0031] Among them, the structure of the atrial fibrillation recognition network is as follows Figure 6 As shown in the figure, the network includes a linear input module, an attention module, and an LSTM module. After the feature vector is input into the linear input module, the extracted features are input into the attention module for feature enhancement; the output features of the attention module are then input into the LSTM module to capture the temporal features between features, and the output features are respectively output through two linear layers, linear-AF and linear-M, to correspond to the output of atrial fibrillation category probability (AF) and movement level (M).

[0032] The linear input module includes a cascaded linear layer Linear and a convolutional layer Conv1D-1; the input feature is input into the linear layer Linear and activated by the ReLU activation function, and then input into the convolutional layer Conv1D-1; the output feature of the convolutional layer Conv1D-1 is activated by the ReLU activation function and then input into the attention module.

[0033] The attention module includes a channel attention submodule and a feature processing submodule; the channel attention submodule is an ECA-Net (Efficient Channel Attention Network) structure; the feature processing submodule includes two cascaded convolutional layers Conv1D-2 and Conv1D-3, each of which performs a convolution operation on the input features and then performs a ReLU activation operation to obtain the output features. The ECA-Net structure (published in the paper "ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks" published in the 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR)) is a core mechanism of ECA-Net that adaptively captures the dependencies between channels through a simple and efficient one-dimensional convolution without the process of dimensionality reduction and dimensionality increase. This design avoids the complex multi-layer perceptron (MLP) structure in the traditional attention mechanism, reducing the model complexity and computational burden. ECA-Net learns the importance of each channel relative to other channels by calculating an adaptive kernel size and applying one-dimensional convolution directly on the channel features.

[0034] The LSTM module includes the LSTM (Long Short-Term Memory) network (published in the paper "Learning to Forget: Continual Prediction with LSTM" in "Neural Computation"). LSTM is a special recurrent neural network (RNN) architecture used to process and predict sequence data. It improves on the traditional RNN and solves the gradient vanishing and gradient exploding problems of the standard RNN when processing long time series data.

[0035] The input and output of each structure in the atrial fibrillation recognition network are shown in Table 1:

[0036] In Table 1, B represents the batch size, L is the length of the input feature, and L US Represents the feature length of the output feature of the linear layer.

[0037] When training the atrial fibrillation recognition network, the dual loss function of multi-task learning is used to consider atrial fibrillation detection and motion level assessment at the same time, that is, the loss function used is the loss function for atrial fibrillation and motion level loss function The linear addition of is: ; in, Represents the balance parameter between the two losses.

[0038] Specifically, atrial fibrillation category detection is regarded as binary classification, so the corresponding loss is binary cross entropy, so that the model outputs the probability of atrial fibrillation or no atrial fibrillation in the sample, that is, the atrial fibrillation category loss function As follows: ; in, represents the true atrial fibrillation category of the nth sample, represents the predicted atrial fibrillation class probability of the nth sample.

[0039] On the other hand, the motion level loss function Use MSE loss (mean square error loss, Mean Squared Error), that is:

[0040] in, represents the motion level of the nth sample, represents the predicted motion level of the nth sample.

[0041] In the training of atrial fibrillation recognition network, the learning rate is set to 0.0001, the training round is 600 rounds, the BatchSize is 16, the optimizer selects the momentum stochastic gradient descent SGDM optimization algorithm, and the momentum coefficient is set to 0.9. The momentum term of the optimizer accumulates the previous gradient to maintain the consistency of the update direction, that is: ; in, represents the speed (momentum) of the sth step, is the momentum factor, which indicates how well the current gradient is combined with the previous gradient. is the learning rate, is the current parameter The gradient of the loss function.

[0042] The parameter update formula becomes: ; This means that the update direction at each step is a weighted sum of past gradients, making the optimization process smoother, reducing jitter and avoiding falling into local minima.

[0043] S4: Repeat the operations of steps S1 to S2 for the facial video to be detected to obtain the rPPG signal to be detected, and input the rPPG signal to be detected into the atrial fibrillation recognition model obtained in step S3 to obtain the corresponding atrial fibrillation category probability.

[0044] In the embodiment of the present invention, the probability of the detected atrial fibrillation category needs to be verified. Among them, atrial fibrillation detection is a binary classification problem, that is, the detection results are atrial fibrillation and non-atrial fibrillation. The accuracy (ACC), sensitivity (SEN), specificity (SPE), positive predictive value (PPV), negative predictive value (NPV) and the area under the curve (AUC) of the receiver operating characteristic curve (ROC) are used to evaluate the accuracy of atrial fibrillation detection. The corresponding formula is as follows: ; ; ; PPV ; NPV ; The AUC value is between 0 and 1, indicating the performance of the model at different thresholds. The closer the AUC value is to 1, the stronger the model's ability to distinguish. In the above formula, TP represents the number of samples correctly identified as atrial fibrillation, that is, the number of samples that are actually atrial fibrillation and predicted as atrial fibrillation, FP represents the number of samples incorrectly identified as atrial fibrillation, that is, the number of samples that are not actually atrial fibrillation but predicted as atrial fibrillation, TN represents the number of samples correctly identified as non-atrial fibrillation, that is, the number of samples that are actually non-atrial fibrillation and predicted as non-atrial fibrillation, and FN represents the number of samples incorrectly identified as non-atrial fibrillation, that is, the number of samples that are actually atrial fibrillation but predicted as non-atrial fibrillation.

[0045] In addition, the chi-square test can be used to check the correlation between test pairs, such as the IBI-derived prediction and the actual occurrence of atrial fibrillation. The results of the chi-square test include the p-value and the chi-square value (χ2). Since the p-value is less than 0.05, the test pairs are considered highly correlated. In addition, the chi-square value (χ2) can also be regarded as a relative indicator, where the larger the χ2, the stronger the correlation between the test pairs.

[0046] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0047] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting atrial fibrillation based on facial video, characterized in that: The following steps are involved: S1: Acquire an atrial fibrillation dataset, the atrial fibrillation dataset comprising multiple sample facial videos and corresponding atrial fibrillation categories; extract facial landmarks from the facial videos, and then obtain a motion level corresponding to the facial videos according to the facial landmarks; S2: Obtain a rough rPPG signal from the facial landmarks in step S1, and then input the rough rPPG signal into the trained GAN network to obtain a rPPG signal; and then extract a feature vector from the rPPG signal; S3: constructing an atrial fibrillation recognition network with an LSTM module and an attention module; using the feature vector obtained in step S2, and the atrial fibrillation category and movement level in step S1, training the atrial fibrillation recognition network to obtain an atrial fibrillation recognition model; S4: Repeat the operations of steps S1 to S2 for the facial video to be detected to obtain the rPPG signal to be detected, and input the rPPG signal to be detected into the atrial fibrillation recognition model obtained in step S3 to obtain the corresponding atrial fibrillation category probability.

2. The facial video-based atrial fibrillation detection method according to claim 1, characterized in that: In step S1, the facial landmarks are extracted from the facial video using the MediaPipe algorithm; and the motion level is calculated using the following formula in combination with the facial landmarks: ； Wherein, m represents the motion level, t represents the time of the facial video, T represents the time window moving along the time axis of the facial video, represents the jth facial landmark point in the facial video at time t, and J represents the total number of facial landmark points.

3. The facial video-based atrial fibrillation detection method according to claim 1, characterized in that: In step S2, a ROI region is determined in the facial landmarks, and a CHROM algorithm is performed on the ROI region to obtain the rough rPPG signal.

4. The method for detecting atrial fibrillation based on facial video according to claim 1, characterized in that: In the process of extracting features from the rPPG signal in step S2, peak point detection is performed on the rPPG signal to obtain the RR interval, HRV time domain analysis, HRV frequency domain analysis and HRV nonlinear analysis are performed based on the RR interval, and the three analysis results are combined to obtain the feature vector.

5. The facial video-based atrial fibrillation detection method according to claim 1, characterized in that: The atrial fibrillation recognition network in step S3 also includes a linear input module; after the feature vector is input into the linear input module, the extracted features are input into the attention module for feature enhancement; the output features of the attention module are then input into the LSTM module to capture the temporal features between features, and the output features are respectively output through two linear layers to output the atrial fibrillation category probability and the motion level.

6. The method for detecting atrial fibrillation based on facial video according to claim 5, characterized in that: The linear input module includes a cascaded linear layer and a convolutional layer; wherein, the input feature is input into the linear layer, activated by a ReLU activation function, and then input into the convolutional layer; the output feature of the convolutional layer is activated by a ReLU activation function and then input into the attention module.

7. The method for detecting atrial fibrillation based on facial video according to any one of claims 1 or 5, characterized in that: The attention module includes a channel attention submodule and a feature processing submodule; wherein the channel attention submodule is an ECA-Net structure; the feature processing submodule includes two cascaded convolutional layers, in each convolutional layer, a convolution operation is performed on the input features and then a ReLU activation operation is performed to obtain the output features.

8. The method for detecting atrial fibrillation based on facial video according to any one of claims 1 or 5, characterized in that: The loss function used in step S3 when training the atrial fibrillation recognition network is the loss function for atrial fibrillation and motion level loss function The linear addition of is: ； in, Represents the balance parameter between the two losses; The atrial fibrillation class loss function As follows: ； in, represents the true atrial fibrillation category of the nth sample, represents the predicted atrial fibrillation category probability of the nth sample; The motion level loss function As follows: ； in, represents the motion level of the nth sample, represents the predicted motion level of the nth sample.

Images