A face video physiological signal extraction method based on gated attention modulation

Abstract

This invention belongs to the fields of computer vision and biosignal processing, and discloses a method for extracting physiological signals from facial videos based on gated attention modulation. The method includes: video input and preprocessing; spatially overlapping 3D convolutional coding to ensure spatial overlap between adjacent receptive fields; a sparse attention mechanism to locally normalize the most relevant key-value pairs for each query; gated modulation of the attention score, suppressing noise response through global pooling and learnable gating; multi-scale temporal modeling to capture temporal dependencies at different scales; and time-frequency fusion prediction, constructing three parallel branches—temporal convolution, frequency domain transformation, and multi-scale convolution—for feature fusion, outputting a predicted waveform. This invention can suppress attention shift and noise interference, enhance the ability to capture weak physiological signals, and improve the accuracy and robustness of vital sign monitoring in complex environments.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and biomedical signal processing technology, and in particular relates to a method for extracting physiological signals from facial videos based on gated attention modulation. Background Technology

[0002] In the early stages, rPPG signal extraction was typically achieved by traditional signal processing techniques to track color changes in specific regions of interest (ROIs) on the face. Specific methods included extracting three color channels from the facial ROI for analysis, such as the G-channel method, or using signal decomposition methods, such as Principal Component Analysis (PCA). However, these methods often rely on prior knowledge such as the optical properties of facial skin, focusing only on specific areas like the forehead or cheekbones and using traditional image processing techniques to monitor color changes. When external interference such as facial expressions or lighting conditions occurs, the accuracy of these methods is significantly affected. With the development of deep learning, techniques for extracting rPPG signals from facial videos using deep learning models have become increasingly sophisticated. Deep learning model-based techniques can be broadly categorized into two types: those based on CNN frameworks and those based on Transformer frameworks. A representative existing model is HR-CNN, a two-step deep learning method based on 2D-CNN that includes a feature extractor and a heart rate estimator. The model first extracts rPPG signals from the video frame sequence and then trains the 2D-CNN extractor to maximize its signal-to-noise ratio. Then, the extracted rPPG signal is input into the HR estimator, which outputs the predicted HR value. The training process minimizes the mean absolute error (MAE) between the predicted value and the labeled HR. Next is DeepPhys, an end-to-end 2D-CNN model based on the VGG style, which simultaneously trains a motion model and an appearance model to recover the BVP signal by identifying key regions in space. However, 2D-CNN-based models suffer from a lack of ability to process temporal information. To address this, the MTTS-CAN model based on DeepPhys was proposed, which introduces a temporal displacement module (TSM) to capture temporal information. Although TSM somewhat compensates for the inability of 2D-CNNs to capture temporal contextual features, it still cannot fundamentally solve the problem. 3D-CNNs can simultaneously analyze the temporal and spatial features of video, making them more suitable for the characteristics of rPPG signals. Therefore, the 3D-CNN-based method PhysNet was proposed. It requires no prior knowledge processing and directly inputs the original RGB video frames into the network backbone, achieving good results. However, CNNs have limited receptive fields, and CNN-based methods struggle with long-term modeling capabilities. With the widespread application of Transformer in the video field, the exploration of using Transformer for rPPG signal estimation is gradually becoming a reality. The attention mechanism of Transformer can learn long sequence information, which is crucial for the modeling process of rPPG signals.Representative models include EfficientPhys-T, which adds several SwinTransformer layers to achieve global spatial attention, and Physformer, which proposes a long-range spatiotemporal attention mechanism based on Transformer. However, most of them do not fully leverage the advantages of Transformer in this task. This may be because they focus too much on the spatial information of long sequences, thus failing to fully utilize Transformer's ability to model longer time series. Furthermore, due to the inherent characteristics of video, the weakness of rPPG signals, and noise from artifacts and head movements, the input is often converted into coarse-grained tokens, reducing the signal-to-noise ratio and affecting the effectiveness of global or sparse attention. In addition, they also suffer from signal-to-noise ratio degradation caused by discontinuous input data slices and overly simplistic signal prediction heads in the final part of the model. In summary, Transformer-based rPPG signal extraction models face the technical challenge of "how to improve Transformer's ability to focus on weak periodic physiological signals under complex environmental noise and reduce the interference of redundant frames on attention distribution." Summary of the Invention

[0003] The purpose of this invention is to provide a method for extracting physiological signals from facial videos based on gated attention modulation, so as to solve the above-mentioned technical problems.

[0004] This invention addresses the following problems existing in the prior art:

[0005] (1) The Transformer structure is prone to attention shift when processing facial videos, causing the model to focus more on static salient areas rather than subtle changes in skin color;

[0006] (2) There is a lot of redundancy and environmental noise in the video frames, which reduces the signal-to-noise ratio of physiological signals;

[0007] (3) Existing prediction structures usually only use a single temporal convolution, which does not make full use of the periodic characteristics of physiological signals in the frequency domain and at multiple scales.

[0008] A method for extracting physiological signals from facial videos based on gated attention modulation is proposed to improve the stability and accuracy of rPPG signal modeling in complex environments.

[0009] To address the aforementioned technical problems, the specific technical solution of the facial video physiological signal extraction method based on gated attention modulation of the present invention is as follows:

[0010] A method for extracting physiological signals from facial videos based on gated attention modulation includes the following steps: Step 1: Video input and preprocessing: Input video sequences and perform normalization processing; Step 2: Spatially overlapping 3D convolutional coding: Perform convolution operations on video frames using 3D convolution kernels, ensuring spatial overlap between adjacent convolutional receptive fields; Step 3: Sparse attention mechanism: Linearly map the input features to obtain query Q, key K, and value V, calculate the relevance matrix, and select the K most relevant key-value pairs for each query for local normalization; Step 4: Gated modulation of attention scores: After the scaling dot product attention calculation is completed, the attention matrix is ​​globally pooled, and a learnable gating modulation module is introduced to generate gating weights to modulate the intensity of the original attention distribution; Step 5: Multi-scale temporal modeling: hierarchical downsampling is performed along the time dimension to obtain feature representations at different time scales, and feature fusion is performed after upsampling alignment; Step 6: Time-frequency fusion prediction: three parallel branches are constructed: temporal convolution, frequency domain transformation, and multi-scale convolution. The features output by the three branches are concatenated and fused, and fused with the input features through learnable gating, finally outputting the predicted rPPG waveform.

[0011] Furthermore, step 1 includes the following steps:

[0012] Input video sequence Where T is the number of frames, H and W are the dimensions of the input image, and C is the number of channels. The images are then input into the model after being standardized and normalized. The normalization formula is: , The input video frame, i.e. .

[0013] Furthermore, step 2 includes the following steps:

[0014] Using 3D convolution kernel With a stride of (1,4,4), the convolution output is: Therefore, it can be concluded that ,in Since the convolution kernel size is 7×7 and the stride is 4×4, there is a 3-pixel spatial overlap between adjacent convolution receptive fields. Therefore, adjacent feature blocks share some pixel information, satisfying the following: .

[0015] Furthermore, step 3 includes the following steps:

[0016] Flatten the input features to obtain Linear mapping is performed on the flattened features to obtain The correlation matrix is ​​calculated for Q, K, and V to obtain And select the K key-value pairs KV that are most relevant to each query vector Q, i.e. Then local normalization ,like By performing Top-K filtering on the relevance matrix, each query feature is connected only with a few of the most relevant features.

[0017] Furthermore, step 4 includes the following steps:

[0018] Step 4: Gating modulation of attention scores: in standard scaled dot product attention After calculation, the features in the aggregated attention matrix A are... ,in A learnable gating modulation module is introduced to nonlinearly modulate the attention weight matrix, which is then activated by the Sigmoid activation function. Dynamic recalibration of attention intensity Suppressing abnormally high response in the noise region, the final output By globally aggregating the attention matrix and generating gating weights, the intensity of the original attention distribution is modulated, thereby suppressing abnormally high response regions and enhancing the weight distribution in stable regions.

[0019] Furthermore, step 5 includes the following steps:

[0020] In the Transformer, hierarchical downsampling is performed along the time dimension to obtain feature representations at different time scales. Then, scale alignment is achieved through upsampling to realize feature fusion. Different convolutional receptive fields are used to model the time dimension, enabling the model to capture both short-term changes and long-term trends simultaneously.

[0021] Furthermore, step 6 includes the following steps:

[0022] The received input features are The time-frequency fusion prediction head includes three parallel branches:

[0023] i) Temporal convolutional branch used to enhance short-term time dependency modeling capabilities The kernel size K is 3. These are learnable parameters;

[0024] ii) Frequency domain transform branch used to enhance the expression of periodic features within the heart rate band This branch performs a discrete Fourier transform on the input features along the time dimension, i.e. After amplitude filtering or bandwidth constraint in the frequency domain, the result is obtained through inverse transformation. ;

[0025] iii) Construct multiple parallel temporal convolutions with different kernel sizes to capture multi-scale convolution branches that dynamically change at different time scales. ;

[0026] The features output from the three branches are concatenated and fused. Meanwhile, the input features are retained as residual paths. Then, learnable fusion coefficients are introduced. ,in For the Sigmoid function, These are learnable parameters; finally, the two are fused using learnable gating to control the fusion ratio. The final output signal Y is the predicted rPPG waveform.

[0027] The facial video physiological signal extraction method based on gated attention modulation of the present invention has the following advantages:

[0028] (1) By using spatially overlapping three-dimensional convolutional coding, spatial continuity is improved and the ability to express subtle skin color changes is enhanced;

[0029] (2) By introducing a gated attention modulation module after the SDPA output, the attention shift phenomenon is effectively suppressed and the noise resistance is improved;

[0030] (3) Improve the ability to capture periodic physiological signals by jointly modeling in the time and frequency domains; Attached Figure Description

[0031] Figure 1 This is a flowchart of the facial video physiological signal extraction method based on gated attention modulation according to the present invention. Detailed Implementation

[0032] To better understand the purpose, structure, and function of this invention, the following detailed description of a remote blood oxygen monitoring system and method based on a multi-temporal phase offset state space model is provided in conjunction with the accompanying drawings.

[0033] The present invention provides a method for extracting physiological signals from facial videos based on gated attention modulation, comprising the following steps:

[0034] Step 1: Video Input and Preprocessing: Input video sequence Where T is the number of frames, H and W are the dimensions of the input image, and C is the number of channels. The images are then input into the model after being standardized and normalized. This preprocessing aims to reduce the impact of different lighting conditions on the signal. The normalization formula is as follows: , The input video frame, i.e. ;

[0035] Step 2: Spatial Overlapping 3D Convolutional Encoding: This part uses 3D convolutional kernels. With a stride of (1,4,4), the convolution output is: Therefore, it can be concluded that ,in Since the convolution kernel size is 7×7 and the stride is 4×4, there is a 3-pixel spatial overlap between adjacent convolution receptive fields. Therefore, adjacent feature blocks share some pixel information, satisfying the following: This spatial overlapping structure can avoid the spatial information fragmentation problem caused by traditional non-overlapping block methods, enhance the spatial continuity of facial skin areas, and help capture subtle color changes caused by blood flow changes, thereby improving the stability of subsequent feature representation.

[0036] Step 3: Sparse attention mechanism: Flatten the input features to obtain Linear mapping is performed on the flattened features to obtain The correlation matrix is ​​calculated for Q, K, and V to obtain And select the K key-value pairs (KV) that are most relevant to each query vector Q (i.e., Then local normalization. (like By performing Top-K filtering on the relevance matrix, each query feature is connected only to a few of the most relevant features, thereby reducing redundant computation and suppressing the interference of background regions on the attention distribution. A local normalization mechanism ensures that attention weights are redistributed within the candidate set, enhancing the response strength of key information regions and improving the targeting of feature representation.

[0037] Step 4: Gating modulation of attention scores: in standard scaled dot product attention After calculation, the features in the aggregated attention matrix A are... (in This involves introducing a learnable gating modulation module to nonlinearly modulate the attention weight matrix, followed by a sigmoid activation function. Dynamic recalibration of attention intensity Suppressing abnormally high response in the noise region, the final output By globally aggregating the attention matrix and generating gated weights, the intensity of the original attention distribution is modulated, suppressing abnormally high-response regions while enhancing the weight distribution in stable regions. This mechanism can reduce noise interference caused by illumination fluctuations, slight head movements, or partial occlusion, thereby improving the signal-to-noise ratio of signal extraction.

[0038] Step 5: Multi-scale temporal modeling: Perform hierarchical downsampling along the time dimension in the Transformer to obtain feature representations at different time scales. Then, scale alignment is achieved through upsampling to realize feature fusion. By employing different convolutional receptive fields to model the time dimension, the model can simultaneously capture short-term changes and long-term trends. Since physiological signals exhibit varying periodic characteristics across individuals, multi-scale temporal modeling helps enhance the model's adaptability to different heart rate zones.

[0039] Step 6: Time-Frequency Fusion Prediction: The input features received in this part are... The time-frequency fusion prediction head includes three parallel branches:

[0040] i) Temporal convolutional branch used to enhance short-term time dependency modeling capabilities The kernel size K is 3. These are learnable parameters;

[0041] ii) Frequency domain transform branch for enhancing the expression of periodic features within the heart rate band (0.7Hz–4Hz) This branch performs a discrete Fourier transform on the input features along the time dimension, i.e. After amplitude filtering or bandwidth constraint in the frequency domain, the result is obtained through inverse transformation. ;

[0042] iii) Construct multiple parallel temporal convolutions with different kernel sizes to capture multi-scale convolution branches that dynamically change at different time scales. ;

[0043] The features output from the three branches are concatenated and fused. Meanwhile, the input features are retained as residual paths. Then, learnable fusion coefficients are introduced. ,in For the Sigmoid function, These are learnable parameters; finally, the two are fused using learnable gating to control the fusion ratio. The final output signal Y is the predicted rPPG waveform. By performing frequency domain transformation on the time features and extracting amplitude spectrum information, the dominant frequency component in the signal can be enhanced, and non-periodic interference noise can be suppressed. Fusion of frequency domain features and time domain features helps to improve the stability and robustness of periodic physiological signals.

[0044] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A method for extracting physiological signals from facial videos based on gated attention modulation, characterized in that, The process includes the following steps: Step 1: Video Input and Preprocessing: Input a video sequence and normalize it; Step 2: Spatially Overlapping 3D Convolutional Encoding: Perform convolution operations on video frames using 3D convolution kernels, ensuring spatial overlap between adjacent receptive fields; Step 3: Sparse Attention Mechanism: Linearly map the input features to obtain query Q, key K, and value V, calculate the relevance matrix, and locally normalize the K most relevant key-value pairs for each query; Step 4: Gated Modulation of Attention Score: After scaling dot product attention calculation, globally pool the attention matrix, introduce a learnable gating modulation module to generate gating weights, and intensity modulate the original attention distribution; Step 5: Multi-Scale Temporal Modeling: Perform hierarchical downsampling along the time dimension to obtain feature representations at different time scales, and perform feature fusion after upsampling alignment; Step 6: Time-Frequency Fusion Prediction: Construct three parallel branches: temporal convolution, frequency domain transformation, and multi-scale convolution. Concatenate and fuse the features output from the three branches, and fuse them with the input features through learnable gating, finally outputting the predicted rPPG waveform.

2. The facial video physiological signal extraction method based on gated attention modulation according to claim 1, characterized in that, Step 1 includes the following steps: Input video sequence Where T is the number of frames, H and W are the dimensions of the input image, and C is the number of channels. The images are then input into the model after being standardized and normalized. The normalization formula is: , The input video frame, i.e. .

3. The facial video physiological signal extraction method based on gated attention modulation according to claim 1, characterized in that, Step 2 includes the following steps: Using 3D convolution kernel With a stride of (1,4,4), the convolution output is: Therefore, it can be concluded that ,in Since the convolution kernel size is 7×7 and the stride is 4×4, there is a 3-pixel spatial overlap between adjacent convolution receptive fields. Therefore, adjacent feature blocks share some pixel information, satisfying the following: .

4. The facial video physiological signal extraction method based on gated attention modulation according to claim 1, characterized in that, Step 3 includes the following steps: Flatten the input features to obtain Linear mapping is performed on the flattened features to obtain The correlation matrix is ​​calculated for Q, K, and V to obtain And select the K key-value pairs KV that are most relevant to each query vector Q, i.e. Then local normalization ,like By performing Top-K filtering on the relevance matrix, each query feature is connected only with a few of the most relevant features.

5. The facial video physiological signal extraction method based on gated attention modulation according to claim 1, characterized in that, Step 4 includes the following steps: Step 4: Gating modulation of attention scores: in standard scaled dot product attention After calculation, the features in the aggregated attention matrix A are... ,in A learnable gating modulation module is introduced to nonlinearly modulate the attention weight matrix, which is then activated by the Sigmoid activation function. Dynamic recalibration of attention intensity Suppressing abnormally high response in the noise region, the final output By globally aggregating the attention matrix and generating gating weights, the intensity of the original attention distribution is modulated, thereby suppressing abnormally high response regions and enhancing the weight distribution in stable regions.

6. The facial video physiological signal extraction method based on gated attention modulation according to claim 1, characterized in that, Step 5 includes the following steps: In the Transformer, hierarchical downsampling is performed along the time dimension to obtain feature representations at different time scales. Then, scale alignment is achieved through upsampling to realize feature fusion. Different convolutional receptive fields are used to model the time dimension, enabling the model to capture both short-term changes and long-term trends simultaneously.

7. The facial video physiological signal extraction method based on gated attention modulation according to claim 1, characterized in that, Step 6 includes the following steps: The received input features are The time-frequency fusion prediction head includes three parallel branches: i) Temporal convolutional branch used to enhance short-term time dependency modeling capabilities The kernel size K is 3. These are learnable parameters; ii) Frequency domain transform branch used to enhance the expression of periodic features within the heart rate band This branch performs a discrete Fourier transform on the input features along the time dimension, i.e. After amplitude filtering or bandwidth constraint in the frequency domain, the result is obtained through inverse transformation. ; iii) Construct multiple parallel temporal convolutions with different kernel sizes to capture multi-scale convolution branches that dynamically change at different time scales. ; The features output from the three branches are concatenated and fused. Meanwhile, the input features are retained as residual paths. Then, learnable fusion coefficients are introduced. ,in For the Sigmoid function, These are learnable parameters; finally, the two are fused using learnable gating to control the fusion ratio. The final output signal Y is the predicted rPPG waveform.

Images