Remote heart rate detection method based on Transform and CNN parallel fusion

Abstract

The invention relates to a remote heart rate detection method based on Transform and CNN parallel fusion, and the method comprises the following steps: carrying out the feature initial extraction of a Transform branch and a CNN branch, and respectively outputting a sum; the data are sent to a CTBlock formed by connecting a CNN branch and a Transform branch in parallel, and feature interaction fusion is carried out through a feature coupling unit FCU; and the rPPG signal prediction module Predictor performs up-sampling on two outputs of the CTBlock in the time dimension, averages the two outputs in the space dimension, and then respectively projects the two outputs into a one-dimensional signal sum, and the average value of the two outputs is the finally predicted rPPG signal. According to the method, global context information is considered, local features are also considered, global and local features are modeled cooperatively, multi-level spatial-temporal features are fused, the heart rate estimation accuracy and robustness are remarkably improved, and the method is suitable for complex real scenes.

Description

Technical Field

[0001] This invention relates to the field of heart rate detection technology based on facial video, and more particularly to a remote heart rate detection method based on the parallel fusion of Transformer and CNN. Background Technology

[0002] With the development of technologies such as telemedicine, emotion recognition, and human-computer interaction, physiological signal detection based on facial videos has gradually become a research hotspot. Among them, the method of extracting heart rate signals from facial videos using camera equipment (remote photoplethysmography, or rPPG) is widely used in health monitoring, exercise assessment, and smart terminals due to its advantages such as being non-contact, low-cost, and convenient.

[0003] rPPG signal is a weakly periodic signal reflecting changes in blood volume, which can be obtained by analyzing subtle color fluctuations in the facial region across consecutive image frames. Traditional methods typically rely on temporal signal changes in specific color channels (such as RGB channels) to estimate heart rate through filtering, signal decomposition, or frequency domain analysis. However, due to complex factors such as lighting variations, head movements, skin color differences, and camera noise in facial videos, traditional algorithms often struggle to accurately extract rPPG signals when the signal-to-noise ratio is low.

[0004] In recent years, with the advancements in deep learning and temporal modeling techniques, researchers have proposed heart rate detection methods based on models such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). These methods, by extracting features and modeling time within continuous image frame sequences, can learn to some extent the subtle color changes in facial regions related to blood flow. However, while CNNs primarily focus on capturing local features and possess excellent feature representation capabilities in image processing, they are prone to temporal information loss and confusion in long-term spatiotemporal video, making it difficult to capture periodic changes over time. Therefore, a heart rate detection method that can capture long-term temporal dependencies while maintaining high sensitivity to short-term inter-frame differences to improve accuracy and robustness has become a pressing technical problem in this field. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a remote heart rate detection method based on the parallel fusion of Transformer and CNN.

[0006] This invention is achieved through the following technical solution: A remote heart rate detection method based on the parallel fusion of Transformer and CNN includes the following steps: S1. Perform initial feature extraction on the Transformer branch and the CNN branch, and output the results respectively. and ; S2. The two-way output and They are fed into a CTBlock, which consists of parallel CNN and Transformer branches, and feature interaction and fusion are performed through the Feature Coupling Unit (FCU). The S3.rPPG signal prediction module Predictor upsamples the two outputs of CTBlock in the time dimension, averages them in the spatial dimension, and then projects them into one-dimensional signals respectively. and The average of the two values ​​is the final predicted rPPG signal.

[0007] According to the above technical solution, preferably, in step S1, for the Transformer branch, Fusion_Stem in RhythmFormer is used to initially extract local spatiotemporal features and fuse the time information of the preceding and following frames of the video; for the CNN branch, a combination of 3×3 2D convolution and max pooling layers is used for initial feature extraction.

[0008] According to the above technical solution, preferably, step S1, for the Transformer branch, includes the following steps: Given RGB video input Time-shift the video to obtain , , , , ; The differences between consecutive frames are calculated in chronological order to obtain... , , , ; The differences between the consecutive frames and the original frame through Two independent instances and Perform main feature extraction and output the... .

[0009] According to the above technical solution, preferably, in step S1, for a CNN branch, given the input... Output , , in, This represents a 3×3 2D convolution. This represents a 5×5 2D max pooling operation.

[0010] According to the above technical solution, preferably, in step S2, the Transformer branch includes multi-head self-attention (MHSA) and a feedforward network. After MHSA, global information in the input sequence is captured through the feedforward network and normalized LN.

[0011] According to the above technical solution, preferably, in step S2, the CNN branch is formed by cascading two identical time-channel learning (TCL) units, and progressive feature extraction is performed using time-channel shifting module (TSMP), 2D convolution, self-attention, batch normalization (BN), and pooling operations.

[0012] According to the above technical solution, preferably, in step S2, the feature coupling unit FCU uses FCUDown and FCUUp to perform feature interaction fusion of the Transformer branch and the CNN branch.

[0013] According to the above technical solution, preferably, in step S2, during the FCUDown operation, the channel dimension is aligned using a 1×1×1 convolutional layer, and the spatial dimension is aligned using an average pooling layer for downsampling. , in, This represents a 1×1×1 3D convolution. This represents a 4×16×16 3D average pooling operation. Representation layer normalization, This represents the activation function.

[0014] According to the above technical solution, preferably, in step S2, in the FCUUp operation, a 1×1×1 3D convolutional layer is used to align the channel dimension, and upsampling is used to align the spatial dimension. , in, This represents a 1×1×1 3D convolution. Indicates batch normalization. This represents the activation function. This indicates an upsampling operation.

[0015] The beneficial effects of this invention are: This invention provides a dual-branch parallel network structure consisting of an attention-based CNN and a Transformer network (CTNet) for heart rate detection based on facial videos. The CNN branch is used to extract local spatial features and subtle color changes in the facial image, while the Transformer branch is used to capture long-range dependencies and global contextual information in the temporal dimension. It considers both global contextual information and local features, and the global and local features are modeled together to improve the accuracy of heart rate estimation. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the CTNet network structure composed of CNN and Transformer disclosed in this invention.

[0017] Figure 2 This is a schematic diagram of the Fusion_Stem structure disclosed in step S1 of the present invention.

[0018] Figure 3 This is a schematic diagram of the CTBlock structure disclosed in step S2 of the present invention.

[0019] Figure 4 This is a schematic diagram of the multi-head self-attention MHSA structure disclosed in step S2 of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and preferred embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0021] As shown in the figure, the present invention includes the following steps: S1. Perform initial feature extraction on the Transformer branch and the CNN branch, and output the results respectively. and .

[0022] For the Transformer branch, the Fusion_Stem from RhythmFormer is used to initially extract local spatiotemporal features, fusing temporal information from consecutive video frames. Specifically, given an RGB video input... Time-shift the video to obtain , , , , Then, the differences between consecutive frames are calculated in chronological order to obtain... , , , The differences between the consecutive frames and the original frame through Two independent instances and Perform main feature extraction. and It consists of 5×5 2D convolutional layers, cascaded with batch normalization (BN), ReLU activation function, and max pooling layers. Given input , and The operation rules are as follows:

[0023] in, express The output, The target feature dimension is defined in this paper, and T, W, and H represent the sequence length, height, and width of the original video, respectively. This represents a 5×5 2D convolution. Indicates batch normalization. For activation function, This represents a 3×3 2D max pooling operation.

[0024] The operation rules for Fusion_Stem are as follows:

[0025]

[0026]

[0027]

[0028] in, This indicates the output of the Fusion_Stem module. Indicates a connection. The target feature dimension is defined in this paper, and T, W, and H represent the sequence length, height, and width of the original video, respectively.

[0029] Then, the output embeddings of Fusion_Stem are divided into non-overlapping sequences through a linear projection layer. This spatially aggregates adjacent semantic information while reducing the computational cost of the converter. Specifically, given the input of the linear projection layer... , will output As input to the Transformer branch, the calculation rules are as follows:

[0030]

[0031] in, This represents a 3×3×3 3D convolution. This represents a 4×4×4 3D convolution. Indicates batch normalization. For activation function, This represents a 1×2×2 3D max pooling operation. This represents the output of the linear projection, where Represents the feature dimension of the sequence. This represents the length of the sequence.

[0032] Furthermore, for the CNN branch, a combination of 3×3 2D convolutions and max pooling layers is used for initial feature extraction. Specifically, given the input... Output The operation rules are as follows:

[0033] in, This represents a 3×3 2D convolution. This represents a 5×5 2D max pooling operation.

[0034] S2. The two-way output and They are fed into a CTBlock, which consists of parallel CNN and Transformer branches, and feature interaction and fusion are performed through the Feature Coupling Unit (FCU).

[0035] CTBlock is a core module that uses the FCU structure to achieve feature interaction and fusion based on the Transformer and CNN branches. In this example, the Feature Coupling Unit (FCU) is used to solve the problem of feature dimension misalignment between the CNN and Transformer branches, which enables the aggregation of local features and global context information.

[0036] The Transformer branch includes Multi-Head Self-Attention (MHSA) and Feedforward networks. For MHSA, given a sequence input... , where d represents the dimension of the feature map and n represents the sequence length. The input is fed into a Transformer encoder, where fine-grained rPPG features are extracted from different time scales via a multi-head self-attention mechanism. Specifically, in the self-attention mechanism, the relationships between sequences are modeled through the similarity between the query (Q)-key (K) pairs of the projection, thus generating an attention score. In this example, temporal difference convolution (TDC) is used instead of pointwise linear projection for both query (Q) and key (K) projections, which can capture fine-grained local temporal difference features for describing subtle color variations.

[0037]

[0038] in This represents the sampling region of a 3D convolution. This indicates the adjacent regions used in 3D convolution. This indicates the current position where convolution calculation is in progress. This represents the relative position offset within the convolution kernel. This represents the learnable weight parameters of the convolution kernel at position . This represents the contribution weights for controlling the time-domain difference information. Then Q and K are projected as:

[0039]

[0040] For the projection of the value (V), a point-by-point linear projection without BN is used. Then... Flatten it into a sequence and divide it into h heads (each head) For the i-th head (i≤h), self-attention (SA) can be expressed as:

[0041] in Controlling sparsity. The output of a multi-head self-attention (MHSA) algorithm is a concatenation of all self-attention heads (SAs) and then applied to a linear projection. :

[0042] Following the multi-head self-attention MHSA, a feedforward network will be used with layer normalization LN applied, and the feedforward network will be consistent with the original Transformer.

[0043] The Transformer branch captures global information from the input sequence through the MHSA mechanism. It effectively captures long-term dependencies and global semantic information from the input sequence, which is advantageous for heart rate detection tasks. Since rPPG signals are time-series signals, capturing global information across the entire time series is necessary for accurate predictions. The Transformer branch can adaptively learn the importance of temporal context, enabling it to better handle different time steps arising from long-term dependencies.

[0044] The CNN branch is composed of two sets of identical time-channel learning (TCL) units cascaded together. It uses time-channel shifting module (TSMP), 2D convolution, self-attention, batch normalization (BN), and pooling operations for stepwise feature extraction, aiming to enhance the model's ability to sequentially model temporal context information.

[0045] Temporal Shift Module (TSM) is a method for shifting channels along the time dimension to capture the temporal context of video. This operation has two advantages: 1) it adds no parameters or computational complexity; 2) it can be inserted into any convolutional or recurrent neural network. However, TSM's ability to perceive temporal context is limited. This example improves upon TSM by extending the temporal depth of channel swapping, allowing for longer-term perception of temporal information, and introduces the more efficient module TSMP.

[0046] TSMP divides a channel into multiple parts, one of which remains unchanged, while the remaining parts are further divided into blocks, each containing a block of size [size missing]. The channel. For any frame The output of the channel shift is The shift operation is performed according to the following rules.

[0047] (a) For the first part (capturing future information), perform the following channel conversion:

[0048] (b) For the second part (capturing past information), perform the following channel conversion:

[0049] (c) For the third part (capturing information from a more distant future time), perform the following channel conversion:

[0050] (d) For Part 4 (capturing information from further back), perform the following channel conversion:

[0051] (e) The remaining channel information remains unchanged:

[0052] Self-attention calculates the attention mask through tensor summation followed by element-wise division, and then normalizes it to eliminate outliers. Given input... The calculation rules for attention weight output are as follows:

[0053] The core TCL unit of the CNN branch consists of modules such as TSMP, convolution, and self-attention. Given an input... The operation rules are as follows:

[0054]

[0055]

[0056]

[0057] in, Indicates the output of the TCL unit. This indicates a time channel shift operation. This represents the activation function. This indicates that the self-attention mask is obtained. This represents a 2×2 2D average pooling operation.

[0058] Next, a Feature Coupling Unit (FCU) is used to address the misalignment between feature maps in the CNN and Transformer branches during feature aggregation. Given the input to the CNN branch... Input to the Transformer branch The Transformer branch and CNN branch are fused by using FCUDown and FCUUp.

[0059] In the FCUDown operation, a 1×1×1 3D convolutional layer is used to align the channel dimensions, and an average pooling layer is used to align the spatial dimensions for downsampling. The operation rules are as follows: , in, This represents a 1×1×1 3D convolution. This represents a 4×16×16 3D average pooling operation. Representation layer normalization, This represents the activation function.

[0060] In the FCUUp operation, a 1×1×1 3D convolutional layer is used to align the channel dimension, and upsampling is used to align the spatial dimension. The operation rules are as follows: , in, This represents a 1×1×1 3D convolution. Indicates batch normalization. This represents the activation function. This indicates an upsampling operation.

[0061] The S3.rPPG signal prediction module Predictor upsamples the two outputs of CTBlock in the time dimension, averages them in the spatial dimension, and then projects them into one-dimensional signals respectively. and The average of the two values ​​is the final predicted rPPG signal.

[0062] In addition to designing the network structure, an appropriate loss function should be formulated to guide the network training. rPPG signals are typically quasi-periodic waves subjected to various nonlinear distortions and exhibit a characteristic of roughly constant variation over any time interval. Meanwhile, facial videos may contain outliers due to factors such as ambient lighting, facial expressions, and occlusion. To accurately capture the characteristics of quasi-periodic signals and eliminate the influence of outliers on prediction results, this paper uses the negative Pearson correlation coefficient to minimize the error between predicting rPPG signals and ground truth signals. It is more robust to outliers than mean squared error (SSE) and can more accurately reflect the characteristics of rPPG signals. The Pearson loss function can be expressed as:

[0063] in Indicates the length of the rPPG signal. and These represent the predicted and actual rPPG signals, respectively.

[0064] In summary, this application provides a dual-branch parallel network structure, CTNet, composed of an attention-based CNN and a Transformer, for heart rate detection based on facial videos. The CNN branch is used to extract local spatial features and subtle color changes in facial images, while the Transformer branch is used to capture long-range dependencies and global contextual information in the temporal dimension. A Feature Coupling Unit (FCU) is then used to address the misalignment of feature dimensions between the CNN and Transformer branches. This enables the effective aggregation of local features and global contextual information, co-modeling of global and local features, and fusion of multi-level spatiotemporal features, significantly improving the accuracy and robustness of heart rate estimation and making it suitable for complex real-world scenarios.

[0065] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A remote heart rate detection method based on parallel fusion of Transformer and CNN, characterized in that, Includes the following steps: S1. The Transformer branch and the CNN branch are subjected to feature initial extraction, and outputs are respectively outputted and ; S2. The two-way output of the and is sent to the CTBlock composed of CNN branches and Transformer branches in parallel, and the feature interaction and fusion are performed through the feature coupling unit FCU. S3. The rPPG signal prediction module Predictor upsamples both outputs of the CTBlock in the time dimension and averages them in the spatial dimension, then projects them into one-dimensional signals, respectively and The average of both is the final predicted rPPG signal.

2. The remote heart rate detection method based on parallel fusion of Transformer and CNN according to claim 1, characterized in that, In step S1, for the Transformer branch, Fusion_Stem in RhythmFormer is used to initially extract local spatiotemporal features and fuse the temporal information of the preceding and following video frames. For the CNN branch, a combination of 3×3 2D convolutions and max pooling layers is used for initial feature extraction.

3. The remote heart rate detection method based on parallel fusion of Transformer and CNN according to claim 2, characterized in that, In step S1, for the Transformer branch, the following steps are included: Given RGB video input Time-shift the video to obtain , , , , ; The differences between consecutive frames are calculated in chronological order to obtain... , , , ; The differences between the consecutive frames and the original frame through Two independent instances and Perform main feature extraction and output the... .

4. The remote heart rate detection method based on the parallel fusion of Transformer and CNN as described in claim 2 or 3, characterized in that, In step S1, for the CNN branch, given the input Output , ， in, This represents a 3×3 2D convolution. This represents a 5×5 2D max pooling operation.

5. The remote heart rate detection method based on the parallel fusion of Transformer and CNN as described in claim 2, characterized in that, In step S2, the Transformer branch includes a multi-head self-attention network (MHSA) and a feedforward network. Following Multi-Head Self-Attention (MHSA), global information in the input sequence will be captured through a feedforward network and by applying Layer Normalized Linear Network (LN).

6. The remote heart rate detection method based on the parallel fusion of Transformer and CNN as described in claim 5, characterized in that, In step S2, the CNN branch is formed by cascading two identical time-channel learning (TCL) units. Stepwise feature extraction is performed using the temporal channel shifting module (TSMP), 2D convolution, self-attention, batch normalization (BN), and pooling operations.

7. The remote heart rate detection method based on the parallel fusion of Transformer and CNN as described in claim 6, characterized in that, In step S2, the feature coupling unit FCU uses FCUDown and FCUUp to perform feature interaction fusion of the Transformer branch and the CNN branch.

8. The remote heart rate detection method based on the parallel fusion of Transformer and CNN as described in claim 7, characterized in that, In step S2, during the FCUDown operation, the channel dimension is aligned using a 1×1×1 convolutional layer, and the spatial dimension is aligned using an average pooling layer for downsampling. ， in, This represents a 1×1×1 3D convolution. This represents a 4×16×16 3D average pooling operation. Representation layer normalization, This represents the activation function.

9. A remote heart rate detection method based on parallel fusion of Transformer and CNN as described in claim 7 or 8, characterized in that, In step S2, during the FCUUp operation, a 1×1×1 3D convolutional layer is used to align the channel dimensions, and upsampling is used to align the spatial dimensions. ， in, This represents a 1×1×1 3D convolution. Indicates batch normalization. This represents the activation function. This indicates an upsampling operation.

Images