A non-contact heart rate signal extraction method based on multi-scale space-time graph and polarized self-attention

Abstract

This invention belongs to the field of pattern recognition technology, specifically a non-contact heart rate signal extraction method based on multi-scale spatiotemporal maps and polarization self-attention. It transforms rigid ROIs into dynamic, flexible models, constructing elastic multi-region partitions. YUV color space conversion and area-weighted multi-scale fusion improve the quality of the spatiotemporal map. Orthogonal projection using the skin color space POS algorithm suppresses illumination interference and enhances the pulse signal. Adversarial learning is employed between a heart rate encoder and a parallel noise encoder to focus on effective features. The signal is reconstructed via a decoder by fusing dual-branch features, achieving robust and high-precision heart rate estimation, effectively improving generalization ability and detection stability in complex scenarios. This invention exhibits stronger robustness, higher accuracy, and better generalization in complex lighting and dynamic interference scenarios, stably outputting clean pulse waveforms and accurate heart rates, providing efficient and reliable technical and data support for applications such as telemedicine, emotion recognition, and intelligent lie detection.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition technology, specifically a non-contact heart rate signal extraction method based on multi-scale spatiotemporal graphs and polarization self-attention. Background Technology

[0002] To improve the convenience and accuracy of intelligent monitoring technologies, non-contact heart rate signal extraction methods have been widely applied in fields such as telemedicine, smart homes, and sports and health. Non-contact heart rate signal extraction technology primarily analyzes facial or skin areas in videos, utilizing computer vision and deep learning techniques to recover heart rate signals from subtle color changes or motion information, thus achieving contactless physiological parameter monitoring. Traditional contact-based heart rate detection requires wearable devices (such as photoelectric pulse wave sensors), which are not only uncomfortable but also difficult to meet the needs of long-term continuous monitoring. Therefore, researching efficient and robust non-contact heart rate signal extraction methods to achieve accurate capture and analysis of subtle physiological signals has significant application value.

[0003] Non-contact heart rate signal extraction methods can be mainly divided into methods based on traditional signal processing and methods based on deep learning. Methods based on traditional signal processing typically extract color changes (e.g., RGB signal separation, independent component analysis) or motion information (e.g., Euler video magnification) from video sequences and use algorithms such as Fourier transform or peak detection to calculate heart rate values. However, these methods are easily affected by changes in ambient lighting, subject motion interference, and video quality, resulting in insufficient stability in signal extraction.

[0004] In recent years, deep learning-based heart rate signal extraction methods have attracted widespread attention due to their powerful spatiotemporal feature modeling capabilities. Deep neural networks can directly learn heart rate-related spatiotemporal representations from raw video data, avoiding the limitations of manually designed signal processing procedures. However, the generalization ability of existing methods still faces challenges when faced with complex scenes (such as low light, motion artifacts, and multiple noise interferences) or individual differences (such as skin color and age). Methods based on multi-scale spatiotemporal maps and polarization self-attention further improve the robustness and accuracy of heart rate estimation by modeling the spatiotemporal dependencies between signals and combining the polarization attention mechanism to focus on key features. Summary of the Invention

[0005] This invention proposes a non-contact heart rate signal extraction method based on multi-scale spatio-temporal maps and polarized self-attention. This method integrates the traditional POS (Plane-Orthogonal-to-Skin) algorithm with a multi-scale spatio-temporal map (MSTmap) to construct a new input representation method, POS-MSTmap, significantly improving the model's robustness to interference in complex environments. By introducing a polarized self-attention (PSA) module, effective decoupling of physiological signals and noise features is achieved, further optimizing the accuracy of heart rate estimation and rPPG signal regression.

[0006] The technical solution adopted in this invention is: a non-contact heart rate signal extraction method based on multi-scale spatiotemporal maps and polarization self-attention. S1 transforms rigid region partitioning into dynamic, adaptive, flexible modeling, designing a more flexible multi-region ROI partitioning strategy.

[0007] S2 converts the original image to the YUV color space and introduces an area-weighted multi-scale fusion strategy to enhance the contribution of high signal-to-noise ratio regions and further improve the quality of the spatiotemporal map.

[0008] S3, based on the POS algorithm of skin color space modeling, uses the skin color reflection characteristics to orthogonally project the signal in order to remove steady-state illumination interference and enhance the pulse signal.

[0009] S4, construct a heart rate encoder to extract deep features highly correlated with heart rate signals from multi-scale spatiotemporal maps.

[0010] S5 designs a parallel branch structure for the noise encoder to model background interference components in the input data that are unrelated to heart rate. Through explicit adversarial mechanisms in the feature space, it assists the main encoder in focusing on heart rate-related feature regions, thereby improving the accuracy and generalization ability of the final prediction.

[0011] In step S6, after the main encoder and noise encoder complete feature extraction, the network fuses their output features and further enhances the learning of image-level reconstruction and heart rate signals through the decoder module. This promotes the alignment of main and auxiliary features and provides an end-to-end positive constraint mechanism for the network.

[0012] S7 uses the Adam optimizer to train the network model. Training ends when the model stabilizes; otherwise, it goes back to S1 and repeats the above steps. A highly accurate and stable full-process enhanced heart rate estimation model is obtained through training.

[0013] Furthermore, in step S1, a more flexible multi-region ROI partitioning strategy is designed, specifically as follows: Based on the dlib 68-point facial landmark model, facial physiologically sensitive sub-regions are dynamically constructed. All ROIs are generated as binary masks using polygon2mask, and the number of effective pixels is counted for subsequent signal weighting.

[0014] Furthermore, in step S2, a multi-scale spatiotemporal map is constructed based on the ROI regions divided in step S1. Specifically, The original image is converted from the RGB color space to the YUV color space, where the Y channel represents luminance information and the U and V channels represent chromatic aberration information. This conversion allows for better separation of the image's color information, facilitating region-weighted fusion. The specific conversion formula is as follows:

[0015] Based on the facial region division, the facial region is divided into multiple physiologically sensitive sub-regions. The RGB and YUV signals of each region are weighted and fused according to their area proportions to enhance the contribution of high signal-to-noise ratio regions to the spatiotemporal map. This weighting strategy effectively suppresses noise caused by small-scale occlusion interference. The signals are then mapped using max-min normalization to form the initial spatiotemporal map.

[0016] The generated spatiotemporal map is a six-channel signal map. The horizontal axis represents the time dimension, the vertical axis is the spatial average intensity value of each ROI region, and the pixel value reflects the color change of that region at that time point. Each spatiotemporal map contains six channels, namely the weighted signal of the three RGB channels and the information of the three channels in the YUV space.

[0017] Furthermore, based on the results in step S3, orthogonal projection and filtering enhancement are performed, specifically as follows: Color signals are projected into a subspace orthogonal to the direction of skin reflection to strip away the illumination-dominated component while preserving subtle pulse-driven color variations. Considering the differences in POS performance across different color spaces, only the three-channel spatiotemporal map of the YUV color space is projected, while the RGB channels retain their original information without processing. This approach enhances the robustness of physiological signals while maintaining the integrity of visual features.

[0018] Suppose the original signal sequence of the YUV channel is a three-dimensional vector. These correspond to the Y luminance, U, and V chrominance components, respectively. Within each frame time window, the signal is first mean-normalized, and then a projection matrix orthogonal to the skin color direction is constructed. P The normalized signal is mapped to this subspace:

[0019] in, The normalized YUV signal matrix, PThese are the pre-defined orthogonal basis vectors. S This is the signal after POS projection processing.

[0020] POS uses two orthogonal projection results and Linear combination extraction of heart rate and pulse signals h :

[0021] in, and These are two orthogonal components. and The standard deviation is used to adjust the weights of the two channels, thereby improving the dynamic adaptability of signal extraction.

[0022] However, the effectiveness of the POS algorithm may still be limited by its linear illumination modeling assumption under strong interference conditions. To further improve robustness, a 5th-order Butterworth bandpass filter is introduced after POS projection to suppress high-frequency noise caused by device jitter and low-frequency drift caused by body temperature fluctuations. The filtered result is then renormalized and stitched together according to the facial ROI structure, and finally fused with the RGB channels to form a six-channel enhanced multi-scale spatiotemporal map—POS-MSTmap.

[0023] Based on steps S1, S2, and S3, multi-scale partitioning, POS projection, and bandpass filtering are combined to construct a multi-scale spatiotemporal graph with high signal-to-noise ratio and strong robustness in dynamic scenes, providing a more reliable input representation for subsequent effective learning of deep networks.

[0024] Furthermore, in step S4, a heart rate encoding module is designed, specifically as follows: The heart rate encoder mainly consists of two components: a convolutional feature extraction module based on an improved ResNet18 architecture, and a polarization self-attention mechanism embedded within each residual block. Compared to traditional ResNet18, this invention incorporates several customized optimizations for non-contact heart rate estimation tasks. First, the input layer supports processing 6-channel POS-MSTmaps to meet the needs of joint modeling of spatiotemporal features. Second, a polarization self-attention module is introduced into each residual unit, effectively enhancing the model's ability to focus on key spatial locations and channel features. Third, the normalization method is changed from batch normalization to instance normalization, enhancing the model's adaptability to differences in data distribution across individuals and scenarios. Furthermore, the output is designed with a multi-task structure, capable of both heart rate value regression and reconstruction of heart rate-related rPPG signals.

[0025] Specifically, the convolutional feature extraction part is based on the ResNet18 structure with adjustments. First, a 7×7 convolutional kernel is used to perform preliminary downsampling on the input feature map to extract low-level texture and edge features. Then, the feature map is processed in depth through multiple improved residual blocks. Each residual block consists of two 3×3 convolutional layers, equipped with InstanceNorm and ReLU activation functions. At the same time, a skip connection mechanism is used to alleviate the gradient vanishing problem and further improve training stability.

[0026] The polarization self-attention mechanism consists of a channel attention branch (Ach) and a spatial attention branch (Asp), which model the dependencies between channels and between spatial regions, respectively. The polarization self-attention module preserves the high-resolution structure in one dimension through "polarization filtering" while compressing and filtering information in another dimension, achieving efficient and refined attention modeling.

[0027] In the channel branch, the input feature map is first compressed in spatial dimension through a 1×1 convolution while preserving high-resolution features across channels. Then, global matrix dot products are used to calculate channel attention weights, capturing semantic relationships across channels. This process significantly suppresses noise interference with channel information while retaining effective channel information. Let the input feature map be... ,in C For the number of channels, H and W The spatial dimension is represented by the feature map obtained through 1×1 convolution dimensionality reduction. for:

[0028] Calculate channel attention weights :

[0029] Long-range dependencies across channels are captured through matrix dot products, and channel attention weights are generated.

[0030] In the spatial branch, the input feature map is first compressed in terms of channel dimension using Global Average Pooling (GAP) to obtain a feature map with 1 channel. :

[0031] Spatial attention weights are generated by weighting the spatial dimensions using the spatial softmax operation. :

[0032] This operation dynamically adjusts the weights of various spatial locations in the feature map, enhancing important parts of the signal and suppressing noise components, especially in non-facial areas or areas affected by occlusion.

[0033] Based on the two branches mentioned above, the polarization attention module introduces a dynamic enhancement mechanism combining Softmax and Sigmoid to enhance signal diversity while suppressing noise and reducing interference from irrelevant components. Specifically, the channel branch enhances multimodal distribution modeling through Softmax, and simultaneously strengthens the target frequency band through the Sigmoid function in the noise band, effectively filtering out high-frequency noise and reducing its impact. The spatial branch, on the other hand, expands the dynamic range through the Sigmoid function to adapt to the distribution of the binarized mask, thereby effectively and dynamically reducing the weight of non-facial regions.

[0034] Furthermore, the two branches are fused using both parallel and serial layouts. The parallel layout directly adds the two branches, preserving their independence and is suitable for static scenarios; the serial layout optimizes features step by step through cascaded filtering, and is suitable for dynamic interference scenarios.

[0035] Furthermore, in step S5, a parallel noise encoder branch is constructed to model background interference components in the input data that are unrelated to heart rate, further enhancing the model's ability to separate heart rate signals. The core of this branch lies in assisting the main encoder to focus on heart rate-related feature regions through explicit adversarial interactions in the feature space, thereby improving the accuracy and generalization ability of the final prediction. Specifically, The noise encoder employs the same backbone structure as the main encoder, built upon the first two layers of ResNet18. During the input stage, convolution and downsampling operations are used for preliminary processing of the input image. Subsequently, the feature maps are sequentially processed through residual blocks to gradually extract intermediate spatial semantic features. This fully leverages the residual representation advantages of ResNet, reducing computational burden and making it suitable for rapidly modeling the distribution patterns of noise features. Finally, the intermediate features output from the noise encoder and the deep features output from the main encoder are fused and used as one of the inputs to the unified reconstruction module. In subsequent reconstruction modules, the fused features will be used for signal recovery and auxiliary supervision.

[0036] Unlike the main branch, the noise encoder does not introduce a polarization attention module, but instead maintains the traditional convolutional path, preserving the ability to model background and low-frequency interference. This design forms an explicit feature contrast mechanism, which helps the network to more clearly distinguish between heart rate signals and environmental noise, improving the robustness of predictions in weak signal scenarios.

[0037] The noise branch, as an auxiliary modeling strategy, effectively improves the purity of the model's representation of heart rate signals and its anti-interference performance by maintaining the simplicity of the network structure through structural parallel alignment and semantic contrastive learning.

[0038] Furthermore, in step S6, after the main encoder and noise encoder complete feature extraction, the network fuses the output features of both and enhances the learning of image-level reconstruction and heart rate signals through the same decoder module. This not only promotes the alignment of main and auxiliary features but also provides an end-to-end positive constraint mechanism for the network.

[0039] The fusion feature first inputs a bottleneck coding structure composed of multiple residual blocks to enhance its expressiveness and task relevance. These residual block structures are similar to those in the encoder but do not include polarization attention modules, aiming to extract more abstract and compact feature representations. Subsequently, the decoder progressively upsamples through a series of deconvolutional modules, restoring the original image resolution layer by layer. Finally, the output stage employs different convolutional head designs depending on the task type. In the multi-task training framework, the reconstruction branch generates an image with a structure consistent with the input image, used to help constrain the spatial structure consistency of the heart rate features. Simultaneously, another branch outputs a one-dimensional heart rate signal, extracting the time series through AdaptiveAvgPool2d and 1×1 convolution and then compressing it.

[0040] By introducing a decoder module, the model not only has the regression capability for heart rate estimation, but can also reconstruct the dynamic changes related to the signal in the image domain, achieving stronger interpretability and cross-modal feature fusion capability, and further improving the model's adaptability and generalization performance in real-world scenarios.

[0041] Furthermore, in step S7, the Adam optimizer is designed to obtain the optimal heart rate signal extraction model.

[0042] The beneficial effects of this invention are as follows: Addressing the problems of poor robustness and signal loss in traditional non-contact heart rate signal extraction methods under motion interference and complex lighting conditions, this invention proposes a fully enhanced heart rate estimation method from feature modeling to decoupling. This invention models the dynamic features of the facial region using multi-scale spatiotemporal maps and integrates a POS mechanism to improve the model's resistance to local motion artifacts at the input end. In the feature extraction part, a polarization self-attention mechanism is embedded in the ResNet18 residual network to achieve spatiotemporal focusing on key physiological signals and noise decoupling. Instance normalization is introduced to replace batch normalization, enhancing the model's adaptability across different scenarios. The output end adopts a multi-task structure design to predict heart rate values ​​and reconstruct remote photoplethysmography (PPG) pulse wave sequence signals, achieving end-to-end feature perception and signal generation. This invention exhibits stronger robustness in complex lighting and motion environments, providing efficient and reliable theoretical support and data support for applications such as automated telemedicine, emotion recognition, and intelligent lie detection. Attached Figure Description

[0043] Figure 1 This is a diagram of the heart rate extraction network structure. Figure 2Comparison chart showing improvements to the heart rate encoder; Figure 3 A framework diagram of the polarization self-attention mechanism; Figure 4 To predict the heart rate versus the true heart rate on the test set; Figure 5 A waveform comparison of the predicted rPPG signal and the actual ECG signal in different samples. Detailed Implementation

[0044] The following is in conjunction with the appendix Figure 1-5 The principles and features of the present invention are described, and the examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0045] This invention provides a non-contact heart rate signal extraction method based on multi-scale spatiotemporal maps and polarization self-attention, specifically including the following steps: S1 transforms rigid region partitioning into dynamic, adaptive, flexible modeling, designing a more flexible multi-region ROI partitioning strategy.

[0046] S2 converts the original image into the YUV (color representation method) color space and introduces an area-weighted multi-scale fusion strategy to enhance the contribution of high signal-to-noise ratio regions and further improve the quality of the spatiotemporal map.

[0047] S3, based on the POS algorithm of skin color space modeling, uses the skin color reflection characteristics to orthogonally project the signal in order to remove steady-state illumination interference and enhance the pulse signal.

[0048] S4, construct a heart rate encoder to extract deep features highly correlated with heart rate signals from multi-scale spatiotemporal maps.

[0049] S5 designs a parallel branch structure for the noise encoder to model background interference components in the input data that are unrelated to heart rate. Through explicit adversarial mechanisms in the feature space, it assists the main encoder in focusing on heart rate-related feature regions, thereby improving the accuracy and generalization ability of the final prediction.

[0050] In step S6, after the main encoder and noise encoder complete feature extraction, the network fuses their output features and further enhances the learning of image-level reconstruction and heart rate signals through the decoder module. This promotes the alignment of main and auxiliary features and provides an end-to-end positive constraint mechanism for the network.

[0051] S7 uses the Adam optimizer to train the network model. Training ends when the model stabilizes; otherwise, it goes back to S1 and repeats the above steps. A highly accurate and stable full-process enhanced heart rate estimation model is obtained through training.

[0052] Specifically, step S1 involves designing a more flexible multi-region ROI partitioning strategy, namely, Based on the dlib 68-point facial landmark model, facial physiologically sensitive sub-regions are dynamically constructed. All ROIs are generated as binary masks using polygon2mask, and the number of effective pixels is counted for subsequent signal weighting.

[0053] Specifically, in step S2, a multi-scale spatiotemporal map is constructed based on the ROI regions divided in step S1. The original image is converted from the RGB color space to the YUV color space (following the ITU-R BT.601 standard), where the Y channel represents luminance information and the U and V channels represent chromatic aberration information. This conversion allows for better separation of the image's color information, facilitating region-weighted fusion. The specific conversion formula is as follows:

[0054] Based on the facial region division, the facial region is divided into multiple physiologically sensitive sub-regions. The RGB and YUV signals of each region are weighted and fused according to their area proportions to enhance the contribution of high signal-to-noise ratio regions to the spatiotemporal map. This weighting strategy effectively suppresses noise caused by small-scale occlusion interference (such as mask or glasses). The signals are then mapped using max-min normalization to form the initial spatiotemporal map.

[0055] The generated spatiotemporal map is a six-channel signal map. The horizontal axis represents the time dimension (i.e., the frame sequence of the video), the vertical axis is the spatial average intensity value of each ROI region, and the pixel value reflects the color change of that region at that time point. Each spatiotemporal map contains six channels, namely the weighted signal of the three RGB channels and the information of the three channels in the YUV space.

[0056] Specifically, orthogonal projection and filtering enhancement are performed based on the results in step S3, specifically as follows: Color signals are projected into a subspace orthogonal to the direction of skin reflection to strip away the illumination-dominated component while preserving subtle pulse-driven color variations. Considering the differences in POS performance across different color spaces, only the three-channel spatiotemporal map of the YUV color space is projected, while the RGB channels retain their original information without processing. This approach enhances the robustness of physiological signals while maintaining the integrity of visual features.

[0057] Suppose the original signal sequence of the YUV channel is a three-dimensional vector. These correspond to the Y (luminance), U, and V (chrominance) components, respectively. Within each frame time window, the signal is first mean-normalized, and then a projection matrix orthogonal to the skin color direction is constructed. P The normalized signal is mapped to this subspace:

[0058] in, The normalized YUV signal matrix, P These are the pre-defined orthogonal basis vectors. S This is the signal after POS projection processing.

[0059] POS uses two orthogonal projection results and Linear combination extraction of heart rate and pulse signals h :

[0060] in, and These are two orthogonal components. and The standard deviation is used to adjust the weights of the two channels, thereby improving the dynamic adaptability of signal extraction.

[0061] However, the effectiveness of the POS algorithm under strong interference conditions (such as LED flicker and partial facial occlusion) may still be limited by its linear illumination modeling assumptions. To further improve robustness, a 5th-order Butterworth bandpass filter is introduced after POS projection to suppress high-frequency noise caused by device jitter and low-frequency drift caused by body temperature fluctuations. The result after re-normalization and filtering is then stitched together according to the facial ROI structure and finally fused with the RGB channels to form a six-channel enhanced multi-scale spatiotemporal map—POS-MSTmap.

[0062] Based on steps S1, S2, and S3, multi-scale partitioning, POS projection, and bandpass filtering are combined to construct a multi-scale spatiotemporal graph with high signal-to-noise ratio and strong robustness in dynamic scenes, providing a more reliable input representation for subsequent effective learning of deep networks.

[0063] Specifically, step S4 involves designing a heart rate encoding module, which mainly comprises two components: a convolutional feature extraction module based on an improved ResNet18 architecture, and a polarization self-attention mechanism embedded within each residual block. Compared to traditional ResNet18, this invention incorporates several customized optimizations for non-contact heart rate estimation tasks. First, the input layer supports processing 6-channel POS-MSTmaps to meet the needs of joint modeling of spatiotemporal features. Second, a polarization self-attention module is introduced into each residual unit to effectively enhance the model's ability to focus on key spatial locations and channel features. Third, the normalization method is changed from batch normalization (BatchNorm) to instance normalization (InstanceNorm), enhancing the model's adaptability to differences in data distribution across individuals and scenarios. Furthermore, the output is designed with a multi-task structure, capable of both regressing heart rate values ​​and reconstructing heart rate-related rPPG signals.

[0064] Specifically, the convolutional feature extraction part is based on the ResNet18 structure with adjustments. First, a 7×7 convolutional kernel is used to perform preliminary downsampling on the input feature map to extract low-level texture and edge features. Then, the feature map is processed in depth through multiple improved residual blocks. Each residual block consists of two 3×3 convolutional layers, equipped with InstanceNorm and ReLU activation functions. At the same time, a skip connection mechanism is used to alleviate the gradient vanishing problem and further improve training stability.

[0065] The polarization self-attention mechanism consists of a channel attention branch (Ach) and a spatial attention branch (Asp), which model the dependencies between channels and between spatial regions, respectively. The polarization self-attention module preserves the high-resolution structure in one dimension through "polarization filtering" while compressing and filtering information in another dimension, achieving efficient and refined attention modeling.

[0066] In the channel branch, the input feature map is first compressed in spatial dimension through a 1×1 convolution while preserving high-resolution features across channels. Then, global matrix dot products are used to calculate channel attention weights, capturing semantic relationships across channels (such as the frequency domain features of blood volume pulses). This process significantly suppresses noise interference with channel information while retaining effective channel information. Let the input feature map be... ,in C For the number of channels, H and W The spatial dimension is represented by the feature map obtained through 1×1 convolution dimensionality reduction. for:

[0067] Calculate channel attention weights :

[0068] Long-range dependencies across channels are captured through matrix dot products, and channel attention weights are generated.

[0069] In the spatial branch, the input feature map is first compressed in terms of channel dimension using Global Average Pooling (GAP) to obtain a feature map with 1 channel. :

[0070] Spatial attention weights are generated by weighting the spatial dimensions using the spatial softmax operation. :

[0071] This operation dynamically adjusts the weights of various spatial locations in the feature map, enhancing important parts of the signal and suppressing noise components, especially in non-facial areas or areas affected by occlusion.

[0072] Based on the two branches mentioned above, the polarization attention module introduces a dynamic enhancement mechanism combining Softmax and Sigmoid to enhance signal diversity while suppressing noise and reducing interference from irrelevant components. Specifically, the channel branch enhances multimodal distribution modeling (such as the Gaussian distribution characteristics of heart rate) through Softmax, while strengthening the target frequency band (corresponding to the heart rate range) through the Sigmoid function in the noise frequency band (such as 0.8-3 Hz), effectively filtering out high-frequency noise and reducing the impact of the noise frequency band; while the spatial branch expands the dynamic range through the Sigmoid function to adapt to the distribution of the binary mask, thereby effectively and dynamically reducing the weight of non-face regions (such as background or occluders).

[0073] Specifically, the two branches are fused using both parallel and serial layouts. The parallel layout directly adds the two branches, preserving their independence, and is suitable for static scenarios; the serial layout optimizes features step by step through cascading filtering, and is suitable for dynamic interference scenarios.

[0074] Specifically, step S5 constructs a parallel noise encoder branch to model background interference components in the input data that are unrelated to heart rate, further enhancing the model's ability to separate heart rate signals. The core of this branch lies in using explicit adversarial mechanisms in the feature space to assist the main encoder in focusing on heart rate-related feature regions, thereby improving the accuracy and generalization ability of the final prediction. The noise encoder employs the same backbone structure as the main encoder, built upon the first two layers of ResNet18. During the input stage, convolution and downsampling operations are used for preliminary processing of the input image. Subsequently, the feature maps are sequentially processed through residual blocks to gradually extract intermediate spatial semantic features. This fully leverages the residual representation advantages of ResNet, reducing computational burden and making it suitable for rapidly modeling the distribution patterns of noise features. Finally, the intermediate features output from the noise encoder and the deep features output from the main encoder are fused and used as one of the inputs to the unified reconstruction module. In subsequent reconstruction modules, the fused features will be used for signal recovery and auxiliary supervision.

[0075] Unlike the main branch, the noise encoder does not introduce a polarization attention module, but instead maintains the traditional convolutional path, preserving the ability to model background and low-frequency interference. This design forms an explicit feature contrast mechanism, which helps the network to more clearly distinguish between heart rate signals and environmental noise, improving the robustness of predictions in weak signal scenarios.

[0076] The noise branch, as an auxiliary modeling strategy, effectively improves the purity of the model's representation of heart rate signals and its anti-interference performance by maintaining the simplicity of the network structure through structural parallel alignment and semantic contrastive learning.

[0077] Specifically, in step S6, after the main encoder and noise encoder complete feature extraction, the network fuses the output features of both and enhances the learning of image-level reconstruction and heart rate signals through the same decoder module. This not only promotes the alignment of main and auxiliary features but also provides an end-to-end positive constraint mechanism for the network.

[0078] The fusion feature first inputs a bottleneck coding structure composed of multiple residual blocks to enhance its expressiveness and task relevance. These residual block structures are similar to those in the encoder but do not include polarization attention modules, aiming to extract more abstract and compact feature representations. Subsequently, the decoder progressively upsamples through a series of deconvolution (ConvTranspose2d) modules, restoring the original image resolution layer by layer. Finally, the output stage employs different convolution head designs depending on the task type. In the multi-task training framework, the reconstruction branch generates an image with a structure consistent with the input image (e.g., a three-channel view) to help constrain the spatial structure consistency of the heart rate features. Simultaneously, another branch outputs a one-dimensional heart rate signal (e.g., rPPG), which is extracted and compressed using AdaptiveAvgPool2d and 1×1 convolutions.

[0079] By introducing a decoder module, the model not only has the regression capability for heart rate estimation, but can also reconstruct the dynamic changes related to the signal in the image domain, achieving stronger interpretability and cross-modal feature fusion capability, and further improving the model's adaptability and generalization performance in real-world scenarios.

[0080] Specifically, in step S7, the Adam optimizer is designed to obtain the optimal heart rate signal extraction model.

[0081] The specific steps of the embodiments of the present invention are as follows: For S1: Six physiologically sensitive sub-regions of the face are dynamically constructed, including the left / right cheeks, the mouth, and the extended forehead region. Specifically, the left and right cheeks are subdivided into upper and lower sub-regions (upper left cheek: feature points 0-2, 31, 41; lower left cheek: feature points 2-5, 48, 31; the same applies to the right cheek), and the mouth region is enclosed by feature points 5-12 and 60-66 forming a closed contour. The forehead region is enhanced in adaptability through geometric expansion: with the forehead reference point (mean of feature points 17-27) as the center, it is offset in the vertical direction (to the distance between the centers of the eyes) to form an expanded convex polygon.

[0082] For step S2: Based on the facial region division, the facial region is divided into 6 physiologically sensitive sub-regions, including the upper and lower parts of the left and right cheeks, the mouth area, and the extended forehead area. The signal is mapped to [0, 255] using max-min normalization to form an initial spatiotemporal map. Testing on the UBFC dataset shows that the signal-to-noise ratio (SNR) is improved by 4.2 dB, effectively mitigating the noise amplification problem caused by uniform weighting.

[0083] For step S3: To further improve robustness, a 5th-order Butterworth bandpass filter (bandpass range of 0.67-4 Hz) is introduced after POS projection to suppress high-frequency noise caused by device jitter and low-frequency drift caused by body temperature fluctuations.

[0084] For step S4: This invention proposes a non-contact heart rate estimation network based on a decoupled structure, mainly composed of three parts: a heart rate signal encoder based on a polarization self-attention mechanism, a noise encoder, and a feature fusion decoding module. Among them, the heart rate encoder is the key module in the network structure of this invention, extracting deep features highly correlated with the heart rate signal from a multi-scale spatiotemporal graph. (Reference) Figure 2 A comparison chart of the improved heart rate encoder is presented. The improved encoder's structural design incorporates residual connections and polarization self-attention mechanisms, enhancing feature extraction capabilities while improving the network's perception and robustness to key physiological signals. (Reference) Figure 3 A framework diagram of the polarization self-attention mechanism is presented.

[0085] For step S5: It is built based on the first two layers of ResNet18, and modularly encapsulated using resnet18_part(). In the input stage, the input image is processed using 7×7 convolutions and downsampling operations with a stride of 2.

[0086] For step S6, each upsampling module is followed by InstanceNorm and ReLU activation to ensure numerical stability and nonlinear modeling capability.

[0087] For step S7: This invention is conducted on the large-scale remote heart rate estimation public dataset VIPL-HR and the non-contact physiological signal analysis dataset UBFC-rPPG. The experimental platform uses an NVIDIA GeForce RTX 3090Ti GPU, with a batch size of 50, the Adam algorithm as the optimizer, an initial learning rate of 0.0005, and a weight decay term of 1e-4 to prevent overfitting. The loss function adopts a multi-task joint optimization strategy, assigning weights to the heart rate estimation loss and the rPPG signal reconstruction loss respectively, to achieve balanced fusion of multi-source supervision signals and obtain the optimal heart rate signal extraction model.

[0088] To verify the effectiveness of the present invention, the following simulation experiments were conducted.

[0089] Table 1 shows that the proposed method achieves a mean absolute error (MAE) of 3.85 bpm in uncontrolled scenarios on the VIPL-HR dataset, representing a 57.9% and 49.2% reduction compared to traditional methods CHROM (9.14 bpm) and POS (7.58 bpm), respectively. It also significantly outperforms deep learning methods RhythmNet (5.30 bpm) and rPPGNet (4.89 bpm). The root mean square error (RMSE) also reaches 4.98 bpm, a 38.8% reduction compared to RhythmNet (8.14 bpm). Paired tests based on multiple subject data show that this performance difference is statistically significant (p < 0.01). The model maintains good estimation performance even in complex scenarios (such as significant head movement and lighting changes). In the violent motion subset (head translation amplitude > 30 pixels), the RMSE of the proposed method is 7.44 bpm, a 20.9% improvement over RhythmNet (9.40 bpm), validating its strong adaptability to dynamic disturbances. Table 1 verifies the synergistic advantages of the improved multi-scale spatiotemporal map and the polarization self-attention module in dynamic interference environments: the multi-scale spatiotemporal map effectively reduces pixel offset noise caused by head movement by fusing spatiotemporal features of multiple regions such as the forehead and cheeks, and combines the orthogonal projection of the YUV channel by the POS algorithm, thereby improving the model's ability to model dynamic interference regions. refer to Figure 4The scatter plot distribution of the predicted and actual heart rates for each video segment in the VIPL-HR dataset test set is presented. It can be observed that the scatter plots are densely distributed along the diagonal, indicating that the model maintains high consistency with the actual values ​​over a relatively wide heart rate range. The results demonstrate that the model has good generalization ability at different heart rate levels.

[0090] refer to Figure 5 The time-domain comparison between the predicted rPPG waveform and the reference ECG waveform is given. The predicted signal is highly aligned with the real ECG signal in terms of periodicity, peak position and waveform morphology, indicating that the model proposed in this invention can not only accurately estimate the average heart rate, but also has a strong ability to restore physiological waveforms.

[0091] Table 1 Comparison of Heart Rate Estimation Results for the VIPL-HR Dataset

[0092] The specific embodiments of the present invention have been described in detail above. The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions of the present invention. Rather, they are technical solutions consistent with the principles and novel features of the present invention and should all be within the scope of protection defined by the claims.

Claims

1. A non-contact heart rate signal extraction method based on multi-scale spatiotemporal graphs and polarization self-attention, characterized in that, Specifically, the steps include the following: S1 transforms rigid region partitioning into dynamic adaptive flexible modeling, constructing a flexible multi-region ROI partitioning strategy; S2 converts the original image into the YUV color space and introduces an area-weighted multi-scale fusion strategy to enhance the contribution of high signal-to-noise ratio regions and improve the quality of multi-scale spatiotemporal maps. S3, based on the POS algorithm of skin color space modeling, uses the skin color reflection characteristics to orthogonally project the signal, remove steady-state illumination interference and enhance the pulse signal; S4, construct a heart rate encoder to extract deep features that are highly correlated with heart rate signals from multi-scale spatiotemporal maps; S5 constructs a parallel noise encoder branch structure to model background interference components unrelated to the heart rate signal in the multi-scale spatiotemporal graph of the input; through explicit adversarial behavior in the feature space, it assists the main encoder in focusing on heart rate-related feature regions, thereby improving the accuracy and generalization ability of the final prediction. S6: After the main encoder and noise encoder complete feature extraction, the network fuses the output features of the two and further enhances the learning of image-level reconstruction and heart rate signals through the decoder module; promotes the mutual alignment of main and auxiliary features and provides an end-to-end positive constraint mechanism for the network. S7. Train the network model using the Adam optimizer. End the training when the model tends to stabilize; otherwise, go to step S1 and repeat the above steps. Finally, a highly accurate and stable full-process enhanced heart rate estimation model was trained.

2. The non-contact heart rate signal extraction method based on multi-scale spatiotemporal graphs and polarization self-attention as described in claim 1, characterized in that, In step S1, a flexible multi-region ROI partitioning strategy is adopted. Specifically, based on the dlib 68-point facial key point model, facial physiologically sensitive sub-regions are dynamically constructed. All ROIs are generated into binary masks through polygon2mask, and the number of effective pixels is counted for subsequent signal weighting.

3. The non-contact heart rate signal extraction method based on multi-scale spatiotemporal graphs and polarization self-attention as described in claim 1, characterized in that, In step S2, a multi-scale spatiotemporal map is constructed based on the ROI regions divided in step S1. Specifically, the original image is converted from the RGB color space to the YUV color space, where the Y channel represents luminance information and the U and V channels represent color difference information. Through this conversion, the color information of the image can be separated and region-weighted fusion is performed. The specific conversion formula is as follows: ； Based on the division of the facial region, the facial region is divided into multiple physiologically sensitive sub-regions; the RGB and YUV signals of each region are weighted and fused according to their area ratio to enhance the contribution of high signal-to-noise ratio regions to the spatiotemporal map; through this weighting strategy, noise caused by small-scale occlusion interference is suppressed; the signals are mapped by max-min normalization to form the initial spatiotemporal map. The generated spatiotemporal map is a six-channel signal map. The horizontal axis represents the time dimension, the vertical axis is the spatial average intensity value of each ROI region, and the pixel value reflects the color change of the region at that time point. Each spatiotemporal map contains six channels, namely the weighted signal of the three RGB channels and the information of the three channels in the YUV space.

4. The non-contact heart rate signal extraction method based on multi-scale spatiotemporal graphs and polarization self-attention as described in claim 1, characterized in that, Based on the results in step S3, orthogonal projection and filtering enhancement are performed. Specifically, the color signal is projected into a subspace orthogonal to the skin reflection direction, stripping away the light-dominated component and retaining the subtle color changes driven by the pulse. Considering the differences in the performance of POS in different color spaces, only the three-channel spatiotemporal map of the YUV color space is projected, while the RGB channels retain the original information without processing. This improves the robustness of physiological signals while maintaining the integrity of visual features. Suppose the original signal sequence of the YUV channel is a three-dimensional vector. These correspond to the Y luminance, U, and V chrominance components, respectively. Within each frame time window, the signal is mean-normalized, and a projection matrix orthogonal to the skin color direction is constructed. P The normalized signal is mapped to this subspace, expressed by the following formula: ； in, The normalized YUV signal matrix, P These are the pre-defined orthogonal basis vectors. S The signal is after POS projection processing; POS uses two orthogonal projection results and Linear combination extraction of heart rate and pulse signals h， The formula is as follows: ； in, and These are two orthogonal components. and The standard deviation is used to adjust the weights of the two channels, thereby improving the dynamic adaptability of signal extraction; The effectiveness of the POS algorithm may still be limited by its linear illumination modeling assumption under strong interference conditions. To further improve robustness, a 5th-order Butterworth bandpass filter is introduced after POS projection to suppress high-frequency noise caused by device jitter and low-frequency drift caused by body temperature fluctuations. The filtered result is renormalized and the channels are stitched according to the facial ROI structure. Finally, it is fused with the RGB channels to form a six-channel enhanced multi-scale spatiotemporal map, namely POS-MSTmap. Based on steps S1, S2, and S3, multi-scale partitioning, POS projection, and bandpass filtering are combined to construct a multi-scale spatiotemporal graph with high signal-to-noise ratio and strong robustness in dynamic scenes, providing a more reliable input representation for subsequent deep network learning.

5. The non-contact heart rate signal extraction method based on multi-scale spatiotemporal graphs and polarization self-attention as described in claim 1, characterized in that, In step S4, a heart rate encoding module is designed. Specifically, the heart rate encoder mainly includes two components: the first is a convolutional feature extraction module based on the improved ResNet18 architecture. The second is the polarization self-attention mechanism embedded within each residual block; Several customized optimizations were made for the non-contact heart rate estimation task. First, the input layer supports processing 6-channel POS-MSTmap to meet the needs of joint modeling of spatiotemporal features. Secondly, a polarization self-attention module is introduced into each residual unit to enhance the model's ability to focus on key spatial locations and channel features. Thirdly, the normalization method is changed from batch normalization (BatchNorm) to instance normalization (InstanceNorm) to enhance the model's adaptability to differences in data distribution across individuals and scenarios. In addition, the output is designed with a multi-task structure, which can both complete heart rate value regression and reconstruct heart rate-related rPPG signals.

6. The non-contact heart rate signal extraction method based on multi-scale spatiotemporal graphs and polarization self-attention according to claim 5, characterized in that, The convolutional feature extraction module is based on the ResNet18 structure and is modified accordingly. First, a 7×7 convolutional kernel is used to perform preliminary downsampling on the input feature map to extract low-level texture and edge features. Then, the feature map is processed in depth through multiple improved residual blocks. Each residual block consists of two 3×3 convolutional layers and is equipped with InstanceNorm and ReLU activation functions. At the same time, a skip connection mechanism is used to alleviate the gradient vanishing problem and further improve training stability. The polarization self-attention mechanism consists of a channel attention branch (Ach) and a spatial attention branch (Asp), which respectively model the dependencies between channels and between spatial regions. The polarization self-attention module retains the high-resolution structure of one dimension through "polarization filtering" while compressing information in another dimension, thereby achieving efficient and refined attention modeling. In the channel branch, the input feature map is first compressed in spatial dimension by a 1×1 convolution while preserving high-resolution features in the channel; Then, global matrix dot product is used to calculate and generate channel attention weights, capturing cross-channel semantic relationships; this process can significantly suppress noise interference with channel information while retaining effective channel information; let the input feature map be... ,in C For the number of channels, H and W The spatial dimension is represented by the feature map obtained through 1×1 convolution dimensionality reduction. The formula is as follows: ； Calculate channel attention weights The formula is as follows: ； Long-range dependencies across channels are captured through matrix dot products, and channel attention weights are generated. In the spatial branch, the input feature map is first compressed in channel dimension through global average pooling to obtain a feature map with 1 channel. The formula is as follows: ； Spatial attention weights are generated by weighting the spatial dimensions using the spatial softmax function. The formula is as follows: ； This operation dynamically adjusts the weights of each spatial location in the feature map, enhancing important parts of the signal and suppressing noise components, especially in non-facial areas or areas affected by occlusion. Based on the two branches mentioned above, the polarization attention module enhances signal diversity while suppressing noise and reducing interference from irrelevant components by introducing a dynamic enhancement mechanism combining Softmax and Sigmoid. Specifically, the channel branch enhances multimodal distribution modeling through the Softmax function, while strengthening the target frequency band through the Sigmoid function in the noise band to filter out high-frequency noise and reduce the impact of the noise band. The spatial branch expands the dynamic range through the Sigmoid function to adapt to the distribution of the binarized mask, thereby effectively and dynamically reducing the weight of non-facial regions.

7. The non-contact heart rate signal extraction method based on multi-scale spatiotemporal graphs and polarization self-attention as described in claim 6, characterized in that, The channel branch and the spatial branch are fused using both parallel and serial layouts. The parallel layout directly adds the two branches, preserving their independence, and is suitable for static scenarios. The serial layout optimizes features step by step through cascaded filtering, and is suitable for dynamic interference scenarios.

8. The non-contact heart rate signal extraction method based on multi-scale spatiotemporal graphs and polarization self-attention according to claim 1, characterized in that, In step S5, a parallel noise encoder branch is constructed to model background interference components in the input data that are unrelated to heart rate, further enhancing the model's ability to separate heart rate signals. The core of this branch lies in assisting the main encoder to focus on heart rate-related feature regions through explicit adversarial interaction in the feature space, thereby improving the accuracy and generalization ability of the final prediction. Specifically, the noise encoder adopts the same backbone structure as the main encoder, built based on the first two layers of ResNet18. In the input stage, convolution and downsampling operations are also used to perform preliminary processing on the input image. Then, the feature map is passed through residual blocks in sequence to gradually extract the mid-level spatial semantic features. The residual representation advantage of ResNet is fully utilized to reduce the computational burden and is suitable for quickly modeling the distribution pattern of noise features. Finally, the mid-level features output by the noise encoder and the deep features output by the main encoder are fused as one of the inputs to the unified reconstruction module. In the subsequent reconstruction module, the fused features will be used for signal recovery and auxiliary supervision. Unlike the main branch, the noise encoder does not introduce a polarization attention module, but instead maintains the traditional convolution path, preserving the ability to model background and low-frequency interference. This design forms an explicit feature contrast mechanism, which helps the network to more clearly distinguish between heart rate signals and environmental noise, improving the robustness of prediction in weak signal scenarios. Noise branches, as an auxiliary modeling strategy, improve the purity of the model's representation of heart rate signals and its anti-interference performance by maintaining the simplicity of the network structure through structural parallel alignment and semantic contrastive learning.

9. The non-contact heart rate signal extraction method based on multi-scale spatiotemporal graphs and polarization self-attention according to claim 1, characterized in that, In step S6, after the main encoder and noise encoder complete feature extraction, the network will fuse the output features of the two and enhance the learning of image-level reconstruction and heart rate signals through the same decoder module; this not only promotes the mutual alignment of main and auxiliary features, but also provides an end-to-end positive constraint mechanism for the network. The fusion features are first input into a bottleneck coding structure consisting of multiple residual blocks to enhance its expressiveness and task relevance; these residual block structures are similar to those in the encoder, but do not contain polarized attention modules, and are designed to extract more abstract and compact feature representations; Then, the decoder gradually upsamples through a series of deconvolution modules to restore the original image resolution layer by layer; Finally, the output stage adopts different convolution head designs according to the task type; in the multi-task training framework, the reconstruction branch generates an image with the same structure as the input graph to help constrain the spatial structure consistency of the heart rate features; at the same time, another branch outputs a one-dimensional heart rate signal, which is extracted and compressed through AdaptiveAvgPool2d and 1×1 convolution. By introducing a decoder module, the model not only has the regression capability for heart rate estimation, but can also reconstruct the dynamic changes related to the signal in the image domain, achieving stronger interpretability and cross-modal feature fusion capability, and further improving the model's adaptability and generalization performance in real-world scenarios.

10. The non-contact heart rate signal extraction method based on multi-scale spatiotemporal graphs and polarization self-attention according to claim 1, characterized in that, In step S7, the Adam optimizer is used to obtain the optimal heart rate signal extraction model.

Images