Non-contact heart rate estimation method fusing physical prior and Transform

Abstract

The invention discloses a non-contact heart rate estimation method fusing physical prior and Transform, and the method comprises the steps: designing a double-branch spatial-temporal feature fusion module which is used for separating interference from the source and enhancing a weak pulse wave signal; the dual-branch spatio-temporal feature extraction module is configured to separate interference signals and enhance pulse wave signals, and comprises a motion branch and an appearance branch. According to the invention, a combined architecture of convolution and Transform is designed, and a relative position coding module is introduced. According to the method, the normalized frame difference is introduced to serve as input of the motion branches, appearance branch feature fusion is combined, and noise interference caused by illumination and non-rigid motion is effectively restrained. The high-precision global modeling of a weak heart rate change mode is realized by combining a 3D CNN and a Transform architecture and utilizing a time difference attention mechanism.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and biomedical engineering, and particularly relates to a non-contact heart rate estimation method fusing physical priori and a Transformer. BACKGROUND

[0002] Physiological estimation is an important means for assessing human health, disease prevention and clinical diagnosis. Heart rate (HR), as one of the most important vital signs, reflects the pumping function of the heart and the regulation state of the autonomic nervous system. Traditional contact heart rate measurement methods, such as electrocardiogram (ECG) and photoplethysmography (PPG), have high measurement accuracy, but they require wearing electrode patches or finger clip sensors, which are not only cumbersome to operate, but also cause discomfort and even skin allergies in long-term wear, and limit the freedom of the subject, making it difficult to meet the needs of unobtrusive and continuous monitoring in daily life. Thanks to the development of computer vision and deep learning in the past decade, computers can capture subtle physiological signal changes from video images. Remote photoplethysmography (rPPG) technology has emerged based on video. This technology uses an ordinary camera to capture facial video, analyzes the subtle changes in facial skin color to extract the pulse wave signal, and then estimates the heart rate. With this advantage, non-contact heart rate estimation technology has been widely applied in remote medical care, driver fatigue monitoring, public safety monitoring, home health care, etc.

[0003] Existing non-contact heart rate estimation methods can be roughly divided into traditional signal processing methods and deep learning-based methods. Traditional rPPG algorithms mainly rely on manually designed features and statistical methods, such as independent component analysis (ICA), principal component analysis (PCA), and color model-based algorithms (such as CHROM, POS, etc.). These methods usually assume that the changes in facial skin reflectance are mainly caused by blood volume changes, but in actual application scenarios, fluctuations in environmental light, head movements of the subject, and facial expression changes will introduce serious noise interference, resulting in poor robustness of traditional algorithms in uncontrolled environments, making it difficult to extract high-quality rPPG signals.

[0004] With the rapid development of deep learning, the heart rate estimation method based on deep learning shows better feature extraction ability than traditional methods. Early deep learning methods mostly use convolutional neural networks to directly learn physiological features from face images or spatio-temporal graphs. For example, 3D CNN is used to directly process video frame sequences to capture spatio-temporal features, or 2D CNN is used to extract spatial features and then combined with recurrent neural network (RNN) for time series modeling. However, the method based on pure CNN is limited by the local receptive field of the convolution kernel, and it is difficult to capture the global context information of the image, and the ability to exclude face background noise (such as hair, glasses obstruction) is limited. Although combining RNN (such as LSTM) can capture long-distance time dependence, it is difficult to perform parallel computation, and it is prone to gradient disappearance problem when processing long video sequences.

[0005] The Transformer architecture has achieved great success in the field of natural language processing, and with its unique self-attention mechanism, it can effectively capture long-distance dependencies and global context information in sequence data. In the non-contact heart rate estimation task, the rPPG signal is essentially a sequence signal that fluctuates over time, and different regions of the face (such as the forehead and cheeks) contain different intensities of pulse waves, with significant spatial correlation. Although some existing research attempts to introduce Transformer, it is often used as a time series classifier behind CNN, and fails to fully utilize its potential in spatio-temporal feature fusion, and ignores how to efficiently focus on the skin area rich in blood flow information from noisy video. Therefore, designing a network structure that can utilize the global modeling capability of Transformer and effectively suppress motion and light noise with physical prior is the key to improving the accuracy of non-contact heart rate estimation. SUMMARY

[0006] To better improve the accuracy of non-contact heart rate estimation and solve the problem of poor robustness of existing technology under motion interference and light changes, a non-contact heart rate estimation method fusing physical prior and spatio-temporal Transformer is provided. The overall architecture of the method is divided into a feature extraction module, a Transformer module and an rPPG prediction head module, as shown in Figure 1 The method mainly includes the following two parts:

[0007] 1. Design a dual-branch spatio-temporal feature fusion module

[0008] The application designs a dual-branch spatio-temporal feature fusion module for separating interference and enhancing weak pulse wave signals from the source. The dual-branch spatio-temporal feature extraction module is configured to separate interference signals and enhance pulse wave signals, and includes a motion branch and an appearance branch. The motion branch serves as a change detector, taking normalized frame difference as input, and removing static appearance information and illumination intensity influence by calculating normalized frame difference, focusing on capturing color changes (rPPG signals) caused by blood flow and displacements caused by head movement. The appearance branch takes the original video frame as input, and is responsible for extracting static and quasi-static facial features, including skin texture, skin color distribution and facial feature spatial position. The features extracted by the above two branches are fused to generate enhanced spatio-temporal tube features (Tube Tokens), which explicitly separate and amplify physiological signals, providing advanced semantic information for subsequent networks.

[0009] 2. Design a combined architecture of convolution and Transformer

[0010] The application designs a combined architecture of convolution and Transformer, and introduces a relative position encoding module. In order to overcome the defects of pure CNN limited to local receptive field and pure Transformer high in calculation cost and lacking position priori, the application divides the features output by the feature extraction module into smaller blocks (Tube Tokens) to aggregate the semantics of adjacent space-time and reduce the calculation cost. Considering that the essential feature of rPPG signal is relative and translation invariant, rather than absolute coordinates, the application introduces relative position encoding in the Transformer module. The encoding provides explicit and quantitative position information for the Transformer, enabling the network to effectively learn the periodicity of rPPG signal, automatically aggregate the facial skin area, and distinguish effective signals from motion artifacts based on the similarity of adjacent skin area signals, so as to learn the physical laws of rPPG signal in space-time. In addition, the application also designs a time difference attention module, which uses time difference convolution for query (Q) and key (K) when calculating attention score. The time difference convolution is composed of standard 3D convolution and time difference term, and the calculated attention score represents the similarity of change pattern, so that the model can globally and adaptively associate and enhance all quasi-periodic rPPG signal features.

[0011] Compared with the prior art, the present application has the following beneficial effects: by introducing normalized frame difference as the input of the motion branch, and combining with the appearance branch feature fusion, the noise interference caused by illumination and non-rigid motion is effectively suppressed. By combining 3D CNN and Transformer architecture, the time difference attention mechanism is used to realize high-precision global modeling of weak heart rate change patterns. The experimental results on the V4V dataset show that the mean absolute error (MAE) of the present application is only 1.77, the root mean square error (RMSE) is 5.48, and the Pearson correlation coefficient (r) reaches 0.92, which is significantly better than the existing mainstream methods such as DeepPhys, PhysNet and DRP-NET. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is the overall flow architecture diagram of the non-contact heart rate estimation method based on spatio-temporal feature fusion Transformer of the present application.

[0013] Figure 2 is a structural schematic diagram of the dual-branch spatio-temporal feature extraction module of the present application.

[0014] Figure 3 is a structural schematic diagram of the Transformer module of the present application.

[0015] Figure 4 is a structural schematic diagram of the rPPG prediction head module of the present application.

[0016] Figure 5 is a heart rate schematic diagram of the V4V dataset of the present application.

[0017] Figure 6 is a blood pressure schematic diagram of the V4V dataset of the present application. DETAILED DESCRIPTION

[0018] The present application will be described in detail below in combination with the drawings and examples.

[0019] The specific implementation process of the non-contact heart rate estimation method based on spatio-temporal feature fusion Transformer (ST-Former) mainly includes constructing a network model based on spatio-temporal feature fusion Transformer and training the network model based on spatio-temporal feature fusion Transformer. The overall network architecture is as shown in Figure 1 The embodiments of the present application will be described in detail below in combination with the drawings.

[0020] Step one, constructing a heart rate estimation network based on ST-Former;

[0021] Step 11, data preprocessing;

[0022] In order to reduce the background redundant information and improve the robustness of the model under different light conditions, the present application adopts a standardized preprocessing procedure to extract the region of interest (ROI). The pre-trained MTCNN model is used to detect the face region in the video frame, and the ROI range containing the face region is determined. The detected ROI region is cropped from the original video frame and uniformly adjusted to a resolution of 128x128 pixels. In order to reduce the influence of light condition changes on signal extraction, the pixel brightness value of the cropped ROI image is normalized to the range of [0, 1]. The cropped ROI sequence is sampled at a sampling rate of 25 frames / second, and is divided into time windows with a length of 160 frames, each window corresponding to a video duration of 6.4 seconds. In order to train and evaluate the model, it is necessary to process the physiological signals synchronized with the video as labels. The present application uses the arterial blood pressure (ABP) signal as the basis, time-synchronizes the ABP signal with the face video, and also resamples at a frequency of 25 Hz. The ABP signal is divided into time windows of 6.4 seconds (corresponding to 160 frames of video). In order to extract the pulse wave features for heart rate estimation from the ABP signal, a pseudo PPG signal is generated by standardization, detrending and band-pass filtering. The detrending algorithm is used to eliminate the baseline drift effect caused by blood pressure changes; the cutoff frequency of the band-pass filter is preset to 0.5 Hz to 3.0 Hz to cover the normal human heart rate range. The generated pseudo PPG signal is subjected to fast Fourier transform to calculate the power spectral density. The frequency corresponding to the maximum amplitude of the power spectral density is analyzed, which is the true value of the heart rate corresponding to the time window. The 160-frame, 128x128-resolution ROI sequence processed by the above-mentioned preprocessing will be further processed and sent to the feature extraction network.

[0023] Step 12, a spatio-temporal feature fusion module;

[0024] In order to separate the interference from the source and enhance the weak pulse wave signal, the present application constructs a double-branch spatio-temporal feature fusion module at the front end of the network, as shown in Figure 2 In traditional heart rate estimation based on light model, the light intensity change reflected by the skin contains specular reflection and diffuse reflection components, and the light model can be modeled as formula (1).

[0025] (1);

[0026] Among them, is the pixel value, is the light intensity, the specular reflection component, the diffuse reflection component, noise. Only the diffuse reflection component contains the pulse wave information modulated by the blood volume change. Finally, the pixel value can be represented as Linear combination. Therefore, the task of extracting the pulse wave signal from the RGB signal is transformed into decomposing As shown in equation (2).

[0027] (2).

[0028] Inspired by DeepPhys and the illumination model, the present application adopts normalized frame difference as the input of the motion branch. Specifically, by calculating the derivative of the time , the constant term contained in the pixel can be eliminated, as shown in equation (3).

[0029]

[0030] (3).

[0031] By normalizing the influence of the illumination intensity , by calculating the difference between the current frame and the previous frame, and dividing by the pixel intensity of the current frame, as shown in equation (4).

[0032] (4).

[0033] The normalized frame difference can be represented as equation (5).

[0034] (5).

[0035] This operation mathematically approximately eliminates the influence of illumination intensity and static texture. Therefore, the motion branch actually acts as a change detector, which can eliminate most of the static background information from the source and focus on capturing the weak color changes caused by blood flow and the non-rigid displacement caused by head movement.

[0036] The appearance branch takes the original video frame as input. Although the motion branch extracts the change information, the structure information of the face is lost. The appearance branch is responsible for learning the static and quasi-static facial features, including the texture of the skin, the basic skin color distribution, and the spatial position of the five organs. These information is crucial for the subsequent network to understand the facial structure, locate the skin area, and distinguish the face from the background.

[0037] After the two branches extract features through 3D convolutional layers respectively, they are spliced in the channel dimension or the features are fused through addition operation, as shown in Figure 2 This process combines the weak physiological signal changes captured by the motion branch with the anatomical context information provided by the appearance branch, generating high-dimensional spatiotemporal features containing rich semantic information. This enhanced feature not only preserves the spatial structure of the original video, but also explicitly amplifies the feature strength of the rPPG signal.​

[0038] Finally, the fused high-dimensional spatio-temporal features are sent to the subsequent Transformer module. Since the signal enhancement and noise suppression have been completed in the input stage, the Transformer module can more efficiently utilize its self-attention mechanism to model the global temporal sequence of Tube Tokens, thereby accurately capturing the periodicity of heart rate fluctuations.

[0039] Step 13, the Transformer module;

[0040] To further extract the periodicity of physiological signals with long-range dependencies from high-dimensional spatio-temporal features, the present application cascades a global temporal modeling module based on video Transformer after the dual-branch feature fusion module. This module mainly consists of a block and position embedding module, a multi-layer time difference Transformer encoder, and a feedforward neural network, as shown in Figure 3 .

[0041] First, to adapt to the serialized input format of the Transformer, the 3D convolution projection operation is used to cut the continuous video frame features into non-overlapping blocks in the spatio-temporal dimension. This process flattens the originally continuous feature map into a sequence of vectors containing local spatio-temporal information, reducing the computational dimension while preserving the spatio-temporal correlation within the local neighborhood, laying the foundation for subsequent global attention calculation.

[0042] The feature sequence then enters the encoder composed of stacked Transformer blocks. Unlike conventional visual Transformers, the present application adopts a time-difference multi-head self-attention mechanism (TD-MHSA). In the conventional self-attention mechanism, the query Q, key K, and value V are usually generated only by linear transformation, while in the TD-MHSA of the present application, the generation of the query Q and the key K introduces a time-difference convolution operation. This operation explicitly captures the rate of change of features over time by calculating the difference between adjacent frame features. Since the pulse wave PPG signal essentially reflects the change of blood volume over time, its derivative information contains rich and essential heart rate features. The attention map generated by the time-difference convolution can guide the network to more sensitively focus on the skin areas that have periodic intensity changes over time, while effectively suppressing the interference caused by static backgrounds or non-periodic head movements.

[0043] The output of each attention layer is further connected to a spatio-temporal feedforward network, which aims to correct local inconsistencies in the features extracted by the attention mechanism and enhance the robustness of the features. After processing by multiple Transformer blocks, the network extracts a latent vector containing global heart rate cycle information. Considering the downsampling effect during the Transformer processing, in order to restore the temporal resolution of the original video, this module includes a temporal upsampling unit to restore the length of the feature sequence to the original number of video frames. Finally, the features with restored temporal resolution are fed into an rPPG regression head composed of one-dimensional convolutional layers, which is responsible for mapping high-dimensional features to a one-dimensional continuous pulse wave signal, from which the accurate heart rate value can be further calculated.

[0044] Step 14, rPPG prediction head;

[0045] After global modeling of spatio-temporal features via the aforementioned Video Transformer module, the network outputs a latent feature sequence containing rich physiological cycle information. In order to map this abstract high-dimensional feature to a specific, continuous pulse wave signal, the present application designs an rPPG signal prediction and temporal reconstruction module, as shown in Figure 4 The module mainly consists of a feature dimension reduction decoding unit and a temporal resolution recovery unit. The high-dimensional spatio-temporal feature sequence from the Transformer module is received, and a one-dimensional convolutional neural network structure is used to compress the channel dimension of the feature sequence step by step. This step maps abstract semantic features to a single-channel physiological signal intensity sequence by learning the combination of channel feature weights; during channel compression, normalization processing and nonlinear activation functions are introduced to normalize the feature distribution and enhance the fitting ability for the nonlinear characteristics of the pulse wave; in view of the resolution loss of the Transformer feature sequence in the time dimension, linear interpolation upsampling technology is used to restore the time length of the single-channel signal sequence to be consistent with the original input video frame number, generating an rPPG pulse wave signal strictly aligned with the video frame in time.

[0046] Step two, training an ST-Former-based heart rate estimation model;

[0047] Step 21, forward propagation;

[0048] The application adopts a spatio-temporal feature fusion Transformer network for rPPG signal prediction. First, a batch of video segment tensors is input, with a size of [B, 3, 160, 128, 128], where B represents the batch size, 160 represents the number of time frames, and 128 represents the height and width of the image. The input video is divided into two paths and processed in parallel: the first path directly inputs the original RGB frame into the appearance branch, and uses 3D convolution layers with kernel sizes of 1x5x5 and 1x3x3 to gradually extract texture features. The spatial size of the feature map is compressed to 1 / 8 of the original size through three times of spatial downsampling while keeping the time dimension unchanged, obtaining appearance features with a size of [B, 48, 160, 16, 16]; the second path first performs normalized difference preprocessing on the input video, calculates the difference between the current frame and the previous frame and divides it by the intensity of the current frame to obtain pure physical motion representation, then inputs the structure symmetric motion branch, and extracts dynamic pulse information through 3x5x5 spatial-temporal convolution with large receptive field. Similarly, the motion features output have a size of [B, 48, 160, 16, 16]. Finally, the two features are concatenated in the channel dimension and fused through 1x1 convolution to obtain a fused spatio-temporal feature map with 96 channels, with a size of [B, 96, 160, 16, 16].

[0049] The fused feature map then enters the spatio-temporal block embedding module, which performs 3D convolution operations with kernel size and step size of [4, 4, 4] to jointly aggregate and downsample the spatio-temporal dimensions. The time dimension is compressed from 160 to 40, and the spatial dimension is compressed from 16x16 to 4x4, resulting in compressed features with a size of [B, 96, 40, 4, 4]. To adapt to the Transformer input, the time, height, and width dimensions are flattened into sequence dimensions to generate a feature sequence containing 640 tokens, with the tensor shape changing to [B, 640, 96]. The sequence is input into a spatio-temporal Transformer encoder containing 12 stacked modules. In each module, the multi-head self-attention mechanism with relative position bias and the 3D convolution feedforward network are used to capture long-range spatio-temporal dependencies, and the output dimension remains [B, 640, 96]. Finally, the encoded feature sequence is transposed to [B, 96, 640] and input into the prediction head. Through two one-dimensional convolution layers, the channel number is gradually reduced to 1, and linear interpolation upsampling is used to restore the time sequence length from 640 to the original frame number 160. Finally, the rPPG pulse wave prediction signal with a shape of [B, 160] is output.

[0050] Step 22, loss function;

[0051] In order to guide the deep neural network to accurately regress the rPPG signal from the complex facial video, a multi-domain hybrid loss function is designed, which constrains the predicted signal in time domain, frequency domain and heart rate value domain. Through the back propagation algorithm, the network parameters are updated. Firstly, the time domain trend constraint loss Considering that the waveform trend and peak position of the rPPG signal are crucial for heart rate calculation, and due to the difference in illumination changes and camera parameters, the absolute amplitude of the predicted signal is often difficult to determine and has no physiological significance, therefore, the embodiment does not use the mean square error sensitive to the amplitude, but uses the negative Pearson correlation coefficient as the time domain constraint, as shown in formula (6). The loss function calculates the ratio of the covariance and the product of the standard deviations between the predicted signal and the real physiological signal, aiming to maximize the linear correlation between the two, so that the predicted waveform is highly consistent with the real signal in trend.

[0052] (6);

[0053] Secondly, in order to enhance the robustness of the model to non-periodic noise (such as head micro-motion or environmental light interference), the frequency domain periodicity constraint loss is introduced, as shown in formula (7). Specifically, the predicted rPPG signal and the real signal are first subjected to fast Fourier transform (FFT) respectively, and the power spectral density (PSD) is calculated. Since the physiological signal has a significant sparse periodicity feature, and the noise usually shows a chaotic wide frequency distribution, by calculating the root mean square error or cross entropy distance in the frequency domain, the model can be forced to generate a signal power concentrated in the heart rate main frequency band, effectively suppressing the low-amplitude non-periodic noise component.

[0054] (7);

[0055] In addition, in order to ensure the accuracy of the final measurement result, the heart rate value regression loss is introduced as a supervision signal in the training stage, as shown in formula (8). The time domain predicted signal is post-processed, and the predicted heart rate value is calculated by peak detection or frequency domain maximum value extraction. The L2 norm distance between the predicted heart rate value and the real heart rate value is calculated.

[0056] (8);

[0057] The final network optimization target is defined as the weighted sum of the above time domain trend loss, frequency domain periodicity loss and heart rate value loss, through multi-task joint learning, the model is prompted to retain the signal form details while obtaining more accurate heart rate estimation results, wherein as shown in formula (9).

[0058] (9);

[0059] The experimental configuration and analysis are briefly described below, and the actual detection effect is improved by comparing and analyzing the experimental results.

[0060] 1. Experimental conditions

[0061] The hardware test platform of the present application is a processor Intel(R) Xeon(R) CPU E5-2620 v4, the main frequency is 2.10 GHz, the memory is 16 GB, and the graphics card is Nvidia GeForce GTX 2080ti. The software platform is Ubuntu 18.04 operating system, and the development environment is PyCharm. The programming language is Python, and the deep learning network architecture is PyTorch. The network uses Adam optimizer and sets the learning rate to 1e-4, and the training epoch is set to 20.

[0062] 2. Experimental data

[0063] The performance evaluation of the present application mainly uses the V4V data set:

[0064] The V4V data set contains a wide range of subjects, a total of 179 subjects, the age range of the subjects is 18 to 66 years old, the subjects come from various countries, the video data of the data set is a face video, the face video has a total of 1358, all videos are recorded at a resolution of 1040x1392 pixels, the video frame rate is 25 FPS. The data set is divided into a training set, a test set and a validation set. The training set contains 100 subjects, a total of 724 videos, and the average video length is 44.2 seconds. The validation set contains 39 subjects, a total of 276 videos, and the average video length is 42.9 seconds. The test set contains 40 subjects, a total of 358 videos, and the average video length is 45.8 seconds. V4V contains physiological signal data synchronized with the video, heart rate HR, blood pressure ABP, as shown in Figure 5 、 6

[0065] 3. Performance comparison

[0066] The five existing technology comparison classification methods used in the present application are as follows:

[0067] (1) The dual-stream network model based on convolution attention mechanism proposed by Chen et al. in 2018 extracts spatial texture and temporal change features through parallel appearance branch and motion branch, and uses attention mask to guide signal extraction, which is referred to as DeepPhys method.

[0068] ​(2) Yu et al. proposed an end-to-end spatio-temporal network model based on 3D convolutional neural network in 2019, which can learn spatio-temporal features directly from the original video frame sequence without preprocessing, referred to as PhysNet method.

[0069] (3) Dae et al. proposed an end-to-end lightweight rPPG estimation method in 2022, aiming to solve the limitations of existing 3DCNN models with high memory occupancy and STMap-based methods relying on complex preprocessing. By introducing an axis projection mechanism and designing a hybrid loss function, the learning ability of the model is effectively enhanced, referred to as APNET method.

[0070] (4) Gyutae et al. proposed a deep network model based on 3D CNN in 2024, which further extracts features by stacking dilated convolution blocks to enhance the representation ability of weak physiological signals, referred to as DRP-NET method.

[0071] (5) Jiho et al. proposed a physiological measurement model based on self-supervised visual Transformer in 2024, which uses a pre-trained DINO model on a large-scale dataset as the feature extraction backbone, and migrates general visual features to the rPPG task, referred to as DINO-rPPG method.

[0072] In the experiment, MAE, RMSE, and Pearson correlation coefficient are used to evaluate the heart rate estimation task. MAE is used to measure the average level of absolute error between predicted heart rate values and true heart rate values. It directly reflects the accuracy of the model. The smaller the value of MAE, the closer the predicted heart rate value is to the true value. RMSE calculates the arithmetic square root of the mean of the square of the prediction error. Compared with MAE, RMSE is more sensitive to outliers. If the model has a large prediction deviation on some specific samples, RMSE will increase significantly. Therefore, RMSE is used to measure the robustness and stability of the model. Pearson correlation coefficient is used to measure the linear correlation between the predicted heart rate sequence and the true heart rate sequence. Its value range is [-1,1], and this index does not focus on the size of the absolute error, but on the consistency of the trend.

[0073] Table 1

[0074]

[0075] From Table 1, it can be seen that in the non-contact heart rate estimation task of the V4V dataset, the detection method ST-Former proposed by the present application achieves the optimal results in the three key indicators of MAE, RMSE and R, which is significantly better than other comparative methods such as DeepPhys, PhysNet, APNET, etc., proving that the model has higher detection accuracy and robustness.

[0076] Table 2

[0077]

[0078] As can be seen from Table 2, the dual-branch architecture proposed in the present application is significantly better than the single-branch method. Relying only on appearance features lacks dynamic capture ability, and methods 2 and 3 lack spatial texture guidance, resulting in model convergence. In contrast, method 4 reduces MAE to 1.77 and improves correlation coefficient R to 0.92 by fusing the spatial constraints of the appearance branch and the physical motion information of the normalized frame difference, fully demonstrating the effectiveness of the space-time complementary strategy in non-contact heart rate estimation.

[0079] Table 3

[0080]

[0081] representing the fusion of the Transformer, representing the space-time feature fusion, representing the relative position encoding. As can be seen from Table 3, the space-time feature fusion module plays a decisive role in improving model accuracy. Compared with using only the basic Transformer , the introduction of significantly reduces MAE from 3.09 to 1.93, with a reduction of 37.5%. In addition, the relative position encoding further optimizes the model's ability to capture time series information. Finally, the indicators of the complete model are all optimal, proving that accurate position perception and efficient feature fusion are necessary conditions for high-precision heart rate estimation under this architecture.

[0082] In summary, in order to solve the problem that existing methods are difficult to balance spatial positioning and weak physiological signal capture in complex scenes, and lack long-distance time series dependence modeling ability, the present application proposes a dual-branch space-time feature extraction module with normalized frame difference and original frame as input, which extracts higher-level features before inputting into the Transformer module. Relative position encoding is added in the Transformer module to adapt to the heart rate estimation task. The combination of dual-branch 3D CNN and Transformer architecture captures high-dimensional features while modeling global time series, achieving relatively advanced performance on the V4V dataset.

Claims

1. A non-contact heart rate estimation method integrating physical priors and Transformer, characterized in that, It includes the following two parts: 1) Design a dual-branch spatiotemporal feature fusion module; A dual-branch spatiotemporal feature fusion module is designed to separate and enhance the pulse wave signal from the source; the dual-branch spatiotemporal feature fusion module is configured to separate the interference signal and enhance the pulse wave signal, including a motion branch and an appearance branch; The motion branch, acting as a change detector, takes normalized frame difference as input and removes static appearance information and the influence of light intensity by calculating the normalized frame difference, capturing color change rPPG signals caused by blood flow and displacement caused by head movement. The appearance branch takes the original video frames as input and extracts static and quasi-static facial features, including skin texture, skin color distribution, and spatial location of facial features. The features extracted by the two branches are fused to generate an enhanced spatiotemporal tubular feature. This enhanced spatiotemporal tubular feature explicitly separates and amplifies physiological signals, providing high-level semantic information for subsequent ST-Former-based heart rate estimation networks. 2) Design a heart rate estimation network based on ST-Former with a combined architecture of convolution and Transformer; We design a combined architecture of convolution and Transformer, and introduce a relative position encoding module. The features output by the dual-branch spatiotemporal feature fusion module are divided into blocks to aggregate semantics of adjacent spatiotemporal regions and reduce computational costs. Relative position encoding is introduced into the Transformer module. We design a temporal difference attention module, which uses temporal difference convolution on query Q and key K when calculating attention scores. The temporal difference convolution consists of standard 3D convolution and temporal difference terms. Guided by the temporal difference, the calculated attention score represents the similarity of change patterns, which can globally and adaptively associate and enhance all quasi-periodic rPPG signal features.

2. The non-contact heart rate estimation method integrating physical priors and Transformer as described in claim 1, characterized in that, The steps for constructing a heart rate estimation network based on ST-Former are as follows; Step 1, Data Preprocessing; A standardized preprocessing workflow was employed to extract Regions of Interest (ROIs). Physiological signals synchronized with the video were processed as labels, using arterial blood pressure (ABP) signals as the basis. The ABP signals were time-synchronized with the facial video and resampled at a frequency of 25Hz. The ABP signals were segmented into 6.4-second time windows. To extract pulse wave features for heart rate estimation from the ABP signals, pseudo-PPG signals were generated through standardization, detrending, and bandpass filtering. A detrending algorithm was used to eliminate baseline drift caused by blood pressure changes. The cutoff frequency of the bandpass filter was preset to 0.5Hz to 3.0Hz to cover the normal human heart rate range. A Fast Fourier Transform (FFT) was performed on the generated pseudo-PPG signals to calculate the power spectral density. The frequency corresponding to the maximum amplitude of the power spectral density was analyzed; this frequency represents the true heart rate value for the corresponding time window. The 160-frame, 128×128 resolution ROI sequence, after the above preprocessing, was further processed and fed into the spatiotemporal feature fusion module. Step 2, Spatiotemporal Feature Fusion Module; To separate interference at the source and enhance weak pulse wave signals, a dual-branch spatiotemporal feature fusion module is constructed at the front end of the ST-Former-based heart rate estimation network. In heart rate estimation based on illumination model, the change in light intensity reflected by the skin includes specular reflection and diffuse reflection components, and the illumination model is modeled as Equation (1). （1）； in, It's the pixel value. It is the light intensity, The component of specular reflection, The component of diffuse reflection, Noise; pixel values ​​are represented as Linear combination; the task of extracting pulse wave signals from RGB signals is transformed into decomposition. Ultimately, a pulse signal was obtained. As shown in formula (2); （2）； Inspired by DeepPhys and lighting models, normalized frame differences are used as input to the motion branch, and time is calculated. The derivative of the value is used to eliminate the constant terms contained in the pixel, as shown in formula (3); （3）； Eliminating light intensity by normalization The effect is determined by calculating the difference between the current frame and the previous frame and dividing it by the pixel intensity of the current frame, as shown in formula (4). （4）； The normalized frame difference is expressed as formula (5); （5）； Step 3: Construct a global temporal modeling module based on video Transformer; To extract the periodic patterns of physiological signals with long-range dependencies from high-dimensional spatiotemporal features, a global temporal modeling module based on video Transformer is cascaded after the dual-branch feature fusion module. The global temporal modeling module based on video Transformer consists of three parts: a block and position embedding module, a multi-layer temporal difference Transformer encoder, and a feedforward neural network. First, to adapt to the serialized input format of the Transformer, 3D convolutional projection operations are used to cut the continuous video frame features into non-overlapping blocks in the spatiotemporal dimension; the originally continuous feature maps are flattened into a series of vector sequences containing local spatiotemporal information; the feature sequences then enter an encoder composed of stacked Transformer blocks; a temporal difference multi-head self-attention mechanism (TD-MHSA) is employed; query Q, key K, and value V are generated through linear transformations, and the generation of query Q and key K introduces temporal difference convolution operations; attention maps are generated through temporal difference convolution; the output of each attention layer is further connected to a spatiotemporal feedforward network. A spatiotemporal feedforward network is used to correct local inconsistencies in features extracted by the attention mechanism, enhancing the robustness of the features. After processing through multiple Transformer blocks, the network extracts a latent vector containing global heart rate cycle information. Considering the downsampling effect during Transformer processing, a temporal upsampling unit is included to restore the temporal resolution of the original video, restoring the length of the feature sequence to the original video frame number. The features with restored temporal resolution are fed into an rPPG regression head composed of one-dimensional convolutional layers, mapping the high-dimensional features into a one-dimensional continuous pulse wave signal, and the accurate heart rate value is calculated through this continuous pulse wave signal. Step 4: Design the rPPG signal prediction and timing reconstruction module; After globally modeling the spatiotemporal features using the aforementioned video Transformer-based global temporal modeling module, a latent feature sequence containing physiological cycle information is output. This latent feature sequence is then mapped to a specific, continuous pulse wave signal. An rPPG signal prediction and temporal reconstruction module is designed, consisting of a feature dimensionality reduction decoding unit and a temporal resolution recovery unit. The module receives a high-dimensional spatiotemporal feature sequence from the video Transformer-based global temporal modeling module and uses a one-dimensional convolutional neural network structure to progressively compress the feature sequence in terms of channel dimensions. By learning the combination of feature weights for each channel, abstract semantic features are mapped to a single-channel physiological signal intensity sequence. During channel compression, normalization and a nonlinear activation function are introduced. Linear interpolation upsampling technology is used to restore the time length of the single-channel signal sequence to match the original input video frame count, generating an rPPG pulse wave signal that is strictly temporally aligned with the video frames.

3. The non-contact heart rate estimation method integrating physical priors and Transformer as described in claim 2, characterized in that, The pre-trained MTCNN model is used to detect face regions in video frames and determine the ROI range containing face regions; the detected ROI regions are cropped from the original video frames and uniformly resized to a resolution of 128×128 pixels. The pixel brightness values ​​of the cropped ROI image are normalized to the range of [0,1]. The cropped ROI sequence is sampled at a sampling rate of 25 frames / second and divided into time windows of 160 frames in length, with each window corresponding to a video duration of 6.4 seconds.

4. The non-contact heart rate estimation method integrating physical priors and Transformer as described in claim 2, characterized in that, The motion branch acts as a change detector, eliminating static background information and capturing subtle color changes caused by blood flow and non-rigid displacements caused by head movements. The appearance branch takes the original video frames as input. The appearance branch learns static and quasi-static facial features, including skin texture, basic skin color distribution, and the spatial position of facial features. The two branches extract features through 3D convolutional layers, and then concatenate or fuse features in the channel dimension. The fused high-dimensional spatiotemporal features are fed into the subsequent Transformer module.

5. The non-contact heart rate estimation method integrating physical priors and Transformer as described in claim 2, characterized in that, Training a heart rate estimation network based on ST-Former involves the following steps: Step 1, forward propagation; A Transformer network based on spatiotemporal feature fusion is used for rPPG signal prediction. First, a batch of video segment tensors with a size of [B,3,160,128,128] is input, where B represents the batch size, 160 represents the number of time frames, and 128 represents the height and width of the image. The input video is processed in parallel by two paths: the first path directly inputs the original RGB frames into the appearance branch, and uses 3D convolutional layers with kernel sizes of 1×5×5 and 1×3×3 to gradually extract texture features. While keeping the temporal dimension unchanged, the spatial size of the feature map is compressed to 1 / 8 of the original size through three spatial downsampling steps, resulting in appearance features with a size of [B,48,160,16,16]. The second path first performs normalized difference preprocessing on the input video, calculates the difference between the current frame and the previous frame and divides it by the intensity of the current frame to obtain a pure physical motion representation. Then, it inputs a symmetrical motion branch and extracts dynamic pulse information through spatiotemporal convolution with a receptive field of 3×5×5, and outputs motion features of size [B,48,160,16,16]. Finally, it concatenates the two features in the channel dimension and fuses them through 1x1 convolution to obtain a fused spatiotemporal feature map with 96 channels and a size of [B,96,160,16,16]. The fused feature maps then enter the spatiotemporal block embedding module. Through a 3D convolution operation with a kernel size and stride of [4,4,4], the spatiotemporal dimensions are jointly aggregated and downsampled, compressing the time dimension from 160 to 40 and the spatial dimension from 16x16 to 4x4, resulting in compressed features of size [B,96,40,4,4]. To adapt to the Transformer input, the time, height, and width dimensions are flattened into sequence dimensions, generating a feature sequence containing 640 tokens, with the tensor shape transformed to [B,640,96]. This sequence is then input into the module containing... In the spatiotemporal Transformer encoder with 12 stacked modules, within each module, a multi-head self-attention mechanism with relative position bias and a 3D convolutional feedforward network are used to capture long-range spatiotemporal dependencies, and the output dimension is maintained at [B, 640, 96]. Finally, the encoded feature sequence is transposed to [B, 96, 640] and enters the prediction head. The number of channels is gradually reduced to 1 through two layers of one-dimensional convolution, and the time series length is restored from 640 to the original frame number of 160 by linear interpolation upsampling. The final output is an rPPG pulse wave prediction signal with shape [B, 160]. Step 2, loss function; To guide a deep neural network to accurately regress the photoplethysmography (rPPG) signal from complex facial videos, a multi-domain hybrid loss function is designed. This function constrains the predicted signal in three domains: time, frequency, and heart rate. The network parameters are jointly updated using a backpropagation algorithm. First, for the time-domain trend constraint loss, a negative Pearson correlation coefficient is used as the time-domain constraint. The loss function maximizes the linear correlation between the predicted signal and the actual physiological signal by calculating the ratio of the product of the covariance and standard deviation of the two signals, ensuring that the predicted waveform maintains a high degree of consistency with the actual signal in terms of trend. By introducing frequency domain periodic constraint loss, the predicted rPPG signal and the real signal are first subjected to Fast Fourier Transform (FFT) to calculate their respective power spectral density (PSD). By calculating the root mean square error or cross-entropy distance between the physiological signal and noise in the frequency domain, the power of the signal generated by the model can be forced to concentrate in the main frequency band of heart rate, effectively suppressing low-amplitude non-periodic noise components. During the training phase, the heart rate numerical regression loss was explicitly introduced as a supervision signal; the time-domain prediction signal was post-processed, and the predicted heart rate value was calculated by peak detection or frequency domain maximum extraction, and the L2 norm distance between the predicted heart rate value and the true heart rate value was calculated. The final network optimization objective is defined as a weighted sum of time-domain trend loss, frequency-domain periodicity loss, and heart rate numerical loss. Through multi-task joint learning, accurate heart rate estimation results are obtained while preserving signal morphology details.

Images