A non-contact heart rate measurement method based on a double-branch feature fusion network

Abstract

The application discloses a non-contact heart rate measurement method based on a double-branch feature fusion network, and aims at the problem of rPPG space-time modeling fragmentation, and proposes a main-vice double-branch collaborative physiological signal modeling network. The main branch extracts global semantic and texture information based on an original face sequence, and combines a factorization self-attention mechanism based on non-negative matrix decomposition to jointly model in three dimensions of space-time-channel, so that the perception ability of local motion and weak pulse signals is enhanced, and high-dimensional global semantics are provided. The vice branch suppresses individual and light difference through input normalization, focuses on interframe pixel dynamic change, and designs a 3D Res-NAM module to highlight key pulse features and provide fine-grained pulse dynamics. The two branches are complementarily fused, physiological signal extraction with high precision and strong robustness is realized while keeping extremely low parameter quantity and computation quantity.

Description

Technical Field

[0001] This invention relates to a heart rate measurement method, specifically a non-contact heart rate measurement method based on a dual-branch feature fusion network. Background Technology

[0002] With the rapid development of telemedicine, intelligent health monitoring, and human-computer interaction scenarios, remote photoplethysmography (rPPG) has attracted widespread attention due to its advantages such as no need for wearing devices, strong applicability, and ease of use. However, existing rPPG methods still face significant challenges in real-world applications: First, complex lighting, skin color differences, local facial expression changes, and head movements can disrupt subtle pulse color changes, making it difficult for models to extract robust temporal physiological signals; second, currently available rPPG datasets are relatively limited in terms of movement types, pose variations, and facial expression diversity, restricting the generalization ability of deep learning models. Summary of the Invention

[0003] To address the aforementioned issues, this invention conducts a systematic study from two directions: model structure design optimization and data construction method innovation, and proposes a non-contact heart rate measurement method based on a dual-branch feature fusion network.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] A non-contact heart rate measurement method based on a dual-branch feature fusion network includes the following steps:

[0006] Step 1: Construct a primary-secondary dual-branch collaborative physiological signal modeling network:

[0007] The main-sub-branch collaborative physiological signal modeling network includes a main branch global semantic feature extraction network and a sub-branch fine-grained pulse feature extraction network, wherein:

[0008] The main branch global semantic feature extraction network includes multi-level 3D convolution and factorized self-attention module FSAM. The main branch global semantic feature extraction network takes the original face video after inter-frame difference as input, extracts global semantic features and texture information through 3D convolution, and realizes temporal-spatial-channel three-dimensional joint modeling through factorized self-attention module FSAM, which enhances the ability to perceive local motion and weak pulse signals and provides high-dimensional global semantic features.

[0009] The sub-branch fine-grained pulse feature extraction network includes multi-level 3D convolution and 3D residual normalization attention modules. The sub-branch fine-grained pulse feature extraction network takes standardized video frame difference data as input and extracts fine-grained pulse dynamic features through 3D convolution and 3D residual normalization attention modules.

[0010] The global semantic features output by the main branch global semantic feature extraction network and the fine-grained pulse dynamic features output by the sub-branch fine-grained pulse feature extraction network are interactively fused into the main branch global semantic feature extraction network through dot product. After passing through the FSAM module, the rPPG signal is output through the fully connected layer.

[0011] Step 2: Obtain the face video sequence and perform face detection and preprocessing to obtain face region video data. The specific steps are as follows:

[0012] Step 21: Use a face detection algorithm to perform face detection on the original face video data to achieve face localization and removal of irrelevant backgrounds, i.e. ;

[0013] Step 22: Data Two operations, inter-frame difference differentiation and normalization, are performed to provide the required data format sources for the main branch global semantic feature extraction network and the sub-branch fine-grained pulse feature extraction network, respectively. , The formulas for standardization and inter-frame difference are as follows:

[0014]

[0015]

[0016] in , ;

[0017] Step 23: Divide the detected face data into blocks along the time dimension to obtain the segmented face region video data. and ;

[0018] Step 3: Process the segmented face region video data and The input is a main-sub-branch collaborative physiological signal modeling network to extract global semantic features and fine-grained pulse features, outputting rPPG signals and calculating heart rate values. The specific steps are as follows:

[0019] Step 31: Extract features from the multi-level 3D convolutional layers with different hyperparameters of the main-sub-branch collaborative physiological signal modeling network, including the main branch features. Feature extraction from secondary branches ,in The number of features is 128;

[0020] Step 32: Integrate the features extracted by the fine-grained pulse feature extraction network of the sub-branch into the global semantic feature extraction network of the main branch, to obtain... ,in, 256 features were obtained;

[0021] Step 33: Analyze the fused features Applying the FSAM operation yields... ;

[0022] Step 34: After two layers of 3D convolution with adjusted feature dimensions, the features are fed into a fully connected layer for regression to obtain the final rPPG data. ;

[0023] Step 35: Calculate the heart rate value based on the rPPG data.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] This invention addresses the problem of fragmented spatiotemporal modeling in rPPG by proposing a main-sub-branch collaborative physiological signal modeling network. The main branch extracts global semantic and texture information from the original face sequence and combines it with a factorized self-attention mechanism based on non-negative matrix factorization to jointly model in three dimensions: space, time, and channel. This enhances the perception of local motion and weak pulse signals, providing high-dimensional global semantics. The sub-branch suppresses individual and illumination differences through input normalization, focuses on inter-frame pixel dynamic changes, and designs a 3D Res-NAM module to highlight key pulse features, providing fine-grained pulse dynamics. The two branches complement each other, achieving high-precision and robust physiological signal extraction while maintaining extremely low parameter and computational costs. Attached Figure Description

[0026] Figure 1 Design the overall architecture of DualPhysNet;

[0027] Figure 2 FSAM architecture diagram;

[0028] Figure 3 The overall architecture is 3D-Res-NAM;

[0029] Figure 4 For channel attention modules;

[0030] Figure 5 For spatial attention modules;

[0031] Figure 6 The YOLOv5s network structure is used.

[0032] Figure 7 Comparison of model predictions and actual values ​​on three datasets: (a) comparison of rPPG signals on UBFC, (b) comparison of rPPG signals on PURE, and (c) comparison of rPPG signals on MMPD.

[0033] Figure 8 Scatter plots of predicted and labeled values: (a) Heart rate scatter plot of the UBFC dataset, (b) Heart rate scatter plot of the PURE dataset;

[0034] Figure 9 Bland-Altman plots of predicted and labeled values: (a) Bland-Altman plot of the UBFC dataset, (b) Bland-Altman plot of the PURE dataset;

[0035] Figure 10 A comparison of the balance between model parameter quantity and accuracy. Detailed Implementation

[0036] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0037] This invention provides a non-contact heart rate measurement method based on a dual-branch feature fusion network. This method addresses the high computational complexity and real-time performance limitations of traditional video spatiotemporal feature extraction algorithms by systematically analyzing the performance bottlenecks of current heartbeat signal feature extraction techniques. To address this, this invention designs and implements a physiological detection network called DualPhysNet. In the experimental verification phase, this invention first designs a comparative experimental scheme with mainstream models, and then systematically describes the construction process of the experimental platform, including the selection of public datasets and the configuration of the hardware and software environment. Based on this, the effectiveness of the proposed method is verified through performance evaluation on multiple datasets. The specific steps are as follows:

[0038] I. Problem Analysis:

[0039] rPPG signal is a weak temporal physiological signal caused by the periodic beating of the heart. Its effective information is simultaneously distributed in multiple dimensions of space, time, and channel, and exhibits significant synergistic variation characteristics. Different facial regions, time scales, and color channels respond differently to changes in blood volume. Therefore, rPPG modeling not only needs to characterize the global semantic information of the face, but also needs to stably perceive the fine-grained pulse dynamics in local regions.

[0040] Existing research typically models video data from a single perspective, such as spatial structure, temporal dynamics, or channel response. However, when the model relies primarily on a single dimension or path, cross-dimensional feature interactions are limited, spatial, temporal, and channel information is difficult to express collaboratively within a unified framework, and the model is easily dominated by high-energy changing factors in the video.

[0041] From the perspective of input features, raw face videos contain both color micro-variation information related to rPPG and non-physiological interference components such as illumination changes, individual differences, and non-rigid motion. Single-path modeling struggles to strike a balance between interference suppression and preservation of physiological information: relying solely on raw videos easily introduces a large amount of static appearance information, while enhancing temporal variation features may weaken the ability to model global spatial semantics.

[0042] Under complex lighting and motion conditions, single-scale or single-path modeling is prone to over-responding to high-energy non-physiological changes, thereby masking weak but physiologically significant pulse signals. This phenomenon reveals the core problem in rPPG modeling at the structural level: how to alleviate the spatiotemporal modeling fragmentation while ensuring low model complexity, so that the model can simultaneously take into account global semantic understanding and fine-grained pulse dynamic perception.

[0043] Based on the above analysis, from the perspective of model structure design, introducing multi-path feature modeling and achieving effective fusion in the high-level semantic space helps to fully leverage the complementary advantages of different feature representations. To address this modeling requirement, this invention proposes a main-sub-branch collaborative physiological signal modeling network in the next section. By decoupling the main-sub-branch structure and collaboratively modeling features, combined with an attention mechanism, it achieves efficient extraction and robust modeling of weak rPPG signals while maintaining a lightweight model.

[0044] II. Model Construction:

[0045] 1. Construction of a main-sub-branch collaborative physiological signal modeling network:

[0046] This invention proposes a dual-branch collaborative physiological signal modeling network (DualPhysNet). This network is based on a 3D CNN to construct a dual-branch structure. The main branch processes the original video data, while the secondary branch processes the standardized video data. The original video frames are divided into tensors of different forms for dual-channel input, enabling full extraction of detailed features and effective suppression of noise and irrelevant variations.

[0047] (1) Main branch design

[0048] The main branch is a global semantic feature extraction network, which includes multi-level 3D convolutions and a Factorized Self-Attention Module (FSAM). First, multiple different 3D CNNs are used to extract the spatiotemporal features of the target. Then, FSAM decouples the target feature dimensions to achieve triple attention across time, space, and channels, thereby reducing computational overhead while enhancing the target signal in multiple dimensions. Finally, a subsequent 3D CNN integrates the features and outputs the rPPG signal.

[0049] (2) Sub-branch design

[0050] The secondary branch is a fine-grained pulse feature extraction network, which has the same 3D CNN architecture as the main branch except for the final 3D convolution that integrates the features. Through the normalization operation on the input data, it can significantly suppress illumination changes, skin color deviations, and background interference, thereby enhancing the weak but meaningful blood flow change signals in the time series. On this basis, an improved 3D Res Normalization-based Attention Module (3D Res-NAM) is introduced to enhance the features.

[0051] (3) Main-sub dual-branch fusion mechanism

[0052] Sub-branch features and main branch features are interactively fused through dot product, thereby providing temporal consistency and signal clarity that are difficult for the main branch to learn directly, and compensating for the problem that the main branch struggles to learn fine-grained temporal features. In the subsequent fusion stage, the sub-branch features effectively complement the high-dimensional features of the main branch, further improving the accuracy and robustness of rPPG estimation.

[0053] Overall, DualPhysNet constructs a "double-insurance system" for physiological signal extraction. The main-sub-branch structure creates complementary frequency bands, enabling the model to accurately detect heart rate information while also possessing strong generalization and robustness. The overall structure of DualPhysNet is as follows: Figure 1 As shown.

[0054] Given a batch of raw video data ,in Indicates batch size. This represents the depth of the video, i.e., the time dimension. and Indicates the height and width of a video frame. This represents the raw number of channels in the video frame; here it's 3-channel RGB data. DualPhysNet aims to extract spatiotemporal features from video and obtain the rPPG signal.

[0055] (1)

[0056] in, , For video frame rate, That is, the final target signal obtained, has .

[0057] like Figure 1 As shown, DualPhysNet employs a main-sub-branch structure, where the sub-branch contains the normalization processing of the input data:

[0058] (2)

[0059] in, This is the original data. The mean of the data. Let V be the variance of the data.

[0060] Both the main and sub-branches include a step of performing inter-frame difference analysis on the input data to highlight the changes in the data at adjacent time points; the main branch also specifically includes subsequent spatiotemporal feature combination and the final regression operation.

[0061] (3)

[0062] in, Indicates the first Frame video data.

[0063] 2. The overall data preprocessing process of the model is summarized as follows:

[0064] (1) Face detection

[0065] By using face detection algorithms, face localization and removal of irrelevant backgrounds are achieved, thereby improving the signal-to-noise ratio of the original data. .

[0066] (2) Inter-frame difference and normalization

[0067] Data Perform two operations: inter-frame differentiation and standardization, to provide the required data format sources for the main and secondary branches, respectively. , ,in , It is worth emphasizing that the DualPhysNet model does not require a detailed region of interest selection step for face data, which further highlights the end-to-end nature of this model.

[0068] (3) Data partitioning

[0069] The detected face data is divided into blocks along the time dimension. In this invention, each block of raw data is set to 160 frames in the time dimension, and any remaining portion less than 160 frames is discarded. Specifically, as follows:

[0070] The total number of frames in the original video data is The total number of frames after segmentation is :

[0071] (4)

[0072] in, This represents the floor operation, denoted as _____. , representing the total number of blocks, then the video data after being divided into blocks can be represented as:

[0073] (5)

[0074] in, For the first Block video data.

[0075] At this time, the above and All are processed in blocks, each sub-block It is a 5-dimensional tensor:

[0076] (6)

[0077] Here we might as well note the result after block processing. and for and .

[0078] 3. Model Feature Extraction:

[0079] (1) Features are extracted through multi-level 3D convolutional layers with different hyperparameters in a main-sub-branch configuration. Main branch features Feature extraction from secondary branches ,in The characteristic number is 128.

[0080] (2) Integrate the features extracted from the secondary branches into the main branch. ,in, 256 features were obtained.

[0081] (3) Features after fusion Applying a multidimensional attention FSAM operation, we obtain .

[0082] (4) After two layers of 3D convolution with adjusted feature dimensions, the features are fed into a fully connected layer for regression to obtain the final rPPG data. .

[0083] In this invention, FSAM references FactorizedPhys. The Factorized Self-Attention Module (FSAM) is a self-attention mechanism based on nonnegative matrix factorization (NMF) used to jointly compute spatial, temporal, and channel-based attention in rPPG tasks. FSAM extracts low-rank representations from voxel embeddings using matrix factorization, thereby enhancing the extraction of physiological signal-related features from video data. Specific details of the FSAM module are as follows... Figure 2 As shown.

[0084] FSAM is based on non-negative matrix factorization (NMF), which decomposes an M×N matrix V into the product of two non-negative matrices W and H. Its basic formula is:

[0085] (7)

[0086] Given a nonnegative matrix Finding nonnegative basis matrices Non-negative coefficient matrix ,in This makes equation (7) true. This decomposition has a "part-whole" interpretability and can discover the underlying structure in the data.

[0087] The preceding convolutional layers extract spatiotemporal features from the original video input to generate voxel embedding features. After that, Perform 3D convolution with a kernel of 1 and ReLU activation to ensure... The nonnegativity of .

[0088] (8)

[0089] After obtaining the initial nonnegative features, FSAM uses the time dimension of the extracted spatiotemporal features as the target matrix V, that is, mapping the time dimension τ to the row dimension M of the matrix; and uses the channel and spatial dimensions as the feature directions, that is, merging the channel κ and the spatial α×β dimensions and mapping them to the column dimension N. This mapping process is specifically shown in the following equation (9):

[0090] (9)

[0091] At this point, each row of the constructed matrix V corresponds to the feature representation at a specific time point, and each column corresponds to the feature evolution of a specific channel and spatial location. Then, NMF decomposition is performed on the constructed matrix V. This invention uses a first-order gradient optimization method to solve the NMF. The objective function used here is shown in equation (10).

[0092] (10)

[0093] Find the corresponding basis matrix sum coefficient matrix At this point, the basis matrix W contains the periodic characteristics of the physiological signal and a large number of temporal dynamic features related to the BVP signal, emphasizing the temporal distribution pattern of the physiological signal; the coefficient matrix H contains the contribution of spatial location to a specific temporal pattern and the importance weights of channel features, emphasizing the spatial distribution pattern of the physiological signal. Thus, a low-rank representation reconstructed from the decomposition is obtained:

[0094] (11)

[0095] After being reconstructed and transformed back to the original dimensions, the reconstructed matrix is ​​used as the attention weight.

[0096] (12)

[0097] Finally, the relevant features are enhanced through element-wise multiplication.

[0098] (13)

[0099] 4. Normalization-based attention mechanism design:

[0100] The model obtains brightness, color, and illumination information from the data through the main branch, thereby providing complete visual information and learning complex rPPG patterns. However, this results in a limited ability of the model to learn rPPG details. Furthermore, the NMF decomposition of a single FSAM may misjudge motion artifacts as valid signals when there are large facial movements, leading to insufficient robustness of the model.

[0101] To further enhance model robustness, this invention designs an improved 3D Res-NAM attention module based on Normalization-based Attention Module (NAM). Normalization-based Attention Module (NAM) is a normalization-based attention mechanism designed to achieve efficient attention without introducing additional parameters. The improved 3D Res-NAM modifies the original NAM, which processes 2D data, to handle 3D data and adds residual connections. Its overall architecture is as follows: Figure 3 As shown.

[0102] The core idea of ​​NAM is to use the scaling factor in batch normalization (BN) to measure the importance of weights, thereby avoiding the computational overhead of additional fully connected layers and convolutional layers in traditional attention mechanisms.

[0103] Each channel in BatchNorm has a learnable scaling factor. This is considered a measure of the channel's importance to the final prediction.

[0104] (14)

[0105] In the formula, and Small batches The mean and standard deviation of γ and β are the trainable affine transformation parameters. Combining the above ideas, the improved channel attention submodule can be expressed as Equation (15).

[0106] (15)

[0107] (16)

[0108] in, Indicates output features, Indicates input characteristics. For each channel, ... The channel normalized weights have the following specific structure: Figure 4 As shown.

[0109] For the spatial attention module, normalization is also used to evaluate pixel importance. Specifically, channel compression is first performed, that is, averaging across the channel dimensions to obtain the spatial response intensity:

[0110] (17)

[0111] in, , , and This represents the number of channels, depth, height, and width of the input feature. Then, the normalized weights for the spatial dimensions are calculated:

[0112] (18)

[0113] It is worth emphasizing that when calculating channel normalized weights, the learnable scaling parameters in the BatchNorm layer are usually initialized to 1 and retain non-zero values ​​during training. Furthermore, due to the independent optimization of BatchNorm, γ will not be entirely zero even with insufficient training. However, the calculation of spatial normalized weights is purely mathematical and has no learnable parameters. Here, the denominator represents the sum of responses of the input feature map at all voxel locations, meaning the denominator is directly related to the input features. Therefore, to prevent issues with the input features from causing the denominator to be zero, 1e-8 is added to the denominator to prevent division by zero.

[0114] In summary, the spatial attention module can be represented as:

[0115] (19)

[0116] Its detailed architecture is as follows Figure 5 As shown.

[0117] 5. Face detection algorithm:

[0118] Current non-contact heart rate estimation methods typically rely on facial video analysis. Compared to other parts of the body, the facial region has richer and more stable blood perfusion characteristics, making it more likely to show subtle skin color changes caused by heartbeats.

[0119] In the data preprocessing stage, accurate face localization is crucial for extracting effective rPPG signals from videos. Specific regions on the face, especially the forehead and cheeks, are considered the most informative areas for rPPG signals. Accurately identifying these key regions helps obtain signals from the most physiologically significant locations. Without effective face detection, the system may misidentify other irrelevant targets as faces, introducing significant noise interference and severely impacting the quality of the rPPG signal. Therefore, introducing a reliable face detection algorithm into the heart rate estimation task is of great importance for reducing background noise and improving signal stability.

[0120] YOLO (You Only Look Once) is a typical single-stage object detection framework that has been widely used in real-time object detection since its inception. YOLOv1, first proposed by Redmon et al., is based on the GoogleNet network structure. It divides the input image into a fixed-size grid, with each grid predicting both the object's class and location in a single forward propagation, thus achieving end-to-end real-time object detection for the first time. However, its detection accuracy remains somewhat insufficient in small object detection and overlapping object scenarios. Subsequently, YOLOv2 introduced anchor boxes on top of the Darknet-19 backbone network, effectively enhancing the model's adaptability to objects of different scales. YOLOv3 further adopted a deeper Darknet-53 network structure and performed object prediction on feature maps at three different scales, significantly enhancing the model's ability to detect small-scale objects. YOLOv4, building upon YOLOv3, incorporates various advanced training and architecture optimization strategies, including the CIOU loss function, PANet structure, and SAM attention module, further improving detection accuracy while maintaining real-time performance. In addition, this version introduces the CSPDarknet structure in the backbone network design and adopts the Mish activation function, which improves the training stability and generalization ability of the network.

[0121] YOLOv5, proposed by the Ultralytics team, features systematic optimizations in model structure design, training strategies, and engineering implementation. Compared to its predecessors, it achieves significant improvements in both detection accuracy and inference speed, while maintaining a smaller model size and lower inference latency, resulting in better engineering deployability. YOLOv5 offers several model versions of varying sizes, including YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x, which differ in network depth and width to accommodate different computing power and application scenario requirements. While YOLOv5x boasts high detection accuracy, its large number of model parameters and high computational cost make it less suitable for real-time deployment. YOLOv5n, while offering faster inference speed, suffers from insufficient detection accuracy in complex scenarios. Considering detection accuracy, computational complexity, and real-time requirements, YOLOv5s achieves a more ideal balance between performance and efficiency.

[0122] The overall network structure of YOLOv5s mainly consists of a CSPDarknet53 backbone network, a Feature Pyramid Network (FPN), a detection head, and a decoding module. This structure can achieve rapid target localization while ensuring high detection accuracy, making it suitable for real-time video analysis tasks. In non-contact heart rate detection applications, especially in practical deployment scenarios based on resource-constrained devices such as cameras or smart terminals, high demands are placed on the real-time performance and model size of face detection algorithms. Therefore, this invention selects YOLOv5s as the face detection model to achieve fast and accurate localization of face regions, providing reliable regional support for subsequent rPPG signal extraction and heart rate estimation.

[0123] 6. Loss function:

[0124] In deep learning, the loss function quantifies the deviation between the model's predictions and the actual values. During model training, the optimizer uses the loss value to iteratively update the network parameters, thereby gradually narrowing the gap between the predicted output and the actual data. Therefore, the selection of the loss function is crucial to the training effect of the deep learning model. For the rPPG signal extraction task, the core optimization objective is to make the signal output by the model highly consistent with the reference physiological signal in terms of periodicity and key waveform characteristics. To this end, this invention uses the negative Pearson correlation coefficient (NegPeaLoss) as the loss function. This function maximizes the consistency of the signal waveform trend while minimizing the localization error of important feature points (such as peaks). Its mathematical expression is as follows:

[0125] (20)

[0126] in, The predicted BVP signal output by the convolutional neural network model. One signal value; The first in the real signal One signal value; This represents the signal duration.

[0127] III. Experimental Setup and Result Analysis:

[0128] 1. Dataset selection:

[0129] The non-contact heart rate detection experiments were conducted on three publicly available datasets, which are briefly described below.

[0130] UBFC-rPPG: This study comprises 42 videos. During recording, the subjects sat in front of the camera, remaining relatively still. These videos were captured using a Logitech C920 HD Pro webcam at 30fps, a resolution of 640x480, and in uncompressed 8-bit RGB format. The experiment was conducted indoors with varying levels of sunlight and ambient lighting. A CMS50E transmissive pulse oximeter was used to acquire real-world electrocardiogram (ECG) data, including ECG waveforms and heart rate data. For ease of description, UBFC-rPPG will be referred to as UBFC below.

[0131] PURE: PURE consists of 60 one-minute videos, with 10 participants engaging in 6 activities. These six activities are as follows:

[0132] (1) Stability: The subject should sit still and look directly at the camera, avoiding head movement.

[0133] (2) Speaking: Simulate a video sequence and ask the subject to speak while avoiding additional head movements. This setup is equivalent to a video conference scenario in a real robotic application.

[0134] (3) Slow Movement: These sequences involved head movements parallel to the camera plane. Therefore, images captured by the camera were displayed on a screen and shown to the subject. A moving rectangle the same size as the face was added to the image, and the subject was asked to keep their face within the rectangle. The rectangle moved horizontally at a controlled speed and in a preset pattern, ensuring that the sequences were repeatable for all individuals. The average speed was 7% of the face height per second, and the average face height was 100 pixels.

[0135] (4) Fast movement: This dataset is set up the same as the slow translation, except that the target moves at twice the speed.

[0136] (5) Small head rotation: This setup involves placing different targets around the camera at a distance of 35 cm. Subjects are asked to look at these targets in a pre-set order. They need to move not only their eyes but also adjust their head position. The head rotation angle is approximately 20° depending on the distance between the camera and the subject.

[0137] (6) Larger head rotation: The settings for these sequences are the same as those for small angle rotation, but the target is placed 70 cm around the camera, resulting in an average head angle of 35°.

[0138] These videos were captured using an eco274CVGE camera at a frame rate of 30Hz, a crop resolution of 640×480 pixels, and a 4.8mm lens. Reference data were captured in parallel using a finger-clip pulse oximeter (pulox CMS50E), which provides pulse rate waveforms and SpO2 readings at a sampling rate of 60Hz.

[0139] MMPD: This dataset comprises 660 one-minute videos from 33 participants with varying skin types, categorized into four groups. Data collection involved four different light intensities and three different light source types: low-power LED lights, medium-power incandescent lights, high-power LED lights, and natural light. Participants performed four different activities under these lighting conditions: static, head rotation, speaking, and walking. In addition to skin color, the dataset includes multiple labels such as gender, whether or not glasses were worn, whether or not hair was present, and whether or not makeup was worn. All videos were recorded using a Samsung Galaxy S22 Ultra at a resolution of 320x240 and a frame rate of 30fps, with PPG signals recorded using an HKG-07C+ pulse oximeter at the gold standard. Notably, MMPD exists in two versions: compressed mini-MMPD and uncompressed MMPD; this invention uses mini-MMPD for the experiments.

[0140] 2. Model parameter settings and experimental environment:

[0141] Training the model requires continuous adjustment of the parameters of each module to achieve the best prediction performance for the overall structure. After multiple experiments, the final parameters are shown in Table 1.

[0142] Table 1 Experimental parameters

[0143]

[0144] To ensure the fairness and validity of the comparative experimental results, the specific hyperparameter configurations for each model were based on the optimal results in the paper. Apart from this, the hardware and environment configurations for all experiments in this invention remained consistent, as shown in Table 2.

[0145] Table 2 Hardware and Environment Configuration

[0146]

[0147] 3. Model performance analysis:

[0148] Table 3 compares the cross-dataset performance of DualPhysNet with various traditional methods and state-of-the-art models. For the first three metrics, lower values ​​are better, and the Pearson coefficient closest to one is better. Underlined values ​​represent the best-performing baseline model; bold values ​​represent the results of the proposed DualPhysNet model; Gains represent the improvement in performance between the proposed model and the best baseline model, with green indicating an improvement in performance and red indicating a decrease. It is important to note that the first seven methods in Table 3 are traditional algorithms, while the rest are deep learning algorithms.

[0149] Table 3 Model Performance Comparison

[0150]

[0151] As can be seen from Table 3, the DualPhysNet proposed in this invention shows significant advantages in several key metrics, demonstrating excellent performance in cross-dataset generalization ability and computational efficiency.

[0152] Specifically, in a cross-dataset evaluation setting where training is performed on the UBFC dataset and testing is performed on the PURE dataset, Ours, or DualPhysNet, achieved a performance of MAE=0.46 bpm, RMSE=1.35 bpm, and MAPE=0.75%, reaching the best performance among all compared methods. Compared to traditional methods, DualPhysNet shows significant advantages, for example, CHROM's MAE is 4.06 bpm; compared to mainstream deep learning methods, DualPhysNet also achieves better results, such as PhysNet and RhythmFormer's MAEs of 0.96 bpm and 0.81 bpm, respectively. The aforementioned results indicate that DualPhysNet can extract heart rate signals more accurately and maintain strong robustness against noise interference and motion artifacts in complex scenes. In terms of correlation metrics, DualPhysNet achieved a Pearson correlation coefficient of ρ=0.99, ranking alongside RhythmMamba as the best, and significantly higher than traditional methods (such as POS with ρ=0.92) and some deep models (such as DeepPhys with ρ=0.62). This result further validates the model's advantages in terms of consistency and stability in heart rate estimation.

[0153] Training on PURE and testing on UBFC also used four evaluation metrics. Only RMSE was reduced by 2.55% compared to RhythmMamba, but the overall error metric was also the best. Compared to the second-best model, MAE, MAPE and ρ were improved by 47.94%, 72.85% and 0.70% respectively.

[0154] Due to the presence of multiple interference factors on the MMPD dataset, such as intense motion, lighting changes, jewelry occlusion, and makeup, most methods have a p-value below 0.3 (e.g., PhysNet's p-value is 0.20, and TS-CAN's p-value is 0.24). In contrast, DualPhysNet achieves a p-value of 0.39, an 8.3% improvement over benchmark methods (e.g., FactorizePhys' p-value is 0.36), indicating its stronger adaptability to extreme conditions. Although DualPhysNet's MAE and RMSE metrics are slightly lower than the best performance on MMPD, they still outperform most other methods.

[0155] Considering the overall performance of the model, the DualPhysNet proposed in this invention outperforms existing methods in terms of model accuracy and robustness.

[0156] To more intuitively understand the effectiveness of the DualPhysNet designed in this invention in extracting regular changes in heart rate information, a direct comparison was made between the predicted rPPG signal results and the actual BVP label signals of some samples from three datasets. The comparison results are as follows: Figure 7 As shown.

[0157] Figure 7 The orange curve represents the rPPG signal predicted by the model, and the green curve represents the corresponding tag signal. To further analyze the model's signal recovery capability, this invention conducts a qualitative comparison based on waveform fitting degree, frequency consistency, phase synchronization, and amplitude fidelity.

[0158] Performance on the UBFC dataset is as follows Figure 7 As shown in (a), the model-predicted signal and the real label signal are highly consistent in overall waveform morphology, signal period, and amplitude variation, demonstrating a strong rPPG rhythm modeling capability. Throughout the 1-6 second time period, the model output is almost completely synchronized with the real signal, indicating that the model can stably extract signal features highly correlated with the real rPPG on this dataset, exhibiting high accuracy and robustness.

[0159] The results on the PURE dataset are as follows Figure 7 As shown in (b), the model can still reproduce the real rPPG waveform very well. Compared to UBFC, the PURE dataset contains more motion interference and illumination variations, but it can be observed from the figure that although there are slight phase shifts and amplitude compressions in some time periods (such as around the 1st second), the model can still effectively maintain the consistency of the signal period and trend, indicating that it has a certain degree of anti-interference ability. The predicted waveform and the label can still maintain good synchronization and rhythm in most time periods, indicating that the model has good generalization performance.

[0160] The MMPD dataset contains more real-world scene disturbances, such as pose changes and occlusion. On the more challenging MMPD dataset, such as... Figure 7 (c) The model exhibits a certain performance degradation. Although the predicted signal remains phase-consistent and periodically synchronized with the true label in the first 2 seconds and the 5-6 second interval, the amplitude of the predicted signal is slightly lower and there is some waveform compression between 2 and 5 seconds, with a slight phase lag in some periods. This is due to complex head movements, occlusion, or strong light interference in the data, which affects the model's feature extraction capabilities. Nevertheless, the model can still capture the rhythmicity and some key fluctuations of the signal on this dataset, demonstrating a certain degree of stability and modeling ability.

[0161] In summary, the DualPhysNet proposed in this invention demonstrates superior rPPG signal recovery capabilities on three datasets—UBFC, PURE, and MMPD—which cover different features. Specifically, in static or low-interference scenarios, the model exhibits highly stable performance, accurately extracting heart rate-related signals with consistent results. Even in challenging high-interference or complex dynamic environments, such as the motion artifacts and lighting variations present in the MMPD dataset, the model maintains the basic waveform structure and effectively suppresses noise, highlighting its robustness and anti-interference capabilities. These combined results demonstrate that the model not only exhibits consistent adaptability across different scenarios but also provides reliable technical support for diverse conditions in practical applications, showcasing its strong cross-scenario generalization potential.

[0162] also, Figure 8 The scatter plot presents the relationship between the model's heart rate predictions (HR-predict) and true label values ​​(HR-gt) on the UBFC and PURE datasets. Each point in the plot represents the prediction result of a sample. The horizontal axis represents the true heart rate value, the vertical axis represents the heart rate value predicted by the DualPhysNet model, and the dashed line represents the ideal prediction line (i.e., y=x), which indicates the ideal situation where the predicted value is completely consistent with the true value. This format allows for an intuitive evaluation of the model's prediction accuracy and consistency across different heart rate ranges.

[0163] As can be clearly observed from the figure, all data points are closely distributed near the regression line with a slope of 1. This distribution feature fully verifies that the method of the present invention can accurately capture the pattern of heart rate changes, and the predicted results have a good linear matching degree with the true values.

[0164] exist Figure 8In the UBFC dataset results (a), most scatter points are highly concentrated near the ideal diagonal, indicating that the model has good heart rate regression ability under static conditions. The predicted values ​​are highly consistent with the true values ​​across the entire heart rate range (approximately 60-140 bpm) without any obvious systematic shift, demonstrating the model's strong accuracy and stability on this dataset. Only a small number of sample points deviate slightly from the ideal line, possibly due to individual samples containing abnormal noise or slight motion interference, but the overall error range is small.

[0165] Figure 8 (b) shows the prediction results for the PURE dataset. In comparison, the scatter distribution is slightly more discrete, especially in the high heart rate range (e.g., >120 bpm), where some points show a slight deviation, but overall it still closely follows the diagonal distribution. This indicates that the model can still accurately estimate heart rate values ​​in natural scenarios such as mild movement or changes in lighting, demonstrating good generalization ability and a certain degree of robustness against interference.

[0166] In summary, Figure 8 The scatter plot results show that the proposed model exhibits good correlation and regression accuracy across different datasets. The model performs best in static or low-dynamic environments, while in more complex dynamic scenarios, although slight performance fluctuations occur, it still effectively maintains consistency in heart rate estimation. These results further visually validate the model's superior performance on the aforementioned quantitative indicators and demonstrate its strong applicability and stability in practical applications.

[0167] The Bland-Altman plot is a visualization tool used to evaluate the consistency between two continuous variable detection methods. The plot is presented as a two-dimensional scatter plot, where each data point corresponds to one detection result: the horizontal axis represents the arithmetic mean of the label and predicted values, the vertical axis reflects the difference between the two, and the two red dashed lines represent the 95% confidence interval for consistency. ,in This represents the average difference between the labeled heart rate value and the predicted heart rate value. This represents the standard deviation between the labeled heart rate value and the predicted heart rate value. When the vast majority of data points fall within this range, the predicted values ​​obtained by the current method can be considered to have good consistency with the labeled data.

[0168] Figure 9This is a Bland-Altman analysis plot showing the relationship between the heart rate values ​​predicted by the model of this invention and the labeled heart rate values ​​on the UBFC and PURE datasets. The solid black line in the plot represents the mean bias, and the dashed red line represents ±1.96 times the standard deviation (i.e., the 95% confidence interval, Limits of Agreement, LoA).

[0169] Figure 9 (a) shows the results on the UBFC dataset. The data points are mainly concentrated near the mean deviation line. The model's prediction error on this dataset is generally small, with a mean deviation of -0.4 bpm, indicating that the model does not have a statistically significant tendency to systematically overestimate or underestimate heart rate. The errors of the vast majority of samples fall within ±1.96 standard deviations, ranging from -6.1 bpm to +5.4 bpm, reflecting good predictive consistency and stability of the model on this dataset. Only two outliers are located outside the confidence interval boundaries, suggesting that individual samples may cause prediction deviations under specific conditions (such as strong noise or poor lighting), but the overall distribution remains concentrated and symmetrical.

[0170] Figure 9 (b) shows that the confidence intervals for the predicted and labeled heart rate values ​​on the PURE dataset range from -4.0 bpm to 3.5 bpm, with a mean deviation of -0.3 bpm. This further demonstrates that the model predictions show no systematic bias, and the prediction errors are essentially symmetrically distributed around 0. It is evident that the vast majority of the model's results fall within the 95% confidence interval, and the overall data points remain tightly distributed without any significant trend deviation, validating the model's reliability on this natural scene dataset.

[0171] pass Figure 9 The Bland-Altman analysis further validates the consistency and error stability of the heart rate prediction model proposed in this invention across different datasets. Whether in the predominantly static UBFC dataset or the PURE dataset containing mild motion interference, the model's prediction error exhibits a small mean bias and a narrow 95% confidence interval, with the vast majority of sample predictions falling within acceptable error ranges. This demonstrates that the model proposed in this invention not only possesses good accuracy and correlation but also exhibits strong consistency and robustness at the sample level.

[0172] 4. Ablation experiment analysis:

[0173] To verify the effectiveness of the model architecture and components proposed in this invention, a detailed ablation analysis was performed on the model of this invention.

[0174] Table 4 Ablation Experiment Results on UBFC Dataset

[0175]

[0176] Table 5 Ablation Experiment Results of PURE Dataset

[0177]

[0178] Table 6 Ablation Experiment Results of MMPD Dataset

[0179]

[0180] The test results on the three datasets all showed similar conclusions. The first sub-model, without any additional operations, performed the worst. Removing the 3D Res-NAM module from the first sub-model resulted in a decrease in the RSME metric on the UBFC dataset, while improvements were observed in other metrics and on the other two datasets. Regarding the performance on the UBFC dataset, this invention understands that the dataset requires the subject's head to remain as stable as possible, without excessive rigid movement. Therefore, the channel and spatial feature distribution of the data in this dataset is balanced, eliminating the need for excessive intervention from the 3DRes-NAM module. After removing the sub-branch from the first sub-model and directly applying 3D Res-NAM to the model backbone, improvements were observed in all evaluation metrics. Among these, the introduction of the 3D Res-NAM module and the sub-branch structure contributed most significantly to the performance improvement, resulting in better results across all metrics. The above ablation experiment results validate the effectiveness of the proposed model architecture and its key component design.

[0181] 5. Model Complexity Analysis:

[0182] Since the ultimate goal of non-contact heart rate detection technology is to be implemented in actual devices, the application scenarios of this invention limit the requirements for hardware devices. Therefore, it is essential to perform complexity analysis on the constructed model. The following section focuses on analyzing the complexity of the model proposed in this invention and comparing it with several deep learning models.

[0183] Specifically, model complexity is measured using two core but complementary metrics: floating-point operations (FLOPs) and parameters. FLOPs represent the total number of floating-point operations required for one forward propagation, reflecting dynamic complexity and determining runtime computational resource requirements. Parameters represent the total number of learnable parameters in the model, reflecting static complexity and determining model size and storage costs. Combining these two metrics provides a complete picture of the model's computational efficiency and resource requirements.

[0184] Table 7 Model Performance Comparison

[0185]

[0186] Looking at the three indicators, the methods in Table 7 show significant differences in resource consumption and error: the number of parameters ranges from 0.05M to 7.38M, FLOPs from 12.05G to 224.38G, and MAE from 0.46 to 12.98. Early and larger models (such as DeepPhys, PhysNet, and PhysFormer) generally have higher computational costs and more parameters, but their error is not necessarily superior; some lightweight methods (such as FactorizePhys) are computationally less expensive, but their accuracy suffers a significant loss.

[0187] In terms of accuracy, Ours, also known as DualPhysNet, achieves a MAE of 0.46, ranking first among all methods. It reduces the error by 41% and 43% compared to the second-best methods, PhysMamba (0.78) and RhythmMamba (0.81), respectively. Compared to the median baseline (MAE=3.11), the error is reduced by 85%. Therefore, within the current comparison range, Ours can be considered to have achieved the best overall accuracy across the entire table with a significant advantage.

[0188] In terms of resource consumption, DualPhysNet has 0.22M parameters and 23.96G FLOPs, ranking second smallest in the table, only larger than FactorizePhys (0.05M and 12.05G). Compared to common strong baselines, DualPhysNet reduces parameters and computation by 70% and 87% respectively compared to PhysMamba; and by 96% and 54% respectively compared to RhythmMamba. Compared to the median size of all baselines (2.23M parameters and 163.54G FLOPs), DualPhysNet reduces these by 90% and 85% respectively, demonstrating a stable advantage in lightweight design.

[0189] From the perspective of Pareto optimality, DualPhysNet dominates 8 out of 9 baselines across the three dimensions (smaller parameters and less computation, while achieving higher accuracy), meaning it achieves superior prediction accuracy with lower or comparable resource consumption. The only method not fully dominated is FactorizePhys, which has the smallest parameter size and FLOPs (0.05M / 12.05G), but a significantly higher prediction error (MAE=1.51). Therefore, DualPhysNet and FactorizePhys together constitute the Pareto frontier in the current comparative experiments. However, DualPhysNet significantly reduces prediction error with only a moderate increase in model size, resulting in a significant performance gain compared to most real-world deployment scenarios.

[0190] The following is a visual comparison of the balance between the number of model parameters and the model accuracy, so as to more intuitively understand the effect of each model on the balance between the number of parameters and the detection accuracy.

[0191] Figure 10 This paper presents a comparison of the balance between parameter count (million) and heart rate estimation accuracy (mean absolute error, MAE, bpm) among different rPPG models. As shown in the figure, Ours, also known as DualPhysNet, maintains the best accuracy of 0.46 bpm with an extremely low parameter count of 0.22M, significantly outperforming other methods. This indicates that this model, while lightweight in design, can still effectively extract key features of the rPPG signal, achieving efficient and high-accuracy heart rate estimation. PhysMamba (0.73M, 0.78 bpm) and RhythmFormer (3.33M, 0.97 bpm) also exhibit high accuracy, but their parameter counts are 3.3 times and 15.1 times higher than Ours, respectively, indicating relatively low computational efficiency. FactorizePhys (0.05M, 1.51 bpm), although with an extremely low parameter count, has a significantly higher error than DualPhysNet, suggesting potential deficiencies in feature representation capabilities. Furthermore, through… Figure 8 It can be concluded that higher parameter count does not necessarily equate to higher accuracy in deep learning models. For example, DeepPhys (2.23M, 5.75 bpm) and PhysNet (0.77M, 7.61 bpm) have significantly higher errors than the lightweight DualPhysNet and PhysMamba, indicating that simply increasing model complexity does not guarantee performance improvement; optimizing the model architecture is more crucial. Regarding model efficiency and practicality, Ours, with only 30% of PhysMamba's and 6.6% of RhythmFormer's parameter count (0.22M), maintains the lowest error, demonstrating its applicability in computationally limited scenarios. In contrast, while RhythmFormer's accuracy is close to DualPhysNet, its 3.33M parameter count may limit its deployment efficiency in real-time applications.

[0192] IV. Summary:

[0193] This invention proposes a main-sub-branch collaborative physiological signal modeling network architecture. Each branch focuses on different feature dimensions in the image, and together they achieve high-precision rPPG signal extraction through information fusion. The model mainly consists of four parts: the main branch, the FSAM module, the sub-branch, and the 3D Res-NAM module. The main branch takes the original face frame sequence as input, preserving the complete visual information of the facial region, such as skin color, brightness, light and shadow structure, and texture. Its feature extraction network learns the complete image context and facial structure, and combines with the FSAM module to obtain complex spatial-temporal-channel changes in the image, capturing high-dimensional visual features related to rPPG, such as subtle color changes caused by blood flow, regional illumination differences, and local facial movements. The sub-branch performs normalization processing on the input image, weakening static visual differences and global illumination effects at the input level, suppressing illumination changes and individual differences, improving the significance and stability of the pulse signal, and making it more focused on pixel-level dynamic changes between frames—the core source of the rPPG signal. The secondary branch, through a constructed 3D Res-NAM feature fusion module, integrates the captured detailed features into the main branch, enabling the main branch to model high-dimensional spatial semantics, while the secondary branch strengthens the underlying temporal pulsation signal. This dual-branch network architecture complements each other in terms of information dimension, achieving spatiotemporal, physiological, and semantic fusion modeling through a feature fusion mechanism, thereby effectively improving the accuracy and robustness of rPPG signal extraction. Experiments on three public datasets and comparisons with other methods validated the reliability of the proposed model. Ablation experiments on different components confirmed the effectiveness of each module. Finally, comparisons were made with existing models in terms of the balance between parameter count, FLOPs, and accuracy, showing that the proposed model achieves the lowest MAE across the entire table while maintaining extremely low parameter count and computational cost, and achieves full-dimensional dominance over 8 / 9 strong baselines. In terms of the overall performance of "computational efficiency-resource requirements-model accuracy," the proposed DualPhysNet model represents the optimal balance point in the current comparison, making it highly suitable for deployment in resource-constrained applications with stringent accuracy requirements. In contrast, existing methods have failed to improve accuracy as the number of parameters increases, while some lightweight models suffer from high errors due to insufficient expressive power.

Claims

1. A non-contact heart rate measurement method based on a dual-branch feature fusion network, characterized in that... The method includes the following steps: Step 1: Construct a primary-secondary dual-branch collaborative physiological signal modeling network: The main-sub-branch collaborative physiological signal modeling network includes a main branch global semantic feature extraction network and a sub-branch fine-grained pulse feature extraction network, wherein: The main branch global semantic feature extraction network includes multi-level 3D convolution and factorized self-attention module FSAM. The main branch global semantic feature extraction network takes the original face video after inter-frame difference as input, extracts global semantic features and texture information through 3D convolution, and realizes temporal-spatial-channel three-dimensional joint modeling through factorized self-attention module FSAM, which enhances the ability to perceive local motion and weak pulse signals and provides high-dimensional global semantic features. The sub-branch fine-grained pulse feature extraction network includes multi-level 3D convolution and 3D residual normalization attention modules. The sub-branch fine-grained pulse feature extraction network takes standardized video frame difference data as input and extracts fine-grained pulse dynamic features through 3D convolution and 3D residual normalization attention modules. The global semantic features output by the main branch global semantic feature extraction network and the fine-grained pulse dynamic features output by the sub-branch fine-grained pulse feature extraction network are interactively fused into the main branch global semantic feature extraction network through dot product. After passing through the FSAM module, the rPPG signal is output through the fully connected layer. Step 2: Obtain the face video sequence and perform face detection and preprocessing to obtain face region video data; Step 3: Input the face region video data into the main-sub-branch collaborative physiological signal modeling network to extract global semantic features and fine-grained pulse features, output rPPG signal and calculate heart rate value.

2. The non-contact heart rate measurement method based on a dual-branch feature fusion network according to claim 1, characterized in that... The specific steps of step 2 are as follows: Step 21: Use a face detection algorithm to perform face detection on the original face video data to achieve face localization and removal of irrelevant backgrounds, i.e. ; Step 22: Data Two operations, inter-frame difference differentiation and normalization, are performed to provide the required data format sources for the main branch global semantic feature extraction network and the sub-branch fine-grained pulse feature extraction network, respectively. , ; Step 23: Divide the detected face data into blocks along the time dimension to obtain the segmented face region video data. and .

3. The non-contact heart rate measurement method based on a dual-branch feature fusion network according to claim 1, characterized in that... The normalization and inter-frame difference formulas are as follows: in, , , Indicates batch size. This represents the depth of the video, i.e., the time dimension. and Indicates the height and width of a video frame. This represents the original number of channels in a video frame. This is the original data. The mean of the data. The variance of the data, Indicates the first Frame video data, This represents the number of video frames.

4. The non-contact heart rate measurement method based on a dual-branch feature fusion network according to claim 1, characterized in that... The specific steps of step 3 are as follows: Step 31: Extract features from the multi-level 3D convolutional layers with different hyperparameters of the main-sub-branch collaborative physiological signal modeling network, including the main branch features. Feature extraction from secondary branches ,in The number of features is 128; Step 32: Integrate the features extracted by the fine-grained pulse feature extraction network of the sub-branch into the global semantic feature extraction network of the main branch, to obtain... ,in, 256 features were obtained; Step 33: Analyze the fused features Applying the FSAM operation yields... ; Step 34: After two layers of 3D convolution with adjusted feature dimensions, the features are fed into a fully connected layer for regression to obtain the final rPPG data. ; Step 35: Calculate the heart rate value based on the rPPG data.

Images