Remote heart rate estimation method and system based on multi-scale fusion

Abstract

The invention discloses a remote heart rate estimation method and system based on multi-scale fusion, and relates to the technical field of non-contact physiological signal detection.The method comprises the steps that a space-time diagram of a face video is calculated; an Embedding operation is carried out on the space-time diagram on the channel dimension; a spatial attention mechanism is adopted, spatial context information is embedded in each frame of the space-time diagram, and spatial feature enhancement is achieved; for the space-time diagram, feature fusion is carried out under different time scales, the fusion features under the large time scale are gradually mapped into the fusion features under the small time scale in sequence, and finally the space-time diagram after time fusion is obtained; a time attention mechanism is adopted, time context information is embedded in each spatial position of the time-space diagram, and time feature enhancement is achieved; and finally, carrying out regression prediction on the space-time diagram, and outputting a predicted rPPG signal. According to the method, the inherent periodicity of the rPPG signal is fully utilized for remote heart rate estimation, and the accuracy of rPPG signal estimation is improved.

Description

Technical Field

[0001] This application relates to the field of non-contact physiological signal detection technology, and in particular to a remote heart rate estimation method and system based on multi-scale fusion. Background Technology

[0002] Historically, remote physiological signal detection has been mainly divided into traditional manual calculation methods based on mathematical formulas, model methods based on prior knowledge such as skin color dynamics, and the recently emerging methods based on deep learning architectures such as CNN convolutional neural networks and Transformer networks.

[0003] Traditional manual remote physiological signal measurement methods primarily employ mathematical techniques to suppress motion artifacts and noise in facial videos. These methods can be broadly categorized into two main types: blind source separation (BSS)-based methods and model-based methods. BSS-based methods treat the facial video signal as a mixture of rPPG signals and various noise sources, such as changes in illumination and head movements. Techniques like principal component analysis (PCA) and independent component analysis (ICA) are applied to separate the rPPG signal from these confounding factors. However, the effectiveness of BSS-based methods is often limited by their reliance on statistical assumptions, making them susceptible to noise in real-world scenarios. In contrast, model-based methods leverage prior knowledge of skin color dynamics to guide the separation process, achieving higher accuracy than BSS-based methods by incorporating domain-specific constraints. Despite these advancements, traditional BSS and model-based methods have limited adaptability to large-scale data and experience performance degradation in complex, unconstrained environments. The recent emergence of deep learning (DL) has revolutionized remote physiological measurement methods by providing superior modeling capabilities. HR-CNN was the first CNN to introduce HR estimation, employing a two-stage pipeline consisting of a feature extractor and an HR estimator. Subsequently, DeepPhys proposed using normalized difference frames to extract rPPG cues, inspiring a series of subsequent improvements such as TS-CAN. Meanwhile, end-to-end architectures have become increasingly popular, allowing direct estimation of rPPG from raw video frames, such as PhysNet, PhysFormer, and its improved version PhysFormer++, which leverage advanced deep learning architectures to enhance rPPG signal extraction. On the other hand, spatiotemporal signal maps (STmaps) based on facial ROIs have been developed to mitigate interference from non-skin regions. This representation has been widely adopted by recent state-of-the-art models, including numerous models such as Dual-GAN and Cluster-Phys, demonstrating its effectiveness in improving robustness. While this research primarily focuses on supervised learning-based methods, it is important to acknowledge the growing interest in self-supervised and unsupervised learning for remote physiological measurements. Several prominent works have explored these paradigms, including Contrast-Phys and its enhanced version ContrastPhys+, as well as recent work by Li and Yue et al. These methods have made significant progress, but their exploration of the periodicity of rPPG signals is limited, and they cannot fully utilize the inherent periodicity of rPPG signals to achieve more robust signal prediction. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, this application provides a remote heart rate estimation method based on multi-scale fusion, which makes full use of the inherent periodicity of rPPG signals for remote heart rate estimation and improves the accuracy of rPPG signal estimation.

[0005] To achieve the above objectives, this application adopts the following technical solution, including: A remote heart rate estimation method based on multi-scale fusion includes the following steps: S1, Input a face video, calculate the spatiotemporal graph of the face video. Spacetime diagram Including channel dimension, spatial dimension, and time dimension; S2, regarding the spacetime diagram Perform embedding operations along the channel dimension to obtain the embedded spatiotemporal graph. ; S3 employs a spatial attention mechanism in the spatiotemporal graph. Spatial context information is embedded in each frame to obtain a spatiotemporal graph with enhanced spatial features. ; S4, for spacetime graphs Feature fusion is performed at different time scales to obtain fused features at different time scales; then, the fused features at larger time scales are successively mapped to the fused features at smaller time scales; finally, a spatiotemporal map after time fusion is obtained. ; S5 employs a time-attention mechanism in the spatiotemporal graph. Temporal context information is embedded in each spatial location to obtain a spatiotemporal map with enhanced temporal features. ; S6, for spacetime diagrams Perform regression prediction and output the predicted rPPG signal.

[0006] Preferably, during the model training process, the mean square error is calculated for the predicted rPPG signal and the real rPPG signal at different time scales to obtain the loss at different time scales, and the losses at different time scales are added together as the total loss of the model.

[0007] Preferably, in step S1, the input face video is processed according to each... A frame is a segment, dividing the video into multiple segments, meaning each segment receives... Each frame of video image is analyzed using a face detection tool to extract facial feature points, and the facial features are defined based on these points. n Each ROI will n Combining individual ROIs yields a combination of single ROIs and combinations of different ROIs. N A combination, N= 2 n -1; For each combination, calculate the average pixel value of all ROIs in the combination across all color channels, with a total of color channels.C One; thus obtaining the spacetime diagram. Spacetime diagram The channel dimension is C Spatial dimension is N The time dimension is F .

[0008] Preferably, in step S3, in the spatiotemporal diagram The f Spatial context information is embedded in the frame, as shown below: , , ; ; ; in, These are learnable parameters; For spacetime diagram The Middle f Feature map of the frame f = 1,..., F ,common Frame; FFN stands for Feedforward Neural Network; LN represents Layer Normalization; softmax represents the activation function; d For dimensions; The first after spatial feature enhancement f Feature map of a frame; By concatenating the spatially enhanced feature maps of all frames, we obtain the spatiotemporal map with enhanced spatial features. .

[0009] Preferably, in step S4, the spatiotemporal graph is... ,exist M Feature fusion was performed at each time scale to obtain... M Fusion features at each time scale; Fusion characteristics at various time scales The calculation method is as follows: ; in, For the first The window size for each time scale, , For spacetime diagram The length of time; Linear mapping is used to progressively map the fused features at larger time scales (coarse-scale features) to the fused features at smaller time scales (fine-scale features). The mapping method is as follows: ; in, ; This represents a linear mapping from a coarse scale to a fine scale; Ultimately As a spacetime diagram after time fusion Output.

[0010] Preferably, in the spatiotemporal diagram The n Temporal context information is embedded in each spatial location, as shown below: , , ; ; ; in, These are learnable parameters; For spacetime diagram The Middle n Feature map of a spatial location n= 1,..., N ,common N Each spatial location; FFN is a feedforward neural network; LN represents layer normalization; softmax represents the activation function; d For dimensions; For the time feature enhancement of the first n Feature map of each spatial location; By stitching together the feature maps with enhanced temporal features of all spatial locations, a spatiotemporal map with enhanced temporal features is obtained. .

[0011] Preferably, in step S6, the spatiotemporal diagram is... The regression prediction is performed as follows: spatiotemporal diagram First, average pooling is performed in the spatial dimension to convert the spatial dimension into 1 dimension. Then, through a linear layer, the channel dimension is mapped to 1 dimension, thereby realizing the mapping from the spatiotemporal map to the predicted rPPG signal.

[0012] This application also provides a remote heart rate estimation system based on multi-scale fusion, applied to the aforementioned remote heart rate estimation method based on multi-scale fusion. The system model includes: The spatiotemporal graph extraction module is used to calculate the spatiotemporal graph of face videos. ; The Embedding module is used for spatiotemporal graphs. Perform embedding operations along the channel dimension to obtain the embedded spatiotemporal graph. ; The Spatial Transformer module employs a spatial attention mechanism for spatiotemporal graphs. Spatial context information is embedded in each frame to obtain a spatiotemporal graph with enhanced spatial features. ; Multi-scale temporal fusion module for spatiotemporal graphs This is used to perform feature fusion at different time scales, obtaining fused features at different time scales. The fused features at larger time scales are then progressively mapped onto the fused features at smaller time scales, ultimately resulting in a spatiotemporal map after time fusion. ; The Time Transformer module employs a time attention mechanism for spatiotemporal graphs. Temporal context information is embedded in each spatial location to obtain a spatiotemporal map with enhanced temporal features. ; The regression prediction module is used for spatiotemporal plots. Perform regression prediction and output the predicted rPPG signal; The hierarchical loss module is used only during model training to calculate the total model loss. The system model is trained using a sample set, which includes face videos and corresponding labels, i.e., real rPPG signals. During the training process, the hierarchical loss module calculates the mean square error of the predicted rPPG signal and the real rPPG signal at different time scales to obtain the loss at different time scales. The losses at different time scales are added together to obtain the total loss of the model.

[0013] This application also provides an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned remote heart rate estimation method based on multi-scale fusion.

[0014] This application also provides a computer program product comprising a computer program / instructions that, when executed by a processor, implement the aforementioned remote heart rate estimation method based on multi-scale fusion.

[0015] The advantages of this application are: (1) This application proposes a multi-scale temporal fusion mechanism to better capture the complex temporal dynamics of rPPG signals. It first decomposes the input features at multiple time scales and then reconstructs them through a learnable fusion mechanism, thereby effectively preserving the short-term and long-term periodic information in rPPG estimation. By explicitly modeling the multi-scale structure of rPPG, the method in this application effectively mitigates the effects of noise and environmental changes, thereby achieving more robust and accurate physiological signal estimation.

[0016] (2) This application introduces a hierarchical loss function hpc, which enforces consistency constraints at different time scales to ensure that the predicted rPPG signal is faithful to the ground real signal, thereby further improving the estimation accuracy.

[0017] (3) This application uses a spatiotemporal graph MSTmap instead of directly input video. When calculating MSTmap, a face recognition tool is used to detect faces, which can reduce the influence of face movement and greatly improve the model prediction accuracy.

[0018] (4) To address the shortcomings of existing methods in fully utilizing the inherent periodicity of rPPG signals, this application designs a multi-scale time fusion module capable of deeply observing multiple periodic scales of rPPG signals. It first decomposes the input features into multi-scale time representations through a downsampling mechanism, and then refines and integrates information at multiple time scales through a learnable upsampling process, thereby capturing local and global time dependencies, i.e., the correlation information between long and short periods. Furthermore, this application proposes a hierarchical loss function that enforces consistency constraints at different time scales, ensuring that the predicted rPPG signal maintains both fine-grained short-period information and coarse-grained long-period information.

[0019] (5) The multi-scale temporal fusion module aims to decompose the input features into multi-scale temporal representations through a downsampling mechanism, followed by a learnable upsampling process to refine and integrate information across multiple time scales, thereby capturing local and global temporal dependencies. This bottom-up multi-scale temporal information modeling approach enables the model to retain short-term dynamic changes (such as minute movements or rapid changes in illumination) at a higher temporal resolution, while capturing long-term dependencies (such as physiological rhythms or slow-changing trends) at a lower temporal resolution. Ultimately, the fused multi-scale temporal features are significantly enhanced in terms of temporal consistency and dynamic robustness, providing a more discriminative temporal feature representation for subsequent temporal modeling and physiological signal estimation.

[0020] (6) The multi-scale fusion strategy plays a crucial role in temporal feature modeling. On the one hand, the coarse-to-fine hierarchical information transmission mechanism helps to utilize macro-scale trend information to guide the dynamic modeling of fine-grained time series, thereby alleviating the problem that single-scale modeling easily ignores global temporal dependencies. On the other hand, the introduction of fine-scale features can supplement high-frequency details, enabling the model to have higher responsiveness to local temporal changes. Through this top-down multi-scale fusion structure, the model can establish a stable and consistent temporal representation at multiple time levels, providing more robust and information-rich feature support for subsequent time-domain signal recovery and rPPG estimation.

[0021] (7) To constrain the consistency of predicted signals during multi-scale time modeling, this application designs a hierarchical Pearson loss. Specifically, the network-predicted rPPG signal and the corresponding real rPPG signal are first sampled at multiple time scales. By setting different time window sizes and step sizes, and using average pooling, time feature sequences with different resolutions can be obtained at short-term, medium-term, and long-term time scales. Subsequently, the negative Pearson correlation loss between the predicted rPPG signal and the real rPPG signal is calculated at each time scale, and the losses at all scales are weighted and summed to form a hierarchical correlation constraint term. This process enables the network to not only focus on waveform similarity at a single time scale during training, but also maintain dynamic consistency at different aggregation levels.

[0022] (8) The introduction of hierarchical Pearson loss can effectively improve the robustness and generalization ability of the model in the time dimension. On the one hand, the multi-scale correlation constraint enables the model to learn fine-grained instantaneous changes (such as heart rate peak features) in short time windows, while maintaining the overall waveform trend and rhythm consistency in long time windows. On the other hand, this mechanism can reduce the model's sensitivity to noise and local jitter, so that the predicted rPPG signal has higher structural consistency and physiological rationality in multiple scales. Finally, the addition of hierarchical Pearson loss significantly improves the temporal smoothness and frequency domain stability of the network output signal, thereby improving the overall accuracy and robustness of the heart rate estimation task.

[0023] (9) The overall scheme proposed in this application achieves high-quality reconstruction and robustness improvement of rPPG signals through the synergistic effect of two mechanisms: "multi-scale temporal feature fusion" and "hierarchical correlation constraint". The multi-scale temporal fusion module extracts and integrates features at different temporal granularities through a downsampling-upsampling structure, effectively capturing short-term dynamic changes and long-term temporal trends; while the hierarchical Pearson loss constrains the consistency between the predicted signal and the real signal at multiple time scales, so that the model maintains structural correlation in both global and local temporal features. This design enables the network to simultaneously focus on rapid instantaneous changes and slow global rhythms, significantly enhancing the model's temporal modeling capability.

[0024] (10) In experiments, the overall framework of this application exhibits stronger temporal smoothness and frequency domain stability. Compared with traditional single-scale or single-loss-constraint models, this application achieves higher Pearson correlation coefficients and lower MAE / RMSE on multiple rPPG datasets, indicating significant advantages in heart rate estimation accuracy and signal morphology reconstruction. Furthermore, multi-scale feature fusion enhances the model's adaptability to complex scenarios such as illumination fluctuations, facial pose changes, and unstable video frame rates, significantly improving the system's robustness and generalization ability in real-world environments. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a remote heart rate estimation system based on multi-scale fusion according to this application.

[0026] Figure 2 This is a schematic diagram of a remote heart rate estimation system based on multi-scale fusion according to this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] This application provides a remote heart rate estimation method based on multi-scale fusion, comprising the following steps: S1, Input a face video, calculate the spatiotemporal graph of the face video. Spacetime diagram This includes channel dimension, spatial dimension, and time dimension.

[0029] For the input facial video, first divide the video into segments... A frame is a segment, dividing the video into multiple segments, meaning each segment receives... In this embodiment, a frame of video image, =300.

[0030] Then, the OpenFace face detection tool was used to detect each frame of the video image and extract facial feature points. Based on these facial feature points, the facial... n In this embodiment, there are one ROI (Region of Interest). n =6, such as Figure 2 As shown, the ROI regions are: forehead, left cheek, right cheek, left triangle area, right triangle area, and chin.

[0031] Will n Combining ROIs yields both individual ROIs and combinations of different ROIs, totaling [number missing]. N A combination, N= 2 n -1. For each combination, calculate the average pixel value of all ROIs in the combination across all color channels, for a total of color channels. C In this embodiment... C= 6, representing RGB and YUV respectively.

[0032] Finally, the dimension is obtained as C × N × F Vectors as spacetime graphs In this embodiment, the spatiotemporal diagram The dimensions are 6×63×300.

[0033] S2, regarding the spacetime diagram Perform embedding operations on the channel dimension, and perform linear mapping on the channel dimension to map the channel dimension to... D Dimension, spacetime graph after Embedding .

[0034] In this embodiment, D= 256, meaning the shape of the spacetime graph after embedding is 256×63×300.

[0035] S3 transforms the shape of the embedded spacetime graph into... (i.e., 300×63×256), using spatial attention mechanism to analyze the spatiotemporal graph. Different spatial locations (i.e.) N Modeling the correlations between combinations enhances the expressive power of spatial features in videos, particularly in spatiotemporal graphs. Spatial context information is embedded in each frame, and the spatiotemporal graph with enhanced spatial features is output. .

[0036] In the spacetime diagram The f Spatial context information embedded in a frame can be formally represented as: , , , , , in, For learnable parameters, For spacetime diagram The Middle f Feature map of the frame f = 1,..., F FFN stands for Feedforward Neural Network, LN represents Layer Normalization, and softmax represents the activation function. d For dimensions; The first after spatial feature enhancement f Feature map of a frame; By concatenating the spatially enhanced feature maps of all frames, we obtain the spatiotemporal map with enhanced spatial features. .

[0037] S4, for spatiotemporal graphs enhanced with spatial features By downsampling ( ) and linear fusion ( The process involves fusing features at different time scales to obtain fused features at different time scales. A linear mapping is then used to progressively map the fused features from larger time scales to the fused features from smaller time scales, ultimately resulting in a spatiotemporal graph after time fusion. This allows for the integration of information from different time scales, resulting in multi-scale temporal feature representations and providing richer temporal hierarchical features for subsequent time modeling. Specifically, as shown below: First, use downsampling ( )Calculate the spacetime graph exist M Fusion features at various time scales, where the first... Fusion characteristics at various time scales The calculation method is as follows: ; in, For the first A window size of a scale, , For spacetime diagram The duration of the time. In this embodiment, a total of 3 scale windows were used, namely... M= 3. The window sizes for the three scales are respectively , , Therefore, the features at three scales were obtained as follows: , , .

[0038] Then, using linear mapping, the fused features at larger time scales (coarse-scale features) are progressively mapped to the fused features at smaller time scales (fine-scale features). Fine-scale changes may introduce noise when capturing macro-level trends, while coarse-scale changes are more likely to provide clear macro-level information. Therefore, a coarse-to-fine feature enhancement approach is adopted to utilize macro-level knowledge at the coarse scale to guide trend modeling at the fine scale. Overall linear fusion ( This can be formally represented as: ; in, is a linear mapping function, representing a linear mapping from coarse to fine scale; In this embodiment, the specific process is as follows: first, pass Transform the dimension to And then Add them together to get Then, pass Transform the dimension to And then Add them together to get .

[0039] Linear fusion ( The ultimate goal is to output As a multi-scale representation of temporal features, i.e., a spatiotemporal graph after temporal fusion. By linearly mapping and deeply fusing coarse-scale and fine-scale features, the subsequent temporal attention mechanism can receive information of different granularities.

[0040] S5 transforms the shape of the time-fused spacetime graph into... By modeling long-range dependencies in the time dimension through temporal attention, this method captures the periodic dynamic changes related to heart rate in the spatiotemporal graph. Temporal context information is embedded in each spatial location to generate a spatiotemporal graph with enhanced temporal features. .

[0041] In the spacetime diagram The n Temporal context information is embedded in each spatial location, as shown below: , , ; ; ; in, For learnable parameters, For spacetime diagram The Middle n Feature map of a spatial location n= 1,..., N FFN stands for Feedforward Neural Network, LN represents Layer Normalization, and softmax represents the activation function. d For dimensions; Indicates the time feature enhancement of the first n Feature map of each spatial location; By stitching together the feature maps with enhanced temporal features of all spatial locations, a spatiotemporal map with enhanced temporal features is obtained. .

[0042] S6 transforms the shape of the time-enhanced spatiotemporal graph into... For spacetime diagrams Regression prediction is performed by mapping the spatiotemporal graph to the signal through a fully connected layer (MLP), outputting the predicted rPPG signal. The details are as follows: spatiotemporal diagram First, average pooling is performed on the spatial dimension to convert the spatial dimension to 1 dimension, resulting in... Then, through a linear layer, the channel dimension is reduced to 1 dimension, resulting in... This enables the mapping from the spatiotemporal diagram to the signal, ultimately predicting the rPPG signal. .

[0043] S7, during the model training process, for the predicted rPPG signal and the real rPPG signal, at different time scales (e.g., , , Under different time scales, the mean squared error (MSELoss) is calculated separately to obtain the loss. The multi-scale loss is then combined as the total loss of the model to guide the model to learn fine temporal features at multiple levels and improve the accuracy and stability of predicting rPPG signals.

[0044] This application designs a novel hierarchical Pearson loss. This method is used to force the model to learn consistent rPPG prediction results under constraints at different time scales. Specifically, average pooling is used to sample the real rPPG signal and the predicted rPPG signal into data at three different time scales. By setting different window sizes, data at different time scales can be obtained; the information aggregation capabilities of different time scales are different. This application sets the stride to be the same as the window size to prevent overlap and ensure that data from different windows do not interfere with each other. The sampled real rPPGs are as follows: , , The predicted rPPGs after sampling are respectively , , .

[0045] Stratified Pearson loss It can be represented as: ; in, ; in, Indicates the first i The actual signal value at each time scale. for The average value, Indicates the first i Predicted signal values ​​at each time scale for average value.

[0046] Depend on Figure 2 As shown, this application provides a remote heart rate estimation system based on multi-scale fusion, and the system model includes the following 7 modules.

[0047] Module 1, Spatiotemporal Graph Extraction Module: The input face video is processed to calculate the spatiotemporal map (MSTmap), which involves extracting motion, structure and texture information from the video to provide necessary visual features for subsequent signal processing.

[0048] Module 2, Embedding module: This module 2 performs an embedding operation on the spatiotemporal graph obtained from module 1 (a necessary operation before inputting it into the Transformer). Through linear mapping and positional encoding, it converts the spatial image information into a sequence of feature vectors, providing a suitable input format for the subsequent Transformer module.

[0049] Module 3, Spatial Transformer Module: This module 3 receives the feature sequence from module 2, uses the spatial attention mechanism to model the correlation between different spatial locations, thereby enhancing the expressive power of spatial features in the video, and outputs a spatially enhanced representation by further fusing spatial context information.

[0050] Module 4, Multi-scale Temporal Fusion Module: This module 4 receives the spatial feature sequence from module 3, and performs downsampling ( ) and linear fusion ( The process involves fusing features at different time scales to obtain fused features at different time scales. A linear mapping is then used to progressively map the fused features from larger time scales to the fused features from smaller time scales, ultimately resulting in a spatiotemporal graph after time fusion. This allows for the integration of information from different time scales, resulting in multi-scale temporal feature representations and providing richer temporal hierarchical features for subsequent time modeling.

[0051] Module 5, Time Transformer Module: This module 5 receives multi-scale temporal feature representations from module 4, models long-range dependencies in the temporal dimension through a temporal attention mechanism, captures periodic dynamic changes related to heart rate, and thus generates a feature sequence with enhanced temporal features.

[0052] Module 6, Regression Prediction Module: This module 6 receives the time feature sequence from module 5, maps the features to the signal through a fully connected layer (MLP), and outputs the final predicted rPPG signal.

[0053] Module 7, Hierarchical Loss Module: This module 7 is used only during model training to calculate the total model loss. It calculates the loss for both the predicted and actual rPPG signals at different time scales (e.g., ...). , , Under different time scales, the mean squared error (MSELoss) is calculated separately to obtain the loss. The multi-scale loss is then combined as the total loss of the model to guide the model to learn fine temporal features at multiple levels and improve the accuracy and stability of predicting rPPG signals.

[0054] This embodiment conducted experimental comparisons under the following four conditions: Case 1: When using only the ordinary transformer structure, without multi-scale fusion and hierarchical Pearson loss, the RMSE is 6.32; Scenario 2: After using the transformer architecture and multi-scale fusion, the RMSE dropped to 6.24; Case 3: Using a transformer structure with hierarchical Pearson loss, the RMSE drops to 6.26; Case 4: When using the transformer structure, the RMSE decreases the most when multi-scale fusion and hierarchical Pearson loss are used simultaneously, dropping to 6.13; RMSE is an indicator for evaluating the accuracy of predicting rPPG signals; the smaller the RMSE, the more accurate the rPPG signal prediction.

[0055] The above are merely preferred embodiments of this application and are not intended to limit the scope of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims

1. A method for remote heart rate estimation based on multi-scale fusion, characterized in that, The method comprises the following steps: S1, input a face video, calculate a space-time graph of the face video space-time graph including a channel dimension, a spatial dimension, and a temporal dimension S2, the spatiotemporal graph The Embedding operation is performed in the channel dimension to obtain the spatiotemporal graph after Embedding ; S3 employs a spatial attention mechanism in the spatiotemporal graph. Spatial context information is embedded in each frame to obtain a spatiotemporal graph with enhanced spatial features. ; S4, for the space-time diagram The feature fusion is respectively performed at different time scales to obtain fused features at different time scales. The fusion features of the larger time scale are sequentially mapped into the fusion features of the smaller time scale step by step; a final spatiotemporal map is obtained ; S5, adopting a time attention mechanism, embedding time context information in each spatial position of the spatiotemporal graph to obtain a spatiotemporal graph with time feature enhancement ; and S5, adopting a time attention mechanism, embedding time context information in each spatial position of the spatiotemporal graph to obtain a spatiotemporal graph with time feature enhancement ; and S6, spatiotemporal map perform regression prediction, output predicted rPPG signal.

2. The method of claim 1, wherein, In the training process of the model, the mean square error is calculated for the predicted rPPG signal and the real rPPG signal at different time scales to obtain the loss at different time scales, and the losses at different time scales are added as the total loss of the model.

3. The method of claim 1, wherein, In step S1, the input face video is divided into multiple segments, i.e. each segment obtains a frame video image, according to each frame. In step S1, the input face video is divided into multiple segments, i.e. each segment obtains a frame video image, according to each frame. In step S1, the input face video is divided into multiple segments, i.e. each segment obtains a frame video image, according to The face feature points are extracted by using a face detection tool to detect each frame of video image, and a ROI of the face is defined based on the face feature points n . The n ROIs are combined to obtain a total of N combinations of single ROIs and combinations of different ROIs, N= 2 n -1. For each combination, average pixel values of all ROIs in the combination on each color channel are calculated, and there are a total of C color channels; thereby obtaining a spatiotemporal graph , the channel dimension of the spatiotemporal graph is C , the spatial dimension is N , and the temporal dimension is F .

4. The method of claim 1, wherein, In step S3, the spatial context information is embedded in the temporal-spatial map at the first f frame, as follows: ， ， ； ； ； wherein, are learnable parameters; is a spatio-temporal graph in the f feature map of the f = 1,..., F , collectively frames; FFN is a feed-forward neural network; LN denotes a layer normalization operation; softmax denotes an activation function; d is a dimension; is a feature map of the f frame after spatial feature enhancement; The spatial feature enhanced feature maps of all frames are spliced to obtain a spatial feature enhanced space-time graph .

5. The method of claim 1, wherein, In step S4, the spatio-temporal graph is constructed for each time scale In step S4, the spatio-temporal graph is constructed for each time scale M In step S4, the spatio-temporal graph is constructed for each time scale M In step S4, the spatio-temporal graph is constructed for each time scale No. Fusion features at various time scales The calculation method is as follows: ； wherein, is the window size for the first time scale, , is the time length of the spatiotemporal map ; The fusion features of the larger time scale are sequentially mapped into the fusion features of the smaller time scale step by step using linear mapping, and the mapping mode is as follows: ； wherein ; denotes a linear mapping from coarse to fine scale; Eventually, the spatiotemporal graph after temporal fusion output.

6. The method of claim 1, wherein, In step S5, the temporal context information is embedded in the spatio-temporal map at the first n spatial location, as follows: ， ， ； ； ； in, These are learnable parameters; For spacetime diagram The Middle n Feature map of a spatial location n= 1,..., N ,common N Each spatial location; FFN is a feedforward neural network; LN represents layer normalization; softmax represents the activation function; d For dimensions; For the time feature enhancement of the first n Feature map of each spatial location; The time feature enhanced feature map of all spatial positions is spliced to obtain a time feature enhanced space-time map .

7. The method of claim 1, wherein, In step S6, the spatiotemporal graph is subjected to regression prediction, as follows: spatio-temporal graph The spatio-temporal graph is mapped to the predicted rPPG signal by first performing an average pooling operation in the spatial dimension to convert the spatial dimension to 1 dimension, and then mapping the channel dimension to 1 dimension through a linear layer.

8. A remote heart rate estimation system based on multi-scale fusion, characterized in that, The system model is applied to the remote heart rate estimation method based on multi-scale fusion according to any one of claims 1-7, and the system model comprises: The spatiotemporal graph extraction module is configured to calculate a spatiotemporal graph of the face video ; Embedding module, configured to embed the spatiotemporal graph perform an Embedding operation on the channel dimension to obtain an Embedding spatiotemporal graph ; The spatial Transformer module adopts a spatial attention mechanism and is used for embedding spatial context information in each frame of the spatiotemporal graph to obtain a spatiotemporal graph with enhanced spatial features . ​ Multi-scale temporal fusion module for spatiotemporal graphs This is used to perform feature fusion at different time scales, obtaining fused features at different time scales. The fused features at larger time scales are then progressively mapped onto the fused features at smaller time scales, ultimately resulting in a spatiotemporal map after time fusion. ; The Time Transformer module employs a time attention mechanism for spatiotemporal graphs. Temporal context information is embedded in each spatial location to obtain a spatiotemporal map with enhanced temporal features. ; The regression prediction module is used for spatiotemporal plots. Perform regression prediction and output the predicted rPPG signal; The hierarchical loss module is used only in the training process of the model and is used for calculating the total loss of the model; The system model is trained by using a sample set, the sample set comprises a face video and a label corresponding to the face video, i.e., a real rPPG signal; in the training process of the model, the hierarchical loss module calculates the mean square error for the predicted rPPG signal and the real rPPG signal at different time scales to obtain the loss at different time scales, and the losses at different time scales are added as the total loss of the model.

9. An electronic device, comprising: The device comprises a processor, a memory, and a computer program stored on the memory and executable on the processor, and the processor implements the remote heart rate estimation method based on multi-scale fusion according to any one of claims 1-7 when executing the computer program.

10. A computer program product, characterised in that, The device comprises a computer program / instruction, and the computer program / instruction is executed by a processor to implement the remote heart rate estimation method based on multi-scale fusion according to any one of claims 1-7.

Images