A heart rate estimation method, system and device based on dynamic gating self-adaptive fusion

Abstract

The application discloses a heart rate estimation method, system and device based on dynamic gating self-adaptive fusion. The method comprises the following steps: acquiring a continuous face video frame sequence; generating a corresponding frame difference sequence according to the face video frame sequence; respectively performing shallow space-time feature extraction on the face video frame sequence and the frame difference sequence to obtain an appearance feature map and a fluctuation feature map; and performing global context perception on the appearance feature map and the fluctuation feature map to generate a context vector for representing a current environment state and the like. The application aims to introduce a learnable dynamic fusion mechanism, adaptively calculate and distribute the fusion weights of the appearance features and the time sequence fluctuation features according to the video context environment, and cooperate with a special training regularization strategy to effectively inhibit the excessive dependence of the network on the static appearance features, thereby improving the robustness and generalization ability of the heart rate estimation network in complex real scenes.

Description

Technical Field

[0001] This invention relates to the field of biomedical signal processing and computer vision, specifically to a non-contact video physiological measurement method and system, and more particularly to a heart rate estimation method, system and device based on dynamic gating adaptive fusion, applicable to non-contact heart rate estimation, respiratory rate estimation and other vital sign monitoring scenarios. Background Technology

[0002] Currently, in the field of non-contact imaging volumetric pulse wave (rPPG) technology, deep learning-based video physiological measurement methods typically incorporate a multimodal feature extraction and fusion module (called a fusion stem) at the network front end to simultaneously utilize both spatial appearance information and temporal fluctuation information from the video for heart rate estimation. Please refer to the appendix. Figure 1 , Figure 1 This paper demonstrates a typical processing flow of a multimodal feature fusion interferometer network. Specifically, the network receives a sequence of consecutive RGB face video frames. To explicitly extract temporal fluctuations, a frame difference image is generated by calculating the pixel differences between adjacent frames or between frames. (Also known as temporal gradient), thereby constructing a parallel bimodal input—the original RGB frame and the frame difference image; subsequently, and The data are fed into two parallel shallow 3D convolutional modules for initial spatiotemporal feature extraction to obtain RGB feature maps. Frame difference feature map ; then will and In existing technologies, feature fusion is typically performed using static operators such as channel concatenation (Concat) or element-wise addition (Add). The fused feature tensor is then fed into a deep backbone network for spatiotemporal dependency modeling, and finally, the heart rate estimation result is output as a regression.

[0003] The aforementioned schemes, as representative architectures of existing rPPG networks, have been reported in numerous publications and patents. For example, Chinese patent CN119700062B (publication date: May 2, 2025) discloses a non-contact heart rate detection method, which designs two paths to process the original video frame and the differential video frame sequence respectively, and then concatenates the features extracted by the two paths in the channel dimension to output the rPPG signal prediction result. Another example is Chinese patent CN121445342A (publication date: February 3, 2026), which discloses a non-contact heart rate estimation method that integrates physical priors and Transformer. This method designs a dual-branch spatiotemporal feature fusion module, including a motion branch (with normalized frame difference as input) and an appearance branch (with the original video frame as input). The features extracted by the two branches are fused to generate enhanced spatiotemporal tubular features. Furthermore, network architectures based on 3D-CNN and spatiotemporal Transformers (such as RhythmFormer) also employ a similar dual-input and static fusion front-end design. For related technical details, please refer to Zou B, Guo Z, Chen J, et al. RhythmFormer: Extracting rPPGSignals Based on Hierarchical Temporal Periodic Transformer[J]. 2024, which describes the basic structure of the aforementioned dual-input construction and static fusion processing.

[0004] However, the above-mentioned existing technologies have the following shortcomings: (1) Lack of environmental adaptability. The fusion weights of the existing technologies are fixed and unlearnable. Regardless of how the actual environmental state of the video clip (such as lighting conditions and motion amplitude) changes, the RGB features and frame difference features are mixed in a fixed ratio. When encountering drastic changes in ambient lighting, the brightness of the RGB image undergoes a global drastic change, directly introducing a large amount of noise; when the subject undergoes large-amplitude rigid movement, the frame difference image is filled with motion edges caused by displacement, and the signal-to-noise ratio of temporal fluctuations drops sharply. Since the network cannot dynamically judge the reliability of each modality according to the current context, the measurement accuracy drops significantly in complex scenes. (2) The network is prone to overfitting of appearance features. During the model training process, since RGB appearance features (such as skin color, face shape, and ambient lighting distribution) are easier to learn than the weak physiological jump signals contained in the frame difference, the existing static fusion mechanism lacks effective constraints on different modal features, causing the network to tend to over-rely on RGB appearance features to "guess" the heart rate, while ignoring the extraction of real physiological temporal fluctuations. The aforementioned problems further lead to a significant decrease in the model's generalization ability when deployed across datasets and scenarios, making it difficult to meet the robustness requirements of actual commercial applications. Summary of the Invention

[0005] To address the issues of insufficient environmental adaptability and overfitting of appearance features caused by static fusion strategies in existing technologies, this invention provides a heart rate estimation method, system, and device based on dynamic gating adaptive fusion. The aim is to introduce a learnable dynamic fusion mechanism to adaptively calculate and allocate fusion weights for appearance features and temporal fluctuation features according to the video context, and to effectively suppress the network's excessive reliance on static appearance features by employing a dedicated training regularization strategy. This, in turn, enhances the robustness and generalization ability of the heart rate estimation network in complex real-world scenarios.

[0006] According to one aspect of the present invention, a heart rate estimation method based on dynamic gating adaptive fusion is provided, the method comprising: Obtain a continuous sequence of face video frames; Generate a corresponding frame difference sequence based on the face video frame sequence; Shallow spatiotemporal feature extraction is performed on the face video frame sequence and the frame difference sequence to obtain appearance feature map and fluctuation feature map, respectively. Global context awareness is performed on the appearance feature map and the fluctuation feature map to generate a context vector representing the current environmental state; The context vector is input into a learnable mapping network to generate a first fusion weight corresponding to the appearance feature map and a second fusion weight corresponding to the fluctuation feature map; Based on the first fusion weight and the second fusion weight, the appearance feature map and the fluctuation feature map are weighted and fused to generate a fused feature; Deep spatiotemporal modeling is performed on the fused features to output heart rate estimation results.

[0007] In the above technical solution, a heart rate estimation method based on dynamic gating adaptive fusion is provided. By dynamically weighting and fusing appearance features (RGB video frames) and temporal fluctuation features (frame difference sequence), the heart rate can be estimated from face videos.

[0008] Compared with existing technologies that use static splicing or element-by-element addition for multimodal fusion, the essential feature of this scheme is that: first, global context awareness is performed on the bimodal features to generate a context vector that reflects the current environmental state (such as lighting conditions and motion amplitude); then, the context vector is input into a learnable mapping network to dynamically generate fusion weights corresponding to each modality; and finally, weighted fusion is completed based on the dynamic weights.

[0009] The direct effect of this distinction is that, in scenarios with sudden changes in lighting, the appearance feature map... Severely contaminated by noise, learnable mapping networks can adaptively reduce the first fusion weights. The numerical value reduces the network's reliance on contaminated appearance features; in scenes with violent motion, frame difference images are filled with rigid displacement edges, reducing the signal-to-noise ratio of temporal fluctuation features, allowing the mapping network to adaptively reduce the second fusion weights. The numerical values ​​are as follows. When the gated network senses high-frequency spatial noise or large-area rigid displacement, the corresponding weight reduction occurs dynamically, without the need for pre-defined rules or external intervention. This mechanism enables the network to maintain stable feature extraction quality under different environmental conditions, and improves measurement robustness in complex uncontrolled scenarios compared to a single static fusion strategy.

[0010] The context vector is obtained by globally compressing the spatiotemporal dimensions of the concatenated bimodal features. This context vector simultaneously encodes the spatiotemporal statistical information of both the appearance modality and the fluctuation modality, rather than generating weights based solely on a single modality or local features. Global spatiotemporal context awareness ensures that the input to the mapping network not only includes information from the current frame but also incorporates the overall spatiotemporal state, determining the global contextual basis of the generated weights. Therefore, in this scheme, the combination of "global context awareness" and "bimodal shallow features" results in a dynamic fusion mechanism sensitive to environmental states, which operates in a substantially different manner from conventional attention mechanisms. Furthermore, this scheme only involves forward propagation computation during inference. The learnable mapping network and weighted fusion module are both lightweight operations (multilayer perceptron and element-wise multiplication), with negligible computational cost compared to the backbone network. This does not introduce significant additional latency during practical deployment, demonstrating good engineering practicality.

[0011] In summary, this solution addresses the technical problem that existing static fusion strategies cannot adaptively adjust the multimodal feature dependencies according to the environmental state by introducing a global context awareness and a learnable dynamic weight generation mechanism. It achieves environment-adaptive dynamic fusion without relying on any training regularization methods during the inference stage, providing high-quality, scene-adaptive fusion feature inputs for the subsequent backbone network.

[0012] In some embodiments, the learnable mapping network is a gating network used to output the first fusion weight and the second fusion weight based on the context vector.

[0013] In the above technical solution, the learnable mapping network is further defined as a gating network, used to output the first fusion weight and the second fusion weight based on the context vector. As a specific implementation of the learnable mapping network, the gating network takes the environmental state information extracted by global context awareness and transforms it into dynamic fusion weights.

[0014] In the field of deep learning, the core function of gating mechanisms is to dynamically control the degree of information path conduction based on input signals, which inherently fits the task requirement of "adjusting fusion weights according to environmental conditions." This is achieved by using context vectors... The input is a gating network, and the activation value of the gating network directly determines the proportion of appearance features and fluctuation features involved in the fusion. There is a clear functional mapping relationship between the input and output - the context vector reflects the environmental state, and the gating output determines the modality confidence. No additional transformation or encoding / decoding structure is required, and its structure is simple and computationally lightweight.

[0015] Compared to existing technologies, current static fusion schemes lack the concept of "gating," and the feature fusion weights are fixed and cannot be adjusted. Furthermore, existing rPPG-related schemes that incorporate gating mechanisms (such as the frequency-adaptive gated FAG in FLNM-Net or the reaction-driven gated RDG in TH-STT) apply their gating mechanisms to frequency domain enhancement and facial motion artifact tracking, respectively, rather than controlling the fusion weight allocation between appearance and fluctuation modes. In this scheme, the input to the gating network is the global context vector, and it controls the generation of bimodal fusion weights, exhibiting a clear difference in application compared to the aforementioned existing gating schemes.

[0016] From an engineering implementation perspective, gating networks can be implemented using various specific structures (such as MLP, GRU, lightweight attention modules, etc.). Those skilled in the art can choose a suitable gating implementation based on the computational constraints of the specific deployment platform. Furthermore, since gating networks only contain fully connected layers and activation functions, their parameter count is typically in the thousands to tens of thousands. Compared to the millions to tens of millions of parameters in the backbone network, their computational overhead is negligible and does not constitute a performance bottleneck in the inference stage, making them advantageous for deployment on resource-constrained edge devices. Using the environmental state information extracted by global context awareness as the input to the gating system to control the allocation of bimodal fusion weights is an application scenario not reported in existing rPPG literature.

[0017] In some embodiments, the gating network is a multilayer perceptron gating network, which includes a dimensionality reduction layer and a dimensionality increase layer. After the context vector is sequentially reduced and increased in dimensionality, the first fusion weight and the second fusion weight are output through a Softmax or Sigmoid activation function.

[0018] In the above technical solution, the gated network is further defined as a multilayer perceptron (MLP) gated network. Specifically, the MLP gated network includes a dimensionality reduction layer and a dimensionality increase layer. After the context vector is sequentially reduced and increased in dimensionality, a first fusion weight and a second fusion weight are output through a Softmax or Sigmoid activation function. This solution clarifies the specific structural form and activation method of the gated network.

[0019] The key feature of this scheme lies in its use of a multilayer perceptron (MLP) to construct a gating network, and this MLP gating network possesses a bottleneck structure that combines dimensionality reduction and dimensionality increase. (Context vector) First, the dimensions are compressed to a lower dimension through a dimensionality reduction layer, then restored to the original dimension through a dimensionality increase layer, and finally the weights are output through an activation function.

[0020] The technical significance of the aforementioned "dimensionality reduction-dimensionality increase" bottleneck structure lies in the following: the dimensionality reduction layer forces the network to compress and encode the information in the context vector, extracting the environmental state features most relevant to the weight decisions (such as whether there are sudden changes in illumination or whether the motion amplitude is too large), while discarding redundant information unrelated to the weight decisions; the dimensionality increase layer then remaps the compressed features to an output space with the same dimensions as the original, outputting two independent weight values. This structure achieves efficient extraction from high-dimensional context vectors to low-dimensional key features with a very small number of parameters, avoiding the overfitting risk that may arise from directly using fully connected layers. This dimensionality reduction design allows the number of parameters in the MLP gated network to be controlled at the thousand level, which is negligible compared to the millions of parameters in the backbone network.

[0021] Regarding the choice of activation function, Softmax... and satisfy Sigmoid is suitable for scenarios where appearance and fluctuation modes compete, meaning the network needs to allocate a limited "trust budget" between the two modes; and Each independently located Within this range, it is applicable to scenarios where two modalities can be trusted or suppressed simultaneously. Those skilled in the art can select an appropriate activation method based on specific application scenarios (such as sensor noise characteristics, environmental change patterns, etc.) without retraining the entire network architecture.

[0022] Furthermore, in this scheme, the input to the MLP is a context vector that has undergone global spatiotemporal compression. Rather than the original features or artificially designed statistics, the context vector itself embodies the core concept of this invention; the selection of the dimensionality reduction ratio needs to strike a balance between "retaining sufficient environmental state discrimination information after dimensionality reduction" and "having a sufficiently small number of parameters to adapt to edge deployment".

[0023] In some embodiments, the global context awareness includes: concatenating the appearance feature map and the fluctuation feature map along the channel dimension, and then compressing the spatiotemporal dimension through a three-dimensional global average pooling layer to generate the context vector.

[0024] In the above technical solution, the specific implementation of the global context awareness is further defined: after concatenating the appearance feature map and the fluctuation feature map along the channel dimension, the spatiotemporal dimension is compressed through a three-dimensional global average pooling layer (3D-GAP) to generate a context vector. This solution clarifies the specific transformation path from bimodal features to context vectors.

[0025] The key feature of this solution lies in its implementation using a "channel splicing + three-dimensional global average pooling" method. The technical significance of this method is that it... and After concatenation along the channel dimension, the feature maps of the two modalities maintain a one-to-one positional relationship in both spatial and temporal dimensions—that is, at the same spatiotemporal location (same frame index, same pixel coordinates), the concatenated vector simultaneously contains both appearance and fluctuation information at that location. The 3D global average pooling layer then performs global compression in the temporal, height, and width dimensions, resulting in a more efficient output context vector. Each channel value is a global statistical response at all spatiotemporal locations on the corresponding channel. This operation ensures that the context vector comprehensively reflects the overall state of the entire video segment in the spatiotemporal dimension.

[0026] With the first and Compared to schemes that perform independent pooling followed by stitching, the advantage of this "stitching before pooling" approach is that the pooling operation occurs after stitching, allowing GAP to simultaneously perceive the joint statistical features of two modalities at the same spatiotemporal location—for example, whether a pixel location simultaneously exhibits strong changes in appearance brightness and a large frame difference response (potentially indicating a sudden change in illumination or motion), or whether a certain time segment simultaneously exhibits stable appearance texture and weak frame difference fluctuations (potentially indicating favorable physiological signal acquisition conditions). This joint perception capability enables the context vector... It can encode collaborative statistical information on "appearance-fluctuation," rather than simply the sum of individual statistics. This context vector comprehensively reflects environmental information such as lighting conditions, intensity of motion, and skin color distribution, and serves as the basis for subsequent dynamic weight generation. Its information richness directly affects the accuracy of the gating network's output weights.

[0027] In the specific context of this scheme, the special nature of this operation lies in its position—it occurs after shallow feature extraction and before dynamic weight generation, serving as a bridge connecting "bimodal feature extraction" and "learnable weight generation." This position determines that the information content carried by the context vector directly determines the quality of the generated dynamic fusion weights. Without this global context-aware step, the learnable mapping network will have no input source, dynamic weight generation cannot proceed, and the entire adaptive fusion mechanism will fail to operate.

[0028] Furthermore, the 3D global average pooling layer itself does not contain trainable parameters, and its calculation is a simple averaging operation with extremely low computational cost. The total computational cost of the entire context awareness and weight generation module is negligible during the inference phase and will not cause any perceptible latency impact on real-time heart rate estimation scenarios.

[0029] In summary, this scheme clarifies the generation path of context vectors by concretizing global context awareness into a "concatenation + 3D-GAP" implementation. This step, as a crucial intermediate link from bimodal features to dynamic weights, ensures that the context vector can encode the joint statistical features of the two modalities at their corresponding spatiotemporal locations through its "concatenation then pooling" operation. This provides an informational foundation for the subsequent gating network to generate accurate environment-adaptive fusion weights.

[0030] In some embodiments, the weighted fusion is an element-wise weighted summation, and the fusion feature ,in, The first fusion weight, This is the second fusion weight.

[0031] In the above technical solution, the specific calculation method of the weighted fusion is further defined as element-wise weighted summation, and is expressed by the formula. The mathematical relationship between the fused features and the two input feature maps and dynamic weights was clarified.

[0032] The essential feature of this scheme lies in specifying the fusion operator as an "element-by-element weighted summation" and providing a clear mathematical expression. This formula has the following technical significance: First, in the formula and It is dynamically generated by a learnable mapping network (gated network) based on the global context vector, rather than by a preset fixed constant. and The value of varies with the environmental conditions of the input video clip. In the inference phase of this scheme, if the lighting is good and the motion is gentle, and Both values ​​are likely within the 0.4~0.6 range, indicating that both modalities are trusted; if a sudden change in illumination causes noise contamination of the RGB features, then... The value is lower than The value (e.g.) If the subject's violent movement causes the frame difference image to be filled with rigid displacement noise, then The value is lower than The value (e.g.) In contrast, existing static fusion schemes, which use element-wise addition, implicitly assign a fixed 1:1 weight ratio to the two modalities, or concatenate them and then have the relative weights implicitly determined by subsequent convolutional layers with fixed kernel parameters. Regardless of the method, the fused weights remain fixed after model deployment. The two approaches differ fundamentally in whether the weights are adaptive to the input.

[0033] Secondly, the element-wise weighted summation fusion operation itself is linear and parameterless, introducing no additional trainable parameters. Compared to alternatives (such as cross-attention mechanisms or dynamic convolution), it has the lowest computational complexity—only element-wise multiplication and addition operations, without relatively high-overhead operations such as matrix multiplication or Softmax. This means that the additional latency introduced by the fusion module during the inference phase can be controlled to below milliseconds, meeting the low-latency requirements of real-time heart rate estimation scenarios (such as intelligent cockpit driver monitoring).

[0034] again, The formula form enables the fusion of features and Maintaining the same feature dimensions (time dimension, spatial dimension, and number of channels) ensures that fused features can be directly input into the subsequent backbone network without additional dimensional transformations or adaptation layers, thus preserving the coherence and complete transmission of feature semantics. The weights in this formula in this scheme... and Instead of being a fixed constant or determined by heuristic rules, the weights are dynamically generated from the global context vector via a learnable mapping network. In other words, this scheme protects not an isolated "weighted summation" operation, but a fusion link in a complete chain of "context awareness → dynamic weight generation → weighted summation". Element-by-element weighted summation, as the final execution step in this chain, makes the entire dynamic fusion mechanism feasible in practical deployment due to its simplicity and computational efficiency. While more complex fusion operators (such as cross-attention) can achieve similar functional results, they would face additional computational and storage pressures on resource-constrained edge devices.

[0035] In summary, this scheme provides a quantifiable implementation method for the fusion step by specifying weighted fusion as element-wise weighted summation and clarifying its mathematical expression. The simplicity of this formula ensures the computational efficiency of the dynamic fusion mechanism in practical deployments.

[0036] In some embodiments, the frame difference sequence is generated by calculating the pixel difference between adjacent frames; or, the frame difference sequence is generated by a center difference algorithm.

[0037] In the above technical solution, two specific methods for generating the frame difference sequence are further defined: generation by calculating the pixel difference between adjacent frames, or generation by a center difference algorithm. This definition clarifies the acquisition path of the temporal fluctuation mode.

[0038] The key features of this scheme are: the adjacent frame difference method directly calculates the pixel value difference between adjacent frames, which has a fast response speed, is sensitive to inter-frame abrupt changes, and can capture frame-by-frame micro-jumps in skin color caused by heartbeats. It is also relatively sensitive to motion displacement and can provide high temporal resolution time-series fluctuation signals in scenes with gentle motion or static conditions; the center difference method calculates the difference by taking the current frame as the center and combining information from previous and subsequent frames. Its response is relatively smooth and has a certain suppression effect on high-frequency noise. It is suitable for scenes with gentle motion or relatively stable lighting and can extract periodic physiological signals more stably.

[0039] The adjacent frame difference method is sensitive to short-term abrupt changes and is suitable for rapidly changing scenarios; the center difference method is smoother for temporal changes, has a suppressive effect on high-frequency noise, and is suitable for extracting periodic physiological signals. Those skilled in the art can choose an appropriate frame difference generation method according to the specific deployment environment (such as the camera's frame rate, lighting stability, the expected motion amplitude of the subject, etc.) to ensure the quality of fluctuation features while taking into account computational efficiency.

[0040] The quality of the frame difference sequence directly affects The quality of this affects the context vector. The richness of the wave mode information ultimately affects the generation of the gating network. The accuracy of the weights. In this scheme, the frame difference generation method, along with subsequent steps such as global context awareness and learnable weight generation, constitutes a complete signal processing chain.

[0041] In some embodiments, the shallow spatiotemporal feature extraction is performed by two shallow 3D convolutional modules that do not share weights on the face video frame sequence and the frame difference sequence, respectively.

[0042] In the above technical solution, the specific execution method of the shallow spatiotemporal feature extraction is further defined: two shallow 3D convolutional modules that do not share weights are executed on the face video frame sequence and the frame difference sequence, respectively. This solution clarifies the parameter relationship between the branches of the dual-modal shallow feature extraction.

[0043] The key feature of this scheme lies in further restricting the sharing of weights between the two shallow 3D convolutional modules. The technical significance of this feature is: Appearance feature diagram Originating from raw RGB video frames, its information is primarily static spatial structure, including relatively stable appearance information such as skin tone, texture, and facial feature distribution. (Fluorescence feature map) Originating from frame difference sequences, the information is primarily dynamic and temporally varied, including pixel jumps and motion information between adjacent frames. The data distributions of the two modalities are fundamentally different, and the convolutional kernel parameters for feature extraction should ideally be different. If a single-branch structure with shared weights is used, the same set of convolutional kernel parameters needs to be adapted to both different input data distributions simultaneously, resulting in a compromise where one is suitable while the other is poorly adapted. In contrast, a dual-branch structure without shared weights allows each branch's shallow 3D convolutional module to optimize independently for the data distribution characteristics of its respective modality—the convolutional kernel parameters of the RGB branch converge towards "extracting stable spatial structures," while the convolutional kernel parameters of the Diff branch converge towards "capturing temporally changing patterns." During training, the two modules receive gradient signals from their respective modalities independently, learning feature extraction parameters more suitable for their respective input data distributions.

[0044] While the single-branch structure with shared weights reduces the number of model parameters, it comes at the cost of feature extraction accuracy. In the rPPG task, the physiological pulse amplitude in the frame difference signal is typically only about 1% of the pixel value change, which is an extremely weak signal, requiring high parameter accuracy from the feature extraction layer. If weight sharing causes the convolutional kernel parameters of the Diff branch to be biased towards the optimization direction of the RGB branch, the already weak fluctuation signal may be submerged in the shallow extraction stage. The dual-branch structure without shared weights provides an independent parameter space for the Diff branch, allowing the convolutional kernel of this branch to respond more sensitively to small changes in the temporal dimension, which is beneficial for preserving the physiological information in the fluctuation features in the early stages of feature extraction.

[0045] From a parameter count perspective, the shallow 3D convolutional module contains only a single convolutional layer (3×3×3 kernel, stride 2, 64 channels), and its parameter count is extremely small compared to the backbone network (which typically contains dozens of 3D convolutional or Transformer layers). Even with a two-branch structure that does not share weights, the increase in parameter count is only twice that of a single-branch architecture, and this only occurs in the shallow layers. This increase is negligible in terms of the overall model parameter count and will not have a substantial impact on model storage and inference speed. Compared to the gain it brings to the accuracy of fluctuating feature extraction, this increase in parameter count is a reasonable engineering trade-off.

[0046] In summary, this scheme further defines the specific implementation method of "extracting two shallow 3D convolutional modules that do not share weights separately", which ensures that appearance features and fluctuation features obtain their respective suitable convolutional kernel parameters in the shallow extraction stage. This ensures that the weak physiological signals in the frame difference mode are not overwhelmed by the optimization direction of the RGB mode in the early stage of feature extraction, and provides more reliable feature input for subsequent dynamic fusion.

[0047] According to another aspect of the present invention, a method for training a heart rate estimation model based on dynamic gating adaptive fusion is provided, the method comprising: Obtain the sample face video frame sequence and the corresponding real heart rate label; Generate a corresponding frame difference sequence based on the sample face video frame sequence; Shallow spatiotemporal feature extraction is performed on the sample face video frame sequence and the frame difference sequence to obtain appearance feature map and fluctuation feature map; Global context awareness is performed on the appearance feature map and the fluctuation feature map to generate a context vector representing the current environmental state; The context vector is input into a learnable mapping network to generate a first fusion weight corresponding to the appearance feature map and a second fusion weight corresponding to the fluctuation feature map; The appearance feature map is set to zero with a preset probability; Based on the first fusion weight and the second fusion weight, the processed appearance feature map and the fluctuation feature map are weighted and fused to generate a fused feature; Deep spatiotemporal modeling is performed on the fused features to obtain the predicted heart rate estimation result; Calculate the prediction loss between the predicted heart rate estimate and the true heart rate label, and calculate the regularization penalty loss for the first fusion weight; The parameters of the heart rate estimation model are updated based on the weighted sum of the prediction loss and the regularization penalty loss.

[0048] The above technical solution provides a training method for a heart rate estimation model based on dynamic gating adaptive fusion. This method introduces two regularization constraints, namely modality random dropout and weight sparsity penalty, on the basis of forward propagation of the inference method, and updates the model parameters through a multi-task loss function.

[0049] Compared to existing rPPG model training methods (which typically only use prediction loss for end-to-end training without imposing additional constraints on the multimodal fusion process), this scheme has two key features: first, it introduces a random modality dropout mechanism, setting the appearance feature map to zero with a certain probability during forward propagation; second, it introduces an additional regularization penalty term for the first fusion weight into the total loss. These two points constitute a dual constraint on appearance overfitting from both the data and loss perspectives.

[0050] The modality random dropout mechanism discards appearance feature maps during training with a preset probability (p=0.3 in this embodiment, preferably within the range of 0.1~0.5). Setting it to zero causes the weighted fusion to degenerate into At this point, the network cannot obtain any appearance information and can only rely on frame difference features for heart rate estimation. Since the physiological pulse signal amplitude in the frame difference is weak (typically only accounting for about 1% of the pixel value change), the network is "forced" to deeply mine the temporal patterns in the fluctuation features in this training step to reduce the loss function. The effect of this mechanism is that by randomly cutting off the appearance modality as a "shortcut" during training, even if the network receives complete bimodal input during the inference phase, the parameters of its backbone network have already learned to extract physiological information from the fluctuation features, rather than simply "guessing" the heart rate based on appearance features. This mechanism is activated during the training phase and deactivated during the inference phase. Regarding the "preset probability," if the preset probability is too small, the suppression effect is insufficient (loss can still be reduced through appearance features in most training steps); if the preset probability is too large, the fluctuation features are not sufficiently trained (appearance information is missing in most training steps, making it difficult for the model to converge). Those skilled in the art can determine the optimal value through conventional experiments based on the specific training set size and task characteristics.

[0051] Regularization penalty loss imposes a direct constraint on the fusion weights at the loss level: when in this batch The average value exceeds the penalty threshold (This embodiment) When the network tends to over-skew the fusion weights to RGB appearance features, an additional penalty is triggered. The two regularization methods work together—modal dropout cuts off appearance dependencies at the data level, while weight penalty constrains weight allocation at the loss level. Both aim at the same core goal: preventing the network from taking shortcuts during training. If only modal dropout is used, although the network may rely on fluctuating features in some training steps, it may still over-skew the fusion weights to RGB appearance features in training steps where they are not dropped. The weights are pushed towards 1; if only weight penalties are used, although the network is constrained, the appearance features during the inference phase may still constitute a source of overfitting. Combining both methods results in a more balanced distribution of fused weights during the inference phase. The average activation level was maintained.

[0052] The position of modality dropout in the inference method (after global context awareness and weight generation, but before weighted fusion) is specific—it does not affect the generation of context vectors or the calculation of weights (because zeroing occurs after weight generation), but only affects the input features of the fusion step. This ensures that the zeroing operation does not interfere with the gating network's ability to perceive the environment state, which is not present in regular Dropout. Weight penalty directly affects... The average value rather than the norm of the network weights constrains the fusion tendency at the modality level rather than the parameter magnitude, and its objective is fundamentally different from that of L1 / L2 regularization.

[0053] In summary, this scheme imposes targeted constraints on the end-to-end training process of the inference method by introducing two constraints during the training phase: random modality dropout and weight sparsity penalty. Through dual constraints at the data level and the loss level, it effectively suppresses the network's tendency to collapse towards static appearance features during training, ensuring the actual effectiveness of the adaptive fusion mechanism in the inference phase.

[0054] According to another aspect of the present invention, a heart rate estimation system based on dynamic gating adaptive fusion is provided, wherein the system comprises, based on the above method: The video acquisition module is used to acquire a continuous sequence of face video frames; The multimodal preprocessing and shallow feature extraction module is used to generate a corresponding frame difference sequence based on the face video frame sequence; and to perform shallow spatiotemporal feature extraction on the face video frame sequence and the frame difference sequence respectively to obtain an appearance feature map and a fluctuation feature map. A learnable spatiotemporal dynamic fusion interferometer module is configured to perform global context awareness on the appearance feature map and the fluctuation feature map to generate a context vector representing the current environmental state; and to input the context vector into a learnable mapping network to generate a first fusion weight corresponding to the appearance feature map and a second fusion weight corresponding to the fluctuation feature map; and to perform weighted fusion of the appearance feature map and the fluctuation feature map according to the first fusion weight and the second fusion weight to generate a fused feature. The spatiotemporal physiological feature backbone decoding module is used to perform deep spatiotemporal modeling on the fused features and output heart rate estimation results.

[0055] In order to better utilize the above method, this application proposes a heart rate estimation system based on dynamic gating adaptive fusion. Each module corresponds to a step of the above method, and its specific principle has been described above and will not be repeated here.

[0056] According to another aspect of the present invention, a heart rate estimation device based on dynamic gating adaptive fusion is provided, comprising: At least one processor and a memory communicatively connected to said at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described above.

[0057] In the above technical solution, to better operate and process the method, the method is stored in memory, and the processor executes the stored method. It should be noted that the principle and effect of each step have been described above and will not be elaborated upon here.

[0058] According to another aspect of the present invention, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the above-described method.

[0059] In the above technical solution, to better operate and use the method, the method is stored in a computer-readable storage medium and implemented using a processor. It should be noted that the principle and effect of each step have been described above and will not be elaborated upon here.

[0060] The technical solution provided by this invention can achieve the following beneficial effects: (1) Environmental Adaptability. Through a global context awareness and dynamic weight generation mechanism, the network can adjust the fusion weights in real time according to the environmental state of the current video segment (lighting conditions, motion amplitude, etc.). Specifically, when the network perceives extremely strong high-frequency spatial noise (such as light flicker), the gating network automatically reduces the appearance feature weight α; when it perceives large-area rigid displacement (such as large head turning), it automatically reduces the fluctuation feature weight β, thereby effectively shielding environmental noise. This mechanism is expected to reduce the average absolute error of heart rate prediction in complex uncontrolled scenarios by 20% to 30%.

[0061] (2) Suppressing appearance overfitting. By introducing a modality random dropout mechanism during the training phase, the appearance feature map is set to zero with a preset probability (e.g., 0.3), cutting off the possibility of the network "taking shortcuts" by relying on static appearance features, and forcing the backbone network to dig deeper into physiological laws from fluctuating features. Combined with the weight sparsity penalty loss, the network is mathematically restricted from tilting the fusion channel completely towards RGB appearance features, ensuring that the network learns the essence of physiological signals rather than static image features.

[0062] (3) Improve cross-domain generalization ability. Due to the above adaptive weight adjustment mechanism and effective decoupling from appearance feature dependence, the spatiotemporal features extracted by the model have stronger domain independence and can maintain stable measurement accuracy in scenarios such as cross datasets, cross skin color, and cross devices, breaking the bottleneck of model deployment limited to a single dataset.

[0063] (4) Architectural compatibility and computational efficiency. The dynamic fusion module relies only on a lightweight multilayer perceptron and global pooling operations, and its parameter count and computational cost are negligible compared to the backbone network. The fusion backbone network, as an independent front-end module, can interface with various mainstream spatiotemporal backbone networks (3D-CNN, Vision Transformer, etc.) and has plug-and-play characteristics. The modality random drop-off mechanism is only activated during the training phase and disabled during the inference phase, without affecting the running speed during actual deployment. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a typical processing flow of multimodal feature fusion stem network (Static Fusion Stem) in existing technologies; Figure 2 This is a flowchart illustrating an embodiment of a heart rate estimation method based on dynamic gating adaptive fusion according to the present invention. Figure 3 This is a flowchart illustrating an embodiment of a heart rate estimation model training method based on dynamic gating adaptive fusion according to the present invention. Figure 4 This is a schematic diagram of the data flow during regularization training to resist appearance overfitting, according to an embodiment of the heart rate estimation model training method based on dynamic gating adaptive fusion of the present invention. Figure 5 This is a schematic diagram of an embodiment of a heart rate estimation system based on dynamic gating adaptive fusion according to the present invention; Figure 6 This is a schematic diagram of the data flow of an embodiment of a heart rate estimation system based on dynamic gating adaptive fusion according to the present invention; Figure 7 This is a schematic diagram of the internal data flow of a learnable fusion backbone network according to an embodiment of a heart rate estimation system based on dynamic gating adaptive fusion of the present invention. Detailed Implementation

[0066] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Example 1 Please see Figure 2 A heart rate estimation method based on dynamic gating adaptive fusion includes: A1. Obtain a continuous sequence of face video frames; For example, a continuous sequence of facial video frames is acquired. This sequence comprises T consecutive RGB images, encompassing the spatiotemporal information of the subject's face. For instance, a continuous RGB facial video frame sequence with a length of T=150 frames, a frame rate of 30fps, and a resolution of 224×224 pixels per frame is acquired from a camera or video file. This video frame sequence, encompassing the spatiotemporal information of the subject's face, is denoted as... .

[0068] It should be noted that, in addition to using standard RGB images, the spatial appearance branch of this invention is also applicable to color spaces such as YUV and YCrCb, which are more conducive to the separation of skin tone and brightness. In specific application scenarios (such as smart cockpits and nighttime security), the RGB branch can be completely replaced with near-infrared (NIR) or thermal infrared video streams. Due to the different penetrating power and imaging mechanism of invisible light, its dynamic fusion with wave characteristics also follows the protection logic of this invention to "prevent overfitting of static features".

[0069] A2. Generate a corresponding frame difference sequence based on the face video frame sequence; In this embodiment, the frame difference sequence is generated by calculating the pixel difference between adjacent frames; or, the frame difference sequence is generated by the center difference algorithm.

[0070] For example, a corresponding frame difference sequence is generated based on the facial video frame sequence. Specifically, the frame difference sequence can be generated by calculating the pixel differences between adjacent frames, or it can be generated using a center difference algorithm. The frame difference sequence contains pixel change information between adjacent frames, including both subtle skin tone changes caused by heartbeats and displacements caused by the person's movement. For example, based on... Calculate the absolute value of the pixel difference between adjacent frames to generate a frame difference sequence. Specifically:

[0071] To maintain and Since the length is consistent across the time dimension, this example uses a padding method that sets the frame difference of the 150th frame to zero. Therefore, we obtain . Clear facial structure information and ambient lighting baselines were preserved; It captures the subtle changes in skin tone caused by the heartbeat and the movement of the figures.

[0072] It should be noted that, in addition to generating fluctuation features by subtracting adjacent frames, optical flow can also be used to extract dense motion fields as input, as optical flow can more accurately describe pixel-level micro-displacements. Alternatively, feature maps preprocessed based on Eulerian Video Magnification (EVM) or Local Phase can be used as temporal input to further highlight physiological micro-jump signals. These input sources are also applicable to the adaptive fusion framework of this invention.

[0073] A3. Perform shallow spatiotemporal feature extraction on the face video frame sequence and the frame difference sequence respectively to obtain appearance feature map and fluctuation feature map; In this embodiment, the shallow spatiotemporal feature extraction is performed by two shallow 3D convolutional modules that do not share weights on the face video frame sequence and the frame difference sequence, respectively.

[0074] For example, shallow spatiotemporal feature extraction is performed on the face video frame sequence and the frame difference sequence respectively to obtain appearance feature maps and fluctuation feature maps. Specifically, the face video frame sequence and the frame difference sequence are respectively input into two shallow 3D convolutional modules, and preliminary downsampling is performed in the spatiotemporal dimension to output appearance feature maps respectively. And fluctuation characteristic diagram In one embodiment, the two shallow 3D convolutional modules have the same structure but do not share weights. For example, ... and Two shallow 3D convolutional modules are input separately. The two modules have the same structure (3×3×3 kernel size, stride of 2, and 64 output channels), but do not share weights. After shallow 3D convolution processing, the appearance feature map is obtained. And fluctuation characteristic diagram .

[0075] A4. Perform global context awareness on the appearance feature map and the fluctuation feature map to generate a context vector representing the current environmental state; In this embodiment, the global context awareness includes: concatenating the appearance feature map and the fluctuation feature map along the channel dimension, and then compressing the spatiotemporal dimension through a three-dimensional global average pooling layer to generate the context vector.

[0076] For example, global context awareness is applied to the appearance feature map and the fluctuation feature map to generate a context vector representing the current environmental state. Specifically, the appearance feature map and the fluctuation feature map are concatenated along the channel dimension, and then compressed in the spatiotemporal dimension through a three-dimensional global average pooling layer to generate a global context vector. This context vector reflects the overall state of the current video clip in both spatial and temporal dimensions, including environmental information such as lighting conditions and the intensity of motion. For example, ... and By splicing along the channel dimension, we obtain .Will The input is a 3D global average pooling layer (3D-GAP), which performs global average pooling in three dimensions: time (75 frames), spatial height (112 pixels), and spatial width (112 pixels), compressing it into a global context vector. This global context vector comprehensively represents the environmental state of the current video segment—including lighting conditions (such as whether there is flicker causing sudden changes in brightness) and motion amplitude (such as whether the subject turns their head sharply). This information directly affects the subsequent dynamic weight allocation.

[0077] A5. Input the context vector into a learnable mapping network to generate a first fusion weight corresponding to the appearance feature map and a second fusion weight corresponding to the fluctuation feature map; In this embodiment, the learnable mapping network is a gating network, used to output the first fusion weight and the second fusion weight based on the context vector.

[0078] In this embodiment, the gated network is a multilayer perceptron gated network, which includes a dimensionality reduction layer and a dimensionality increase layer. After the context vector is sequentially reduced and increased in dimensionality, the first fusion weight and the second fusion weight are output through a Softmax or Sigmoid activation function.

[0079] For example, the context vector is input into a learnable mapping network to generate a first fusion weight corresponding to the appearance feature map and a second fusion weight corresponding to the fluctuation feature map.

[0080] In one example, the learnable mapping network is a gating network.

[0081] In one example, the gating network is a multilayer perceptron (MLP) gating network. After dimensionality reduction and dimensionality increase operations, the context vector is output as dynamically normalized first fusion weights through an activation function. Second fusion weight In the embodiment employing the Softmax activation function, and satisfy In embodiments employing the Sigmoid activation function, and Each independently located Within the interval. For example, The input is a gated network consisting of a multilayer perceptron (MLP). This network includes a dimensionality reduction fully connected layer (128-dimensional → 8-dimensional, with ReLU activation function) and a dimensionality increase fully connected layer (8-dimensional → 128-dimensional). After dimensionality increase, a pair of dynamically normalized weights is output through the Softmax activation function. and ,satisfy .in, For appearance feature diagram The first fusion weight, For fluctuation characteristic diagram The second fusion weight. In the test of this embodiment, Strong light flicker noise was detected, and the gating network output... , That is, adaptively reducing the weight of contaminated appearance features and increasing the weight of relatively reliable fluctuation features.

[0082] It should be noted that, in addition to using Global Average Pooling (GAP) in conjunction with Multilayer Perceptron (MLP) to generate scalar weights... and In addition, channel-space attention mechanisms (CBAM or SE-Net) can be introduced. That is, instead of assigning weights at the global modality level, a 3D weight tensor with the same dimension as the feature map is generated to achieve adaptive dynamic fusion at the pixel and channel levels (e.g., dynamically assigning high Diff weights to the cheek region and low weights to the static background region).

[0083] A6. Based on the first fusion weight and the second fusion weight, the appearance feature map and the fluctuation feature map are weighted and fused to generate a fused feature; In this embodiment, the weighted fusion is an element-wise weighted summation, and the fusion feature... ,in, The first fusion weight, This is the second fusion weight.

[0084] For example, the appearance feature map and the fluctuation feature map are weighted and fused according to the first fusion weight and the second fusion weight to generate a fused feature. Specifically, the fused feature is generated by element-wise weighted summation. For example, based on the generated dynamic weights, the appearance feature map and the fluctuation feature map are summed element-wise to generate a fused feature: ,in This fusion feature was then fully preserved and passed on to the backbone network.

[0085] It's worth noting that besides weighted addition, cross-attention can also be used. This involves using the diff fluctuation features as the query and the RGB appearance features as the key and value, dynamically retrieving and fusing spatial context information through the attention mechanism. Alternatively, dynamic convolution can be employed, transforming the environmental context vector into convolution kernel parameters and directly performing sliding convolution extraction on the concatenated bimodal feature map.

[0086] A7. Perform deep spatiotemporal modeling on the fused features and output the heart rate estimation results.

[0087] For example, deep spatiotemporal modeling is performed on the fused features to output heart rate estimation results. Specifically, the fused features... The data is fed into the backbone network for deep spatiotemporal dependency modeling. After extracting the pure rPPG feature sequence, it is mapped through a one-dimensional convolutional layer or fully connected layer to output a one-dimensional time series prediction signal. Then, the specific heart rate value (BPM) is calculated through peak detection or frequency domain analysis. For example, ... The input is a 3D-ResNet backbone network (containing 4 residual blocks, each containing 2 3D convolutional layers) to perform deep spatiotemporal dependency modeling and extract clean rPPG feature sequences. Then, a 1D convolutional layer maps the high-dimensional features to a 1D time-series prediction signal. Then, the specific heart rate value is calculated using a peak detection algorithm.

[0088] Example 2 Please see Figure 3 , Figure 4 A training method for a heart rate estimation model based on dynamic gating adaptive fusion is provided, which is executed during the training phase before model deployment. The method includes: B1. Obtain the sample face video frame sequence and the corresponding real heart rate label; For example, sample face video frame sequences and their corresponding real heart rate labels are obtained from publicly available rPPG datasets (such as PURE, UBFC-rPPG, etc.) (synchronously acquired electrocardiogram signals are used as the gold standard). A training batch (BatchSize=16) is taken, with each sample containing a T=150-frame face video frame sequence. and the corresponding true heart rate label .

[0089] B2. Generate the corresponding frame difference sequence based on the sample face video frame sequence; For example, according to Generate frame difference sequence (The method is the same as in Example 1A2); B3. Perform shallow spatiotemporal feature extraction on the sample face video frame sequence and the frame difference sequence respectively to obtain appearance feature map and fluctuation feature map; For example, will and Inputting each layer into a shallow 3D convolutional module yields the following results: and (The method is the same as in Example 1, A3); B4. Perform global context awareness on the appearance feature map and the fluctuation feature map to generate a context vector representing the current environmental state; For example, for and Perform global context awareness, and obtain 3D-GAP (The method is the same as in Example 1, A4); B5. Input the context vector into a learnable mapping network to generate a first fusion weight corresponding to the appearance feature map and a second fusion weight corresponding to the fluctuation feature map; For example, will Input to an MLP gating network, output via Softmax and ( (The method is the same as in Example 1, A5); B6. Set the appearance feature map to zero with a preset probability; in this embodiment, the preset probability is 0.3. The value of this preset probability can be adjusted within the range of 0.1 to 0.5: if the probability value is too small, it will not be sufficient to effectively suppress appearance overfitting; if the probability value is too large, the fluctuation features may not have been sufficiently trained, affecting the model's ability to extract physiological signals. Those skilled in the art can determine the specific probability value through conventional experiments based on the specific training set size and task characteristics.

[0090] For example, with a preset probability Will Zeroing: In each training batch, for each sample, independently zero out the zeros with a 30% probability. Replace with a zero matrix, maintaining it with a 70% probability. constant.

[0091] It's worth noting that besides directly setting the RGB features to zero, modality noise injection can also be used. This involves, during the training phase, adding high-variance Gaussian noise to the shallow RGB features with a certain probability, or applying strong spatial cutouts to disrupt their structural integrity. This can also force the network to shift its attention to the more robust fluctuating features.

[0092] B7. Based on the first fusion weight and the second fusion weight, the processed appearance feature map and the fluctuation feature map are weighted and fused to generate a fused feature; For example, weighted fusion is performed based on weights: if If set to zero, then If it is not set to zero, then The direct effect of the zeroing operation is to make the weighted fusion process more efficient. That is, the fluctuation characteristics become the only source of information, thus forcing the backbone network to extract physiological patterns from frame difference characteristics (in the same way as in Example 1A6). B8. Perform deep spatiotemporal modeling on the fused features to obtain the predicted heart rate estimation result; Will The input backbone network (3D-ResNet) is used for deep spatiotemporal modeling, and the predicted heart rate signal is output after mapping through a one-dimensional convolutional layer. (The method is the same as in Example 1 A7).

[0093] B9. Calculate the prediction loss between the predicted heart rate estimation result and the true heart rate label, and calculate the regularization penalty loss for the first fusion weight; For example, calculate the total loss :

[0094] in, This refers to the loss of the negative Pearson correlation coefficient between the predicted heart rate signal and the true label; That is, when in this batch A regularization penalty term is triggered when the average value exceeds 0.7; λ=0.1, which is the weight coefficient of the penalty term; for example, in this training batch, if ,but This penalty mechanism ensures that the network cannot indefinitely tilt the fusion weights toward RGB appearance features, forcing... Always maintain a meaningful level of activation.

[0095] It should be noted that, in addition to using weighted penalty loss, contrastive learning loss can also be introduced. In the feature space, the fused features are maximized. With pure wave characteristics The mutual information between features (using InfoNCE Loss) is used to distance the network from pure appearance features, thereby strictly constraining the network from the feature manifold level to avoid fitting to static appearance.

[0096] In addition, a gradient reversal layer (GRL) can be introduced in conjunction with domain adversarial training. Different subject IDs (identity appearances) in the training set are treated as different domains, forcing the pre-fusion network to "remove" static appearance information related to the subject's identity (so that the discriminator cannot distinguish which person the current feature belongs to), thereby preserving the pure heart rate cycle signal.

[0097] B10. Update the parameters of the heart rate estimation model based on the weighted sum of the prediction loss and the regularization penalty loss.

[0098] according to Calculate the gradient and update all trainable parameters in the heart rate estimation model (including parameters of shallow 3D convolutional modules, MLP gated networks, backbone networks, etc.) using the backpropagation algorithm to complete one training step iteration. Repeat the above process until the model converges.

[0099] After training is complete, the modality random drop-off mechanism is turned off (i.e., during the inference phase, no further modality drops are performed). (Perform zeroing operation), fused features restored to This allows for full utilization of bimodal information to achieve heart rate estimation. The trained model can then be deployed and used as described in Example 1.

[0100] This training method forces the model to retain the highly difficult-to-learn diff features during adaptive fusion, preventing it from degenerating into an ineffective model that relies solely on rote memorization of RGB appearances. Modality Dropout and WeightPenalty Loss are introduced during the model's backpropagation training phase.

[0101] During training, the former forces the appearance features to be included before the fusion step with a set probability p (p=0.3). Zeroing out the data. This Dropout-like operation prevents the network from always relying on easily obtainable appearance features such as skin color and face shape during training. It is "forced" to delve deeper and learn frame difference features. This involves extremely subtle physiological patterns. During the inference phase, this mechanism is disabled, allowing the network to fully benefit from the robustness provided by the bimodal approach. In the overall loss function, besides the negative Pearson correlation coefficient loss used for heartbeat waveform regression, an additional RGB-gated weight is introduced. Regularization penalty term .if If the average value is too large (approaching 1) in a particular batch, the penalty is increased. This mathematically limits the network's tendency to completely tilt the fusion channels towards RGB, ensuring... The (fluctuation feature weights) are always maintained at a meaningful activation level. By combining the above architecture and training strategy, a highly dynamic, environmentally resistant, and feature-representation-capable rPPG heart rate estimation system was successfully implemented.

[0102] Example 3 Please see Figure 5 , Figure 6 A heart rate estimation system based on dynamic gating adaptive fusion, based on the method described in one embodiment, the system comprising: The video acquisition module is used to acquire a continuous sequence of face video frames; The multimodal preprocessing and shallow feature extraction module is used to generate a corresponding frame difference sequence based on the face video frame sequence; and to perform shallow spatiotemporal feature extraction on the face video frame sequence and the frame difference sequence respectively to obtain an appearance feature map and a fluctuation feature map. For example, this module is responsible for transforming the original single video stream into a decoupled "appearance space" and "temporal fluctuation" bimodal input, and extracting the initial shallow high-dimensional features.

[0103] Dual-modal data stream construction: Receive a sequence of face video frames of length T. By calculating the pixel differences between adjacent frames or using a center difference algorithm, a corresponding frame difference sequence is generated. .at this time, It provides a clear baseline for facial structure and ambient lighting, while It captured subtle changes in skin tone caused by heartbeats and the movement of the characters.

[0104] Independent shallow mapping: and The inputs are fed into two shallow 3D convolutional modules (Stem-3DCNN) with identical structures but no shared weights. These two modules perform initial downsampling in the spatiotemporal dimensions and output appearance feature maps respectively. And fluctuation characteristic diagram .

[0105] A learnable spatiotemporal dynamic fusion interferometer module is configured to perform global context awareness on the appearance feature map and the fluctuation feature map to generate a context vector representing the current environmental state; and to input the context vector into a learnable mapping network to generate a first fusion weight corresponding to the appearance feature map and a second fusion weight corresponding to the fluctuation feature map; and to perform weighted fusion of the appearance feature map and the fluctuation feature map according to the first fusion weight and the second fusion weight to generate a fused feature. Please see the appendix Figure 7 This diagram illustrates in detail the internal data flow of the learnable fusion backbone network. This module is the core structural innovation of this invention, responsible for dynamically calculating and allocating the fusion weights of the two modalities based on the context of the current video segment (sudden changes in lighting, violent motion).

[0106] Global Context-Aware Embedding: First, the appearance feature map is... And fluctuation characteristic diagram By concatenating along the channel dimension and passing through a three-dimensional global average pooling layer (3D-GAP), the extremely high-dimensional spatiotemporal features are compressed into a global context vector that perceives the current environmental state. .

[0107] Dynamic Gating Weight Generation: This involves generating context vectors... The input is fed into a gated network consisting of a multilayer perceptron (MLP). This network undergoes dimensionality reduction and expansion operations, and outputs a pair of dynamically normalized weights using either a Softmax or Sigmoid activation function. and (and satisfy) + = 1 or each dynamically scales within the [0,1] interval). When the network senses extremely strong high-frequency spatial noise (light flicker), the gating network automatically reduces... (RGB weighting); When a large-area rigid displacement (large head rotation) is detected, the weighting is automatically reduced. (Diff weights).

[0108] Adaptive Weighted Fusion: Utilizes generated dynamic weights to perform element-wise weighted summation on the original shallow features, generating the final fused feature tensor. This fusion feature was then fully preserved and passed on to the backbone network.

[0109] The spatiotemporal physiological feature backbone decoding module is used to perform deep spatiotemporal modeling on the fused features and output heart rate estimation results.

[0110] For example, receiving high-quality features after fusion. It performs in-depth spatiotemporal dependency modeling and outputs predicted heartbeat waveforms. The backend of this invention is compatible with various mainstream architectures (3D-ResNet or Vision Transformer). After extracting the pure rPPG feature sequence through multiple deep networks, a one-dimensional time series prediction signal is output through a one-dimensional convolutional (1D-CNN) head or a fully connected layer. The specific heart rate value (BPM) is then calculated through peak detection or frequency domain analysis.

[0111] Compared with existing video physiological measurement networks that mainly use static concatenation or element-wise addition for multimodal feature fusion, the heart rate estimation neural network construction method and system with a learnable spatiotemporal feature fusion module provided by this invention has the following significant advantages: 1. It has a strong ability to adapt to environmental context, which significantly improves the measurement robustness in complex lighting and violent motion scenarios.

[0112] Existing technologies use fixed feature fusion weights, which cannot cope with complex and ever-changing real-world environments. When there is flickering light outside the car window, the RGB features will introduce huge noise; when the subject turns their head violently, the frame difference feature (Diff) will be filled with rigid displacement noise, causing traditional networks to collapse instantly.

[0113] This invention utilizes a unique Learnable Dynamic Fusion Stem to extract global spatiotemporal context vectors in real time and dynamically output fusion weights α and β. The model acts like a "smart valve," adaptively reducing the fusion ratio of appearance features during dramatic changes in lighting and dynamically attenuating contaminated fluctuation features during strenuous exercise. This mechanism of dynamically shielding environmental noise is expected to reduce the mean absolute error (MAE) of heart rate prediction by 20% to 30% in complex, uncontrolled scenarios, significantly expanding the application boundaries of non-contact vital sign monitoring.

[0114] 2. The unique modality discarding and regularization mechanism completely overcomes the industry pain point of "appearance overfitting (network laziness)" in rPPG networks.

[0115] In the training process, existing technologies, due to the lack of independent constraints on multimodal branches, tend to fit the static appearance features (specific skin color, facial structure) that are easiest to learn, while ignoring the extremely weak skin micro-jump signals that truly represent heartbeats. As a result, what is learned is only the "facial appearance" rather than the "physiological essence".

[0116] This invention introduces a modality dropout mechanism and gated weight penalty loss during the training phase. By forcibly setting the RGB spatial features to zero with a certain probability (p=0.3), the possibility of the network "taking shortcuts" by relying on static appearance is completely eliminated, forcing the backbone network to deeply mine the micro-circulatory fluctuation patterns from the frame difference temporal features. This mechanism ensures that the features extracted by the network have extremely high physiological signal purity and fidelity.

[0117] 3. It has achieved a qualitative leap in cross-domain generalization ability, breaking the bottleneck of model deployment limited to a single dataset.

[0118] Existing static fusion networks, due to their deep binding to the specific environmental lighting and skin color distribution of the training set (Source Domain), will experience a precipitous drop in prediction accuracy once deployed to a new scene (Target Domain) with completely different lighting conditions, camera sensors, or skin colors.

[0119] This invention benefits from the dynamic adjustment of adaptive weights and strong decoupling from dependence on appearance features, resulting in "domain-independent" spatiotemporal features. Experimental verification shows that in cross-dataset cross-validation tests, the predicted waveforms of this invention exhibit a significant improvement in Pearson correlation coefficients between real electrocardiograms (ECGs) and existing static networks, truly meeting the high standard requirement of "one-time training, multi-scenario generalization deployment" for commercial products.

[0120] 4. The model architecture has extremely high compatibility and does not increase the additional computing power burden in the inference stage, achieving a balance between high performance and lightweight design.

[0121] Existing technologies often require the introduction of extremely complex cascaded denoising networks or cumbersome pre-processing image algorithms to address environmental interference, which severely slows down the system's operating speed.

[0122] The learnable fusion mechanism of this invention relies solely on a lightweight multilayer perceptron (MLP) and global pooling operations, with a negligible number of parameters and computational complexity (FLOPs) compared to the backbone network. More importantly, the core "modal dropout mechanism" is activated only during offline training and directly disabled during actual online inference, allowing the network to run at full speed. Furthermore, the Fusion Stem of this invention, as an independent front-end module, possesses strong plug-and-play characteristics, seamlessly integrating with any mainstream spatiotemporal backbone network (3D-CNN, Vision Transformer, or Mamba), demonstrating high engineering practicality and widespread application value.

[0123] Example 4 A heart rate estimation device based on dynamic gating adaptive fusion includes: At least one processor and a memory communicatively connected to said at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method as described in one of the embodiments.

[0124] In this embodiment, to better run and process the method described in one of the embodiments, the above method is stored in a memory, and the stored method is executed using a processor. It should be noted that the principle and effect of each step have been described above and will not be elaborated further here.

[0125] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A heart rate estimation method based on dynamic gating adaptive fusion, characterized in that, The method includes: Obtain a continuous sequence of face video frames; Generate a corresponding frame difference sequence based on the face video frame sequence; Shallow spatiotemporal feature extraction is performed on the face video frame sequence and the frame difference sequence to obtain appearance feature map and fluctuation feature map, respectively. Global context awareness is performed on the appearance feature map and the fluctuation feature map to generate a context vector representing the current environmental state; The context vector is input into a learnable mapping network to generate a first fusion weight corresponding to the appearance feature map and a second fusion weight corresponding to the fluctuation feature map; Based on the first fusion weight and the second fusion weight, the appearance feature map and the fluctuation feature map are weighted and fused to generate a fused feature; Deep spatiotemporal modeling is performed on the fused features to output heart rate estimation results.

2. The heart rate estimation method based on dynamic gating adaptive fusion as described in claim 1, characterized in that, The learnable mapping network is a gated network used to output the first fusion weight and the second fusion weight based on the context vector.

3. The heart rate estimation method based on dynamic gating adaptive fusion as described in claim 2, characterized in that, The gated network is a multilayer perceptron gated network, which includes a dimensionality reduction layer and a dimensionality increase layer. After the context vector is successively reduced and increased in dimensionality, the first fusion weight and the second fusion weight are output through a Softmax or Sigmoid activation function.

4. The heart rate estimation method based on dynamic gating adaptive fusion as described in claim 1, characterized in that, The global context awareness includes: concatenating the appearance feature map and the fluctuation feature map along the channel dimension, and then compressing the spatiotemporal dimension through a three-dimensional global average pooling layer to generate the context vector.

5. The heart rate estimation method based on dynamic gating adaptive fusion as described in claim 1, characterized in that, The weighted fusion is an element-wise weighted summation, and the fusion feature... ,in, The first fusion weight, This is the second fusion weight.

6. The heart rate estimation method based on dynamic gating adaptive fusion as described in claim 1, characterized in that, The frame difference sequence is generated by calculating the pixel difference between adjacent frames; or, the frame difference sequence is generated by the center difference algorithm.

7. The heart rate estimation method based on dynamic gating adaptive fusion as described in claim 1, characterized in that, The shallow spatiotemporal feature extraction is performed by two shallow 3D convolutional modules that do not share weights on the face video frame sequence and the frame difference sequence, respectively.

8. A training method for a heart rate estimation model based on dynamic gating adaptive fusion, characterized in that, The method includes: Obtain the sample face video frame sequence and the corresponding real heart rate label; Generate a corresponding frame difference sequence based on the sample face video frame sequence; Shallow spatiotemporal feature extraction is performed on the sample face video frame sequence and the frame difference sequence to obtain appearance feature map and fluctuation feature map; Global context awareness is performed on the appearance feature map and the fluctuation feature map to generate a context vector representing the current environmental state; The context vector is input into a learnable mapping network to generate a first fusion weight corresponding to the appearance feature map and a second fusion weight corresponding to the fluctuation feature map; The appearance feature map is set to zero with a preset probability; or, Gaussian noise is superimposed on the appearance feature map with a preset probability; or, spatial perturbation block occlusion is applied to the appearance feature map with a preset probability. Based on the first fusion weight and the second fusion weight, the processed appearance feature map and the fluctuation feature map are weighted and fused to generate a fused feature; Deep spatiotemporal modeling is performed on the fused features to obtain the predicted heart rate estimation result; Calculate the prediction loss between the predicted heart rate estimate and the true heart rate label, and calculate the regularization penalty loss for the first fusion weight; The parameters of the heart rate estimation model are updated based on the weighted sum of the prediction loss and the regularization penalty loss.

9. A heart rate estimation system based on dynamic gating adaptive fusion, characterized in that, Based on the method according to any one of claims 1 to 7, the system comprises: The video acquisition module is used to acquire a continuous sequence of face video frames; The multimodal preprocessing and shallow feature extraction module is used to generate a corresponding frame difference sequence based on the face video frame sequence; and to perform shallow spatiotemporal feature extraction on the face video frame sequence and the frame difference sequence respectively to obtain an appearance feature map and a fluctuation feature map. A learnable spatiotemporal dynamic fusion interferometer module is configured to perform global context awareness on the appearance feature map and the fluctuation feature map to generate a context vector representing the current environmental state; and to input the context vector into a learnable mapping network to generate a first fusion weight corresponding to the appearance feature map and a second fusion weight corresponding to the fluctuation feature map; and to perform weighted fusion of the appearance feature map and the fluctuation feature map according to the first fusion weight and the second fusion weight to generate a fused feature. The spatiotemporal physiological feature backbone decoding module is used to perform deep spatiotemporal modeling on the fused features and output heart rate estimation results.

10. A heart rate estimation device based on dynamic gating adaptive fusion, characterized in that, include: At least one processor and a memory communicatively connected to said at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method as described in any one of claims 1 to 7.

Images