Contactless physiological monitoring method and apparatus based on cbam attention mechanism and bidirectional mamba rppg signal extraction network

Abstract

The application discloses a rPPG signal extraction network based on a CBAM attention mechanism and a bidirectional Mamba, and a non-contact physiological monitoring method and device, wherein spatial noise is filtered by introducing a CBAM attention mechanism, long time sequence dependence is modeled at low cost by using a Mamba architecture, and signal purity is improved by combining a frequency domain and a confidence gate. The rPPG signal extraction network effectively extracts weak skin color change features through an adaptive differential fusion mechanism of the fusion backbone network, realizes long time sequence global modeling through a Mamba architecture (with linear calculation complexity) in the space-time encoder, performs double cleaning on the features in the spatial domain and the frequency domain through a confidence perception gate module and a frequency domain gate module, and finally reconstructs a high-fidelity rPPG waveform through the bidirectional Mamba regression head.

Description

Technical Field

[0001] This invention relates to computer vision and biomedical signal processing technology, and more particularly to an rPPG signal extraction network based on the CBAM attention mechanism and bidirectional Mamba, as well as a non-contact physiological monitoring method and device. Background Technology

[0002] In recent years, physiological signal monitoring technology based on remote photoplethysmography (rPPG) has become a research hotspot. This technology can capture subtle periodic changes in skin color caused by the periodic beating of the heart using a regular camera, thereby non-contactly extracting blood volume pulse waves and measuring physiological indicators related to the heart cycle, such as heart rate, respiratory rate, and heart rate variability. Compared with traditional contact sensors, rPPG has advantages such as being non-contact, portable, and low-cost, and has broad application prospects in fields such as remote medical diagnosis, driver status monitoring, and emotion computing.

[0003] Early rPPG methods primarily relied on blind source separation (such as ICA and PCA) or hand-designed algorithms based on chromaticity features. In recent years, deep learning has become mainstream. Existing deep learning methods are mainly divided into two categories: methods based on 3D convolutional neural networks (3D-CNN) and methods based on Transformers. 3D-CNN utilizes convolutional kernels to extract features in the spatiotemporal dimension, exhibiting excellent performance in short-term feature extraction; Transformers utilize self-attention mechanisms to capture global features, solving the problem of long-range dependencies.

[0004] However, existing technical solutions have the following significant drawbacks in practical applications: The trade-off between long-term modeling capability and computational efficiency: rPPG signals are a typical long-term periodic signal. While 3D-CNNs have low computational complexity, they are limited by their local receptive field, making it difficult to capture the global periodicity of long videos; while the problem with Transformers is that their computational complexity increases quadratically with video length. This results in a large number of model parameters, a huge amount of computation, and a slow inference speed.

[0005] Insufficient robustness in complex environments: In uncontrolled scenarios, videos contain a large amount of non-skin background (such as walls and clothing) and interference from the subject's head movements. Many existing networks lack effective spatial attention mechanisms, causing models to easily misinterpret background noise as physiological signals. Although some methods use simple attention modules, they often fail to achieve fine-grained filtering of channels and spatial dimensions, making it difficult to maintain high robustness under drastic motion or changes in lighting.

[0006] Lack of frequency domain constraints and confidence awareness: Most existing models only regress signals in the time domain, ignoring the sparsity and regularity of rPPG signals in the frequency domain. In addition, for low-quality segments in the video (such as occlusions or large-angle head turns), existing models lack a confidence evaluation mechanism, often forcibly extracting features, introducing a lot of noise, making it difficult for the model to converge, and reducing the accuracy of the final measurement.

[0007] In summary, designing an rPPG extraction network that combines linear computational complexity (low cost) and global modeling capability (high accuracy) while effectively suppressing background and motion noise is a problem that urgently needs to be solved in the current technological field. Summary of the Invention

[0008] To address the shortcomings of the existing technologies, this invention provides an rPPG signal extraction network based on the CBAM attention mechanism and bidirectional Mamba, as well as a non-contact physiological monitoring method and device. By introducing the CBAM attention mechanism to filter spatial noise, utilizing the Mamba architecture to model long-term time-series dependencies at low cost, and combining frequency domain and confidence gating to improve signal purity.

[0009] The technical problem to be solved by the present invention is achieved through the following technical solution: The rPPG signal extraction network based on the CBAM attention mechanism and bidirectional Mamba includes: A fusion backbone network is used to receive face video tensors. A multi-scale differential feature containing static appearance features and dynamic differential features is constructed on the face video tensors through a temporal differential mechanism. The static appearance features and dynamic differential features are adaptively fused to output shallow spatiotemporal fusion features. The spatiotemporal encoder, connected to the fusion backbone network, is used to receive the shallow spatiotemporal fusion features, perform multi-scale temporal modeling through a time pyramid architecture, and output deep spatiotemporal features. The confidence perception gating module is connected to the spatiotemporal encoder and is used to receive the deep spatiotemporal features, generate a spatiotemporal confidence map, and perform weighted cleaning and spatial aggregation on the deep spatiotemporal features based on the spatiotemporal confidence map, and output the aggregated two-dimensional temporal features. The frequency domain gating module, connected to the confidence perception gating module, is used to receive the two-dimensional time-series features, transform them to the frequency domain, filter out non-physiological frequency band noise, restore them to the time domain, and output time-domain enhanced features. A bidirectional Mamba regression head, connected to the frequency domain gating module, is used to receive the time domain enhancement features, capture the time-series dependencies through a bidirectional state-space model, and map them into a one-dimensional remote photoplethysmography (PPG) signal.

[0010] Furthermore, the converged backbone network includes: The temporal difference construction module is used to perform temporal misalignment slicing on the face video tensor, calculate the pixel difference between the current frame and the neighboring frames, and splice them to generate a multi-channel difference video tensor. The dual-branch feature extraction layer includes an appearance feature extraction branch and a motion feature extraction branch. The appearance feature extraction branch is used to process the face video tensor to extract static appearance features, and the motion feature extraction branch is used to process the difference video tensor to extract dynamic difference features. An adaptive fusion layer is used to introduce learnable first and second scalar parameters, as well as a first fusion transformation network and a second fusion transformation network, to adaptively fuse the static appearance features and dynamic difference features according to a preset adaptive fusion function, and output shallow spatiotemporal fusion features.

[0011] Furthermore, the spatiotemporal encoder includes: The slice embedding layer is used to perform 3D convolution processing on the input shallow spatiotemporal fusion features, mapping their video voxels into high-dimensional feature vectors and performing spatial downsampling; The system comprises multiple cascaded processing stages, each including a temporal downsampling layer and stacked multiple improved spatiotemporal Mamba modules. These modules are used to progressively downsample the temporal dimension of the high-dimensional feature vector while maintaining constant spatial resolution and channel number, extracting deep spatiotemporal abstract features from local transient changes to global periodic patterns, and outputting the deep spatiotemporal features.

[0012] Furthermore, the improved spatiotemporal Mamba module includes: The central difference convolutional layer is used to calculate the intensity difference between neighboring pixels and the central pixel within the receptive field of the convolutional kernel, introducing gradient information to enhance the kernel's sensitivity to subtle texture details. The 3DCBAM attention module, connected to the central differential convolutional layer, is used to sequentially perform channel attention weighting and spatial attention weighting on the feature map output by the convolution, adaptively enhancing the channel containing the pulse signal and brightening the facial skin area. The bidirectional Mamba layer, connected to the 3DCBAM attention module, is used to flatten the attention-weighted feature map in the spatial dimension and convert it into a sequence format. It captures causal historical dependencies and inverse future dependencies through forward scanning paths and backward scanning paths, respectively, and outputs the deep spatiotemporal features.

[0013] Furthermore, the 3DCBAM attention module includes: The channel attention submodule is used to perform global average pooling and global max pooling on the input feature map to compress the spatiotemporal dimension. It learns the importance weights of each channel by sharing a multilayer perceptron, generates channel weights, and multiplies the channel weights with the original input feature map element-wise. The spatial attention submodule, connected to the channel attention submodule, is used to generate a spatial attention map using 3D convolution based on the channel-weighted feature map, and then perform element-wise multiplication of the spatial attention map with the original input feature map.

[0014] Furthermore, the confidence-aware gating module includes: The confidence prediction network is used to perform three-dimensional convolution and activation processing on the input deep spatiotemporal features to generate a spatiotemporal confidence map. The confidence weighting module, connected to the confidence prediction network, is used to perform layer normalization and multilayer perceptron projection on the deep spatiotemporal features, then perform element-wise multiplication with the spatiotemporal confidence map, and add it to the original input deep spatiotemporal features through residual connection to output the refined deep spatiotemporal features. The spatial aggregation module, connected to the confidence weighting module, is used to perform weighted average pooling on the refined deep spatiotemporal features according to a preset spatial aggregation function, using the spatiotemporal confidence map as spatial weights, to aggregate the four-dimensional spatiotemporal features into two-dimensional time-series features, and output the aggregated two-dimensional time-series features.

[0015] Furthermore, the frequency domain gating module includes: The Fourier transform unit is used to transform the input two-dimensional time-series features from the time domain to the frequency domain through the fast Fourier transform, to obtain the amplitude spectrum and phase spectrum. A frequency-domain gated network, connected to the Fourier transform unit, is used to adaptively compress the frequency dimension of the amplitude spectrum using average pooling. A frequency mask is generated through a one-dimensional convolutional layer and a Sigmoid activation function. The frequency mask is then multiplied with the amplitude spectrum before compression and interpolated and filtered to restore the original frequency resolution. The inverse Fourier transform unit, connected to the frequency domain gated network, is used to combine the filtered amplitude spectrum with the phase spectrum and restore it to the time domain through inverse fast Fourier transform, and output the restored time domain features. The temporal residual connection unit, connected to the inverse Fourier transform unit, is used to add the temporal feature to the two-dimensional temporal feature and output the temporal enhancement feature.

[0016] Furthermore, the bidirectional Mamba return header includes: The forward Mamba branch is used to perform a forward scan on the input temporal augmentation features to capture causal history dependencies in the time series signal; The backward Mamba branch is used to perform a backward scan on the input time-domain augmentation features to capture the inverse future dependencies in the time-series signal; The Mamba fusion layer, connected to the forward Mamba branch and the backward Mamba branch, is used to fuse the outputs of the two branches to obtain high-dimensional fusion features; A linear projection layer, connected to the Mamba fusion layer, is used to map the high-dimensional fusion features into a one-dimensional remote photoplethysmography (PPG) waveform signal.

[0017] A non-contact physiological monitoring method includes the following steps: Step 1: Obtain a continuous video frame sequence of the monitored object, and perform face extraction and normalization processing on each video image in the video frame sequence to obtain the face video tensor of the video frame sequence; Step 2: Input the face video tensor into the above-mentioned rPPG signal extraction network, and receive the remote photoplethysmography (PPG) signal output by the rPPG signal extraction network; Step 3: Analyze the remote photoplethysmography (PPG) signal to extract physiological indicators related to the heart cycle of the monitored object.

[0018] A non-contact physiological monitoring device includes a processor and a memory connected to each other. The memory stores the aforementioned rPPG signal extraction network and a computer program for the processor to execute. When the processor executes the computer program, it retrieves the rPPG signal extraction network from the memory to perform the aforementioned remote photoplethysmography (PPG) extraction method.

[0019] The present invention has the following beneficial effects: The rPPG signal extraction network of this invention effectively extracts subtle skin color variation features through the adaptive differential fusion mechanism of the fusion backbone network, achieves long-term global modeling through the Mamba architecture (with linear computational complexity) in the spatiotemporal encoder, performs dual feature cleaning in the spatial and frequency domains through the confidence-aware gating module and the frequency domain gating module, and finally reconstructs high-fidelity rPPG waveforms by the bidirectional Mamba regression head. The synergistic effect of the above modules enables the network to have powerful long-term dependency modeling capabilities while maintaining low computational complexity.

[0020] The rPPG signal extraction network of the present invention generates a spatiotemporal confidence map through the confidence perception gating module, performs soft suppression on low-quality regions, and combines the frequency domain gating module to adaptively filter non-physiological frequency noise, effectively improving the robustness of the network in complex environments (motion, occlusion, and changes in illumination).

[0021] The rPPG signal extraction network of the present invention benefits from the linear computational complexity of the Mamba architecture and the efficient design of the network as a whole. It has a small number of parameters, fast inference speed, and can be deployed on ordinary computing devices to realize real-time non-contact physiological monitoring. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the primary architecture and data flow of the rPPG signal extraction network provided by the present invention.

[0023] Figure 2 This is a schematic diagram of the two-level architecture and data flow of the fusion backbone network in the rPPG signal extraction network provided by the present invention.

[0024] Figure 3 This is a schematic diagram of the two-level architecture and data flow of the spatiotemporal encoder in the rPPG signal extraction network provided by the present invention.

[0025] Figure 4 This is a schematic diagram of the two-level architecture and data flow of the confidence-aware gating module in the rPPG signal extraction network provided by the present invention.

[0026] Figure 5 This is a schematic diagram of the two-level architecture and data flow of the frequency domain gating module in the rPPG signal extraction network provided by the present invention.

[0027] Figure 6 This is a schematic diagram of the two-level architecture and data flow of the bidirectional Mamba regression head in the rPPG signal extraction network provided by the present invention.

[0028] Figure 7 This is a flowchart illustrating the steps of the non-contact physiological monitoring method provided by the present invention. Detailed Implementation

[0029] The present invention will now be described in detail with reference to the accompanying drawings and embodiments, examples of which are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0030] In the description of this invention, it should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0031] Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature. In the description of this invention, "multiple" means two or more, unless otherwise explicitly specified.

[0032] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," "fixing," and "setting," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0033] Example 1 like Figure 1 As shown, an rPPG signal extraction network based on the CBAM attention mechanism and bidirectional Mamba includes: A fusion backbone network is used to receive face video tensors. A multi-scale differential feature containing static appearance features and dynamic differential features is constructed on the face video tensors through a temporal differential mechanism. The static appearance features and dynamic differential features are adaptively fused to output shallow spatiotemporal fusion features. The spatiotemporal encoder, connected to the fusion backbone network, is used to receive the shallow spatiotemporal fusion features, perform multi-scale temporal modeling through a time pyramid architecture, and output deep spatiotemporal features. The confidence perception gating module is connected to the spatiotemporal encoder and is used to receive the deep spatiotemporal features, generate a spatiotemporal confidence map, and perform weighted cleaning and spatial aggregation on the deep spatiotemporal features based on the spatiotemporal confidence map, and output the aggregated two-dimensional temporal features. The frequency domain gating module, connected to the confidence perception gating module, is used to receive the two-dimensional time-series features, transform them to the frequency domain, filter out non-physiological frequency band noise, restore them to the time domain, and output time-domain enhanced features. A bidirectional Mamba regression head, connected to the frequency domain gating module, is used to receive the time domain enhancement features, capture the time-series dependencies through a bidirectional state-space model, and map them into a one-dimensional remote photoplethysmography (PPG) signal.

[0034] The rPPG signal extraction network of this invention achieves a complete extraction process from the face video tensor to the remote photoplethysmography (PPG) signal through an end-to-end cascaded architecture consisting of a fusion backbone network, a spatiotemporal encoder, a confidence-aware gating module, a frequency-domain gating module, and a bidirectional Mamba regression head. Specifically, the fusion backbone network enhances subtle skin color variation features through adaptive differential fusion; the spatiotemporal encoder utilizes a time pyramid structure for multi-scale temporal modeling; the confidence-aware gating module and the frequency-domain gating module perform dual feature cleaning in the spatial and frequency domains, respectively; and the bidirectional Mamba regression head reconstructs the waveform with linear computational complexity. The synergistic effect of these modules enables the network to balance long-term global modeling capabilities with low computational overhead, solving the problem of balancing modeling efficiency and accuracy in existing technologies.

[0035] The data structure of the face video tensor is B×C×T×H×W, where B represents the batch size, C represents the number of color channels, T represents the number of frames in the time dimension, and H and W represent the height and width of the image, respectively.

[0036] like Figure 2 As shown, the converged backbone network includes: The temporal difference construction module is used to perform temporal misalignment slicing on the face video tensor, calculate the pixel difference between the current frame and the neighboring frames, and splice them to generate a multi-channel difference video tensor. The dual-branch feature extraction layer includes an appearance feature extraction branch and a motion feature extraction branch. The appearance feature extraction branch is used to process the face video tensor to extract static appearance features, and the motion feature extraction branch is used to process the difference video tensor to extract dynamic difference features. An adaptive fusion layer is used to introduce learnable first and second scalar parameters, as well as a first fusion transformation network and a second fusion transformation network, to adaptively fuse the static appearance features and dynamic difference features according to a preset adaptive fusion function, and output shallow spatiotemporal fusion features.

[0037] In the fusion backbone network of this invention, the temporal difference construction module calculates the pixel difference between the current frame and neighboring frames, which can enhance the sensitivity to micron-level skin color fluctuations caused by changes in blood volume; the dual-branch feature extraction layer extracts static appearance features and dynamic difference features respectively, preserving facial spatial texture while amplifying pulse-related dynamic information; the adaptive fusion layer uses learnable parameters α and β to dynamically balance the contributions of the two types of features, avoiding feature imbalance caused by fixed-weight fusion; this limitation enables the network to obtain shallow features rich in pulse information at the input stage, laying the foundation for subsequent high-precision extraction.

[0038] In this embodiment, the adaptive fusion function is as follows:

[0039] Where Out represents the shallow spatiotemporal fusion feature finally output by the fusion backbone network; and The first and second scalar parameters are learned and used to dynamically adjust the contribution ratio of static appearance features and dynamic difference features in the fusion process. This refers to the static appearance features obtained after the face video tensor has been processed by the appearance feature extraction branch (which includes convolutional layers, normalization layers, and activation functions); This represents the dynamic difference features obtained after the differential video tensor constructed within the network is processed by the motion feature extraction branch. The first fusion transformation network consists of convolutional layers, group normalization layers, and ReLU activation functions, and is used to extract high-level representations of fusion features. This represents the second fusion transform network, which has the same structure as the first fusion transform network and is used to supplement the pure dynamic difference features.

[0040] like Figure 3 As shown, the spatiotemporal encoder includes: The slice embedding layer is used to perform 3D convolution processing on the input shallow spatiotemporal fusion features, mapping their video voxels into high-dimensional feature vectors and performing spatial downsampling; The system comprises multiple cascaded processing stages, each including a temporal downsampling layer and stacked multiple improved spatiotemporal Mamba modules. These modules are used to progressively downsample the temporal dimension of the high-dimensional feature vector while maintaining constant spatial resolution and channel number, extracting deep spatiotemporal abstract features from local transient changes to global periodic patterns, and outputting the deep spatiotemporal features.

[0041] In the spatiotemporal encoder of this invention, the slice embedding layer performs spatial downsampling through three-dimensional convolution, which reduces redundancy while preserving key skin textures; the cascaded multiple processing stages progressively downsample the temporal domain while maintaining constant spatial resolution and number of channels, constructing a temporal pyramid structure, enabling the network to simultaneously extract multi-scale temporal features from local transient changes (a single cardiac cycle) to global periodic patterns (stable heart rate over tens of seconds), significantly improving its adaptability to different heart rate ranges (such as resting heart rate and exercise heart rate).

[0042] Specifically, the improved spatiotemporal Mamba module includes: The central difference convolutional layer is used to calculate the intensity difference between neighboring pixels and the central pixel within the receptive field of the convolutional kernel, introducing gradient information to enhance the kernel's sensitivity to subtle texture details. The 3DCBAM attention module, connected to the central differential convolutional layer, is used to sequentially perform channel attention weighting and spatial attention weighting on the feature map output by the convolution, adaptively enhancing the channel containing the pulse signal and brightening the facial skin area. The bidirectional Mamba layer, connected to the 3DCBAM attention module, is used to flatten the attention-weighted feature map in the spatial dimension and convert it into a sequence format. It captures causal historical dependencies and inverse future dependencies through forward scanning paths and backward scanning paths, respectively, and outputs the deep spatiotemporal features.

[0043] In the improved spatiotemporal Mamba module of this invention, the central difference convolutional layer enhances the ability to capture subtle skin color gradient changes by performing difference convolution on the neighborhood and the center pixel; the 3DCBAM attention module performs channel and spatial weighting sequentially, automatically focusing on pulse-related channels (such as the green channel) and facial skin regions, suppressing background and motion noise; the bidirectional Mamba layer realizes forward and backward global temporal modeling with linear complexity. The three are connected to form a collaborative mechanism of "sensitive capture - attention cleaning - efficient temporal modeling", which is the core unit for the network to achieve high-precision anti-interference extraction.

[0044] Specifically, the 3DCBAM attention module includes: The channel attention submodule is used to perform global average pooling and global max pooling on the input feature map to compress the spatiotemporal dimension. It learns the importance weights of each channel by sharing a multilayer perceptron, generates channel weights, and multiplies the channel weights with the original input feature map element-wise. The spatial attention submodule, connected to the channel attention submodule, is used to generate a spatial attention map using 3D convolution based on the channel-weighted feature map, and then perform element-wise multiplication of the spatial attention map with the original input feature map.

[0045] In the 3DCBAM attention module of the present invention, the channel attention submodule utilizes a dual pooling strategy of average pooling and max pooling to more comprehensively characterize the channel statistical characteristics; the spatial attention submodule uses three-dimensional convolution to generate spatial masks, which can enhance the localization accuracy of the skin region by utilizing spatiotemporal context, thereby further improving the adaptability and robustness of the attention mechanism.

[0046] like Figure 4 As shown, the confidence-aware gating module includes: The confidence prediction network is used to perform three-dimensional convolution and activation processing on the input deep spatiotemporal features to generate a spatiotemporal confidence map. The confidence weighting module, connected to the confidence prediction network, is used to perform layer normalization and multilayer perceptron projection on the deep spatiotemporal features, then perform element-wise multiplication with the spatiotemporal confidence map, and add it to the original input deep spatiotemporal features through residual connection to output the refined deep spatiotemporal features. The spatial aggregation module, connected to the confidence weighting module, is used to perform weighted average pooling on the refined deep spatiotemporal features according to a preset spatial aggregation function, using the spatiotemporal confidence map as spatial weights, to aggregate the four-dimensional spatiotemporal features into two-dimensional time-series features, and output the aggregated two-dimensional time-series features.

[0047] In the confidence perception gating module of this invention, the confidence prediction network generates a spatiotemporal confidence map to evaluate the signal quality at different spatial locations and times in the video. The values ​​of the spatiotemporal confidence map after Sigmoid activation are between 0 and 1. The confidence weighting module performs layer normalization and nonlinear projection on deep spatiotemporal features, multiplies them with the spatiotemporal confidence map, and introduces residual connections to achieve soft suppression of low-quality region feature responses while preserving the integrity of the original information. The spatial aggregation module performs weighted average pooling with the spatiotemporal confidence map as weight, aggregating four-dimensional features into two-dimensional temporal features. This can automatically identify and weaken the influence of low-quality regions such as occlusion, large-angle head turns, and strong background noise, significantly improving the stability and signal extraction accuracy of the network in uncontrolled scenarios.

[0048] In this embodiment, the spatial aggregation function is as follows:

[0049] This represents the two-dimensional temporal features output after spatial aggregation, where the spatial dimension has been eliminated. Only batch B, channel C, and time dimension T are retained; This indicates the refined deep spatiotemporal characteristics; This represents the spatiotemporal confidence map; This represents a summation operation performed on the height (H) and width (W) dimensions of the feature map, i.e., aggregation of all spatial pixels; This represents element-wise multiplication. Represents a very small constant (e.g.) (This is used to prevent division by zero and ensure the stability of numerical calculations.)

[0050] The refined deep spatiotemporal characteristics are shown below:

[0051] Deep spatiotemporal features representing the original input; Represents projection features The calculation process; Mask represents the spatiotemporal confidence map, which is generated by the confidence prediction network through 3D convolution and Sigmoid activation of the deep spatiotemporal features of the original input, mapping each pixel value to... Interval.

[0052] like Figure 5 As shown, the frequency domain gating module includes: The Fourier transform unit is used to transform the input two-dimensional time-series features from the time domain to the frequency domain through the fast Fourier transform, to obtain the amplitude spectrum and phase spectrum. A frequency-domain gated network, connected to the Fourier transform unit, is used to adaptively compress the frequency dimension of the amplitude spectrum using average pooling. A frequency mask is generated through a one-dimensional convolutional layer and a Sigmoid activation function. The frequency mask is then multiplied with the amplitude spectrum before compression and interpolated and filtered to restore the original frequency resolution. The inverse Fourier transform unit, connected to the frequency domain gated network, is used to combine the filtered amplitude spectrum with the phase spectrum and restore it to the time domain through inverse fast Fourier transform, and output the restored time domain features. The temporal residual connection unit, connected to the inverse Fourier transform unit, is used to add the temporal feature to the two-dimensional temporal feature and output the temporal enhancement feature.

[0053] In the frequency domain gating module of this invention, the frequency domain gating network compresses the frequency dimension of the amplitude spectrum transformed by the Fourier transform unit through adaptive average pooling to generate a frequency mask, enabling the network to automatically learn the effective heart rate frequency band (e.g., 0.7~3.0Hz) and filter out extremely low frequency motion noise and high frequency lighting flicker; the time domain residual connection unit can avoid signal distortion caused by over-filtering, so as to enhance the rPPG signal with clear physical meaning from the frequency domain dimension, making up for the defect of the pure time domain regression method ignoring the sparsity of the frequency domain, and effectively improving the signal-to-noise ratio.

[0054] like Figure 6 As shown, the bidirectional Mamba return header includes: The forward Mamba branch is used to perform a forward scan on the input temporal augmentation features to capture causal history dependencies in the time series signal; The backward Mamba branch is used to perform a backward scan on the input time-domain augmentation features to capture the inverse future dependencies in the time-series signal; The Mamba fusion layer, connected to the forward Mamba branch and the backward Mamba branch, is used to fuse the outputs of the two branches to obtain high-dimensional fusion features; A linear projection layer, connected to the Mamba fusion layer, is used to map the high-dimensional fusion features into a one-dimensional remote photoplethysmography (PPG) waveform signal.

[0055] The bidirectional Mamba regression head of this invention uses bidirectional Mamba again for time-dependent capture after frequency domain enhancement, which can further smooth the waveform and compensate for the phase shift that may be introduced by frequency domain filtering. The fusion of forward and backward branches allows the regression head to utilize past and future context information at the same time, which has higher waveform reconstruction accuracy compared with unidirectional modeling. This ensures that the final output rPPG signal has a clear waveform shape, which is convenient for the accurate calculation of subsequent physiological indicators such as heart rate and heart rate variability.

[0056] Example 2 like Figure 7 As shown, a non-contact physiological monitoring method includes the following steps: Step 1: Obtain a continuous video frame sequence of the monitored object, and perform face extraction and normalization processing on each video image in the video frame sequence to obtain the face video tensor of the video frame sequence.

[0057] In step 1, the system calls the device's image acquisition interface (such as a USB camera or built-in webcam) to capture a continuous sequence of video frames in real time at a fixed frame rate (e.g., 30fps). To ensure smooth interaction, the system adopts a multi-threaded architecture, with image rendering performed in the main thread and data processing in a background thread. For each video frame, a facial landmark detection algorithm is used to locate the face, and an exponential moving average algorithm is used... The detection box coordinates are smoothed and filtered to eliminate jitter. Then, the facial region of interest is cropped and adjusted to 96×96 pixels. After normalization, it is pushed into a sliding window queue of fixed length (e.g., 240 frames). When the sliding queue is full, the background inference engine infers the face video tensor within the current time window.

[0058] Specifically, step 1 includes the following steps: Step 11: Perform feature analysis on each frame of video image using a facial landmark detection algorithm to extract the coordinate information of key points on the face. Then, based on the key point coordinate information, construct an initial bounding box covering the face region. Next, in order to eliminate detection box jitter caused by unintentional movement of the subject or camera vibration during video acquisition, apply a temporal smoothing filtering algorithm (including but not limited to the exponential moving average algorithm) to smooth the bounding box coordinates between the video frame sequences. Based on the smoothed bounding box coordinates, crop the region of interest image containing the face from the original video image, and uniformly adjust the cropped face image sequence to a preset spatial resolution (e.g., H×W pixels) to meet the input dimension requirements of the subsequent deep neural network.

[0059] Step 12: Convert the cropped and resized face image sequence into tensor format and perform numerical normalization. This normalization process aims to eliminate numerical differences under different lighting intensities and accelerate model convergence. Specifically, it maps the original pixel values ​​to the [-1, 1] interval through a linear transformation (e.g., using a formula...). To implement this, construct a face video tensor of shape B×C×T×H×W, where B represents the batch size, C represents the number of color channels, T represents the number of frames in the time dimension, and H and W represent the height and width of the image, respectively.

[0060] Step 2: Input the face video tensor into the rPPG signal extraction network described in Embodiment 1, and receive the remote photoplethysmography (PPG) signal output by the rPPG signal extraction network; Step 3: Analyze the remote photoplethysmography (PPG) signal to extract physiological indicators related to the heart cycle of the monitored object.

[0061] In this step, the remote photoplethysmography (PPG) signal is detrended and analyzed to calculate the instantaneous heart rate value. The smoothed heart rate value is then displayed in real-time on the video frame layer of the user interface. Furthermore, the system assesses the current signal quality based on the "spatiotemporal confidence map" output from the network. If the system detects that the user is off-center from the camera, the lighting is too dim, or the face is obscured, causing the confidence level to fall below a preset threshold, the system will trigger a visual prompt on the interface (such as displaying "Please keep your face facing the camera" or highlighting a detection box) to guide the user to adjust their posture, thus forming a complete closed-loop interactive system.

[0062] Example 3 A non-contact physiological monitoring device includes a processor and a memory connected to each other. The memory stores the rPPG signal extraction network described in Embodiment 1 and a computer program executed by the processor. When the processor executes the computer program, it retrieves the rPPG signal extraction network from the memory to perform the non-contact physiological monitoring method described in Embodiment 2.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention and not to limit them. Although the embodiments of the present invention have been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the embodiments of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An rPPG signal extraction network based on CBAM attention mechanism and bidirectional Mamba, characterized in that, include: A fusion backbone network is used to receive face video tensors. A multi-scale differential feature containing static appearance features and dynamic differential features is constructed on the face video tensors through a temporal differential mechanism. The static appearance features and dynamic differential features are adaptively fused to output shallow spatiotemporal fusion features. The spatiotemporal encoder, connected to the fusion backbone network, is used to receive the shallow spatiotemporal fusion features, perform multi-scale temporal modeling through a time pyramid architecture, and output deep spatiotemporal features. The confidence perception gating module is connected to the spatiotemporal encoder and is used to receive the deep spatiotemporal features, generate a spatiotemporal confidence map, and perform weighted cleaning and spatial aggregation on the deep spatiotemporal features based on the spatiotemporal confidence map, and output the aggregated two-dimensional temporal features. The frequency domain gating module, connected to the confidence perception gating module, is used to receive the two-dimensional time-series features, transform them to the frequency domain, filter out non-physiological frequency band noise, restore them to the time domain, and output time-domain enhanced features. A bidirectional Mamba regression head, connected to the frequency domain gating module, is used to receive the time domain enhancement features, capture the time-series dependencies through a bidirectional state-space model, and map them into a one-dimensional remote photoplethysmography (PPG) signal.

2. The rPPG signal extraction network according to claim 1, characterized in that, The converged backbone network includes: The temporal difference construction module is used to perform temporal misalignment slicing on the face video tensor, calculate the pixel difference between the current frame and the neighboring frames, and splice them to generate a multi-channel difference video tensor. The dual-branch feature extraction layer includes an appearance feature extraction branch and a motion feature extraction branch. The appearance feature extraction branch is used to process the face video tensor to extract static appearance features, and the motion feature extraction branch is used to process the difference video tensor to extract dynamic difference features. An adaptive fusion layer is used to introduce learnable first and second scalar parameters, as well as a first fusion transformation network and a second fusion transformation network, to adaptively fuse the static appearance features and dynamic difference features according to a preset adaptive fusion function, and output shallow spatiotemporal fusion features.

3. The rPPG signal extraction network according to claim 1, characterized in that, The spatiotemporal encoder includes: The slice embedding layer is used to perform 3D convolution processing on the input shallow spatiotemporal fusion features, mapping their video voxels into high-dimensional feature vectors and performing spatial downsampling; The system comprises multiple cascaded processing stages, each including a temporal downsampling layer and stacked multiple improved spatiotemporal Mamba modules. These modules are used to progressively downsample the temporal dimension of the high-dimensional feature vector while maintaining constant spatial resolution and channel number, extracting deep spatiotemporal abstract features from local transient changes to global periodic patterns, and outputting the deep spatiotemporal features.

4. The rPPG signal extraction network according to claim 3, characterized in that, The improved spatiotemporal Mamba module includes: The central difference convolutional layer is used to calculate the intensity difference between neighboring pixels and the central pixel within the receptive field of the convolutional kernel, introducing gradient information to enhance the kernel's sensitivity to subtle texture details. The 3D CBAM attention module, connected to the central differential convolutional layer, is used to sequentially perform channel attention weighting and spatial attention weighting on the feature map output by the convolution, adaptively enhancing the channel containing the pulse signal and brightening the facial skin area; The bidirectional Mamba layer, connected to the 3D CBAM attention module, is used to flatten the attention-weighted feature map in the spatial dimension and convert it into a sequence format. It captures causal historical dependencies and inverse future dependencies through forward and backward scanning paths, respectively, and outputs the deep spatiotemporal features.

5. The rPPG signal extraction network according to claim 4, characterized in that, The 3D CBAM attention module includes: The channel attention submodule is used to perform global average pooling and global max pooling on the input feature map to compress the spatiotemporal dimension. It learns the importance weights of each channel by sharing a multilayer perceptron, generates channel weights, and multiplies the channel weights with the original input feature map element-wise. The spatial attention submodule, connected to the channel attention submodule, is used to generate a spatial attention map using 3D convolution based on the channel-weighted feature map, and then perform element-wise multiplication of the spatial attention map with the original input feature map.

6. The rPPG signal extraction network according to claim 1, characterized in that, The confidence-sensing gating module includes: The confidence prediction network is used to perform three-dimensional convolution and activation processing on the input deep spatiotemporal features to generate a spatiotemporal confidence map. The confidence weighting module, connected to the confidence prediction network, is used to perform layer normalization and multilayer perceptron projection on the deep spatiotemporal features, then perform element-wise multiplication with the spatiotemporal confidence map, and add it to the original input deep spatiotemporal features through residual connection to output the refined deep spatiotemporal features. The spatial aggregation module, connected to the confidence weighting module, is used to perform weighted average pooling on the refined deep spatiotemporal features according to a preset spatial aggregation function, using the spatiotemporal confidence map as spatial weights, to aggregate the four-dimensional spatiotemporal features into two-dimensional time-series features, and output the aggregated two-dimensional time-series features.

7. The rPPG signal extraction network according to claim 1, characterized in that, The frequency domain gating module includes: The Fourier transform unit is used to transform the input two-dimensional time-series features from the time domain to the frequency domain through the fast Fourier transform, to obtain the amplitude spectrum and phase spectrum. A frequency-domain gated network, connected to the Fourier transform unit, is used to adaptively compress the frequency dimension of the amplitude spectrum using average pooling. A frequency mask is generated through a one-dimensional convolutional layer and a Sigmoid activation function. The frequency mask is then multiplied with the amplitude spectrum before compression and interpolated and filtered to restore the original frequency resolution. The inverse Fourier transform unit, connected to the frequency domain gated network, is used to combine the filtered amplitude spectrum with the phase spectrum and restore it to the time domain through inverse fast Fourier transform, and output the restored time domain features. The temporal residual connection unit, connected to the inverse Fourier transform unit, is used to add the temporal feature to the two-dimensional temporal feature and output the temporal enhancement feature.

8. The rPPG signal extraction network according to claim 1, characterized in that, The bidirectional Mamba return header includes: The forward Mamba branch is used to perform a forward scan on the input temporal augmentation features to capture causal history dependencies in the time series signal; The backward Mamba branch is used to perform a backward scan on the input temporal augmentation features to capture the inverse future dependencies in the time series signal; The Mamba fusion layer, connected to the forward Mamba branch and the backward Mamba branch, is used to fuse the outputs of the two branches to obtain high-dimensional fusion features; A linear projection layer, connected to the Mamba fusion layer, is used to map the high-dimensional fusion features into a one-dimensional remote photoplethysmography (PPG) waveform signal.

9. A non-contact physiological monitoring method, characterized in that, Includes the following steps: Step 1: Obtain a continuous video frame sequence of the monitored object, and perform face extraction and normalization processing on each video image in the video frame sequence to obtain the face video tensor of the video frame sequence; Step 2: Input the face video tensor into the rPPG signal extraction network of claim 1, and receive the remote photoplethysmography (PPG) signal output by the rPPG signal extraction network; Step 3: Analyze the remote photoplethysmography (PPG) signal to extract physiological indicators related to the heart cycle of the monitored object.

10. A non-contact physiological monitoring device, comprising a processor and a memory connected to each other, characterized in that, The memory stores the rPPG signal extraction network as described in claim 1, and a computer program for the processor to execute. When the processor executes the computer program, it retrieves the rPPG signal extraction network from the memory to perform the remote photoplethysmography extraction method as described in claim 9.

Images