Remote photoplethysmography method and system based on long short-term spatio-temporal convolutional network

Abstract

The application discloses a remote photoplethysmography (rPPG) signal processing method and system based on a long short-term spatio-temporal convolution network, and belongs to the cross field of biomedical signal processing and computer vision. In view of the signal distortion problem caused by illumination fluctuation, motion artifact and skin color difference, the method constructs a multi-scale spatio-temporal modeling framework: a 3D convolution network is used to extract the local spatio-temporal features of facial microvessel transient changes, a 1D dilated convolution network is combined to capture the long-range periodic features of heart rate rhythm, and the spatio-temporal features are dynamically fused through self-attention weights and a gated residual structure to improve the anti-interference ability. In the preprocessing stage, the face region is located through MTCNN, the motion artifact is compensated by using an optical flow equation, and the signal purity is enhanced in combination with a skin color mask and a YUV color space. The system realizes light weight by using a depth separable convolution and a parallel acceleration strategy, and the network is jointly optimized through a time domain MSE loss and a frequency domain KL divergence. Experiments show that the phase error is reduced by 40% under a dynamic scene, the signal amplitude of a deep skin color group is increased by 60%, the method is suitable for a non-contact health monitoring device, and the robustness and measurement accuracy of the rPPG technology in a complex environment are significantly improved.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of biomedical signal processing and computer vision, specifically relating to a non-contact physiological signal detection method and system that integrates spatiotemporal convolutional networks and long short-term memory networks. Background Technology

[0002] Photoplethysmography (PPG) technology monitors physiological parameters by detecting changes in the absorption of light of a specific wavelength by blood. However, traditional contact sensors are prone to causing skin discomfort, and fluctuations in contact pressure can increase heart rate errors by 15%-20% (Smith et al., "Contact Pressure Effects on PPG Signals", IEEE TBME, 2017), which limits its application in groups such as burn patients. Remote photoplethysmography (rPPG) uses cameras to capture micro-light changes in skin for non-contact monitoring, but it faces three major challenges: 1) Illumination interference reduces the signal-to-noise ratio by more than 40% (when fluctuating by more than 200 lux, Wang et al., "Illumination-Robust rPPG via Chromaticity Separation", CVPR, 2021.); 2) Micro-head movements (>3 mm) cause motion artifacts, with motion state errors reaching 5-8 bpm; 3) The signal amplitude is reduced by about 60% in people with dark skin (McDuff et al., "The Impact of Skin Type on rPPG", Nature NPJ Digital Medicine, 2020.).

[0003] Among existing deep learning methods, CNN models (such as PhysNet) lack temporal modeling capabilities and struggle to capture heart rate variability; RNN approaches are insufficient in suppressing motion artifacts. While the ST-rPPG network (Chen et al., "ST-rPPG: Spatial-Temporal Network for Remote Physiological Measurement", ICCV, 2021.) combines 3D-CNN and LSTM, it still suffers from problems such as noise propagation caused by shallow spatiotemporal coupling and the inability of fixed-weight fusion mechanisms to adapt to dynamic scenes. Its phase error in motion scenes increases by 2.3 times compared to static scenes.

[0004] Current technical bottlenecks are concentrated in: ① mismatch of spatiotemporal features—cascaded processing ignores the spatiotemporal physical correlation of microvascular motion; ② weak dynamic adaptability—difficulty in coping with the coordinated changes in light, motion, and physiological state; ③ imbalance of multi-parameter accuracy—existing single-camera systems have a blood oxygen detection error exceeding ±3%, surpassing the clinical ±2% standard. These shortcomings restrict the reliable application of rPPG technology in real-world scenarios. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a method for extracting remote photoplethysmography (rPPG) signals based on long short-term spatiotemporal convolutional networks. This method employs multi-scale spatiotemporal modeling, combining 3D convolution (local spatiotemporal features) with 1D dilated convolution (long-range periodic patterns), and introduces spatiotemporal graph convolution (GCN) to model spatial topological relationships, achieving dual capture of microscopic transient and macroscopic periodic features. Dynamic feature fusion is implemented, generating spatiotemporal weights through a self-attention mechanism, combined with a gated residual structure (such as a sigmoid-gated β). t Adaptive fusion of long and short-term features enhances anti-interference capabilities; lightweight and efficient design employs depthwise separable convolution and channel shuffling to compress parameters, combined with parallel GPU acceleration and spatiotemporal pooling strategies to improve computational efficiency; end-to-end joint optimization combines temporal (MSE loss) and frequency (KL divergence) supervision, and integrates a probabilistic prediction framework (TCN encoder-decoder) to ensure both waveform accuracy and physiological interpretability. This effectively improves the perception of rPPG signal fluctuations in videos, enhancing the accuracy and robustness of rPPG signal extraction.

[0006] The technical solution provided by this invention includes the following:

[0007] This application provides an rPPG signal processing method based on long short-term spatiotemporal convolutional networks, comprising: determining preprocessed data based on acquired face videos; determining short-term spatiotemporal convolutional features based on the preprocessed data; determining long-term spatiotemporal convolutional features based on the preprocessed data; determining dynamic spatiotemporal fusion features based on the long and short-term spatiotemporal convolutional features; and determining rPPG signals based on the dynamic spatiotemporal fusion features.

[0008] In some embodiments, preprocessed data is determined based on the acquired face video; the target face video ROI region is determined based on the target face video; face motion points of the face ROI are determined based on the face ROI region; face ROI segmentation and noise reduction information is determined based on the face ROI motion points; YUV color conversion space is determined based on the face ROI segmentation and noise reduction information; and time series is determined based on the YUV color conversion space.

[0009] In some embodiments, short-spatial-temporal convolutional features are determined based on the preprocessed data; local spatiotemporal information in the video is determined based on the time series; spatiotemporal nonlinear activation information is determined based on the local spatiotemporal information; spatiotemporal pooling information is determined based on the spatiotemporal nonlinear activation information; normalization information is determined based on the spatiotemporal pooling information; residual connection information is determined based on the normalization information; and short-spatial-temporal convolutional features are determined based on the residual connection information.

[0010] In some embodiments, long-term spatiotemporal convolutional features are determined based on the preprocessed data; dilated temporal convolutions in the video are determined based on the time series; temporal nonlinear activation information is determined based on the dilated temporal convolutions; temporal pooling information is determined based on the temporal nonlinear activation information; normalization information is determined based on the temporal pooling information; residual connection information is determined based on the normalization information; temporal attention mechanism information is determined based on the residual connection information; and long-term spatiotemporal feature information is determined based on the above information.

[0011] In some embodiments, dynamic spatiotemporal fusion features are determined based on the long and short spatiotemporal convolution features; feature alignment information is determined based on the short spatiotemporal convolution features and long spatiotemporal feature information; joint feature representation is determined based on the feature alignment information; attention weight information is determined based on the joint features; and gating fusion information and residual output information are determined based on the attention weight information.

[0012] In some embodiments, the rPPG signal is determined based on the dynamic spatiotemporal fusion characteristics. A 1D signal is determined based on the dynamic spatiotemporal fusion characteristics; a high-quality rPPG signal is then determined by bandpass filtering of the 1D signal.

[0013] The beneficial effects of this invention are as follows: This application acquires a target face video and determines the effective region of the target face video based on the target face video; determines the long and short-term spatiotemporal convolution features of the target face video based on the effective region of the target face video; determines dynamic spatiotemporal fusion features based on the long and short-term spatiotemporal convolution features; and determines a high-quality rPPG signal based on the determined spatiotemporal fusion features. Through the long and short-term spatiotemporal convolution method, environmental interference such as moving light can be effectively avoided, improving the accuracy and robustness of rPPG signal extraction. Attached Figure Description

[0014] The scope of this disclosure can be better understood by reading the following detailed description of exemplary embodiments in conjunction with the accompanying drawings. The accompanying drawings are:

[0015] Figure 1 A flowchart illustrating the overall process of extracting rPPG signals based on a long short-term spatiotemporal convolutional network, as provided in this embodiment of the invention.

[0016] Figure 2 A flowchart for short-term spatiotemporal information extraction provided in an embodiment of the present invention;

[0017] Figure 3 A flowchart for long-term spatiotemporal information extraction provided in an embodiment of the present invention;

[0018] Figure 4 This is a structural block diagram of long and short spatiotemporal feature fusion provided in an embodiment of the present invention. Detailed Implementation

[0019] Achieving high-precision remote photoplethysmography (rPPG) signal detection under complex environmental interference presents significant technical challenges. Existing deep learning models struggle to effectively separate noise components unrelated to physiological signals (such as abrupt changes in ambient lighting and facial motion artifacts) when processing video data. Furthermore, subtle blood flow-related light intensity changes on the facial skin surface (typically less than 0.1% brightness fluctuations) exhibit a significant coupling effect with background noise, leading to a severe degradation of the signal-to-noise ratio in signal extraction. Therefore, methods such as... Figure 1 As shown, this application proposes an rPPG signal extraction method based on Long Short-Term Spatiotemporal Convolutional Networks (LSSNs), which can be deployed on various electronic device platforms, including but not limited to: cloud servers, mobile terminals (smartphones / tablets), embedded computers, and dedicated neural network accelerators. The rPPG signal extraction method includes:

[0020] In some embodiments, preprocessed data is determined based on the acquired face video; the target face video ROI region is determined based on the target face video; face motion points of the face ROI are determined based on the face ROI region; face ROI segmentation and noise reduction information is determined based on the face ROI motion points; YUV color conversion space is determined based on the face ROI segmentation and noise reduction information; and time series is determined based on the YUV color conversion space.

[0021] Step S1: Acquire the target face video and determine the preprocessing data based on the acquired face video. Specific steps include:

[0022] Step S11: Determine the ROI region of the target face video based on the target face video, and use the MTCNN algorithm for face detection. First, use P-Net to quickly generate candidate face regions, then use R-Net to filter false candidate boxes, and finally use O-Net to further refine the position.

[0023] Step S12: Determine the face motion points of the face ROI based on the face ROI region. First, calculate the grayscale images of the current frame and the next frame, and then use the image gradient (x-direction derivative I) to... x y-direction derivative I y ) and time derivative I t Characterize pixel changes; establish the optical flow equation, I x ·u+I y ·v+I t =0; Solve the displacement vector using the Lucas-Kanade algorithm; perform pixel aggregation, average the RGB values ​​of all pixels within the ROI by channel, and generate a time series signal (mean R, mean G, mean B for each frame).

[0024] Step S13: Determine face ROI segmentation and noise reduction information based on the face ROI motion points, apply Gaussian blur or median filtering to the ROI region to reduce camera noise; use a skin mask to exclude non-skin pixels (such as background and hair).

[0025] Step S14: Determine the YUV color conversion space based on the face ROI segmentation and noise reduction information, and use the CHROM algorithm to convert the RGB signal into the YUV color space, which is more sensitive to blood flow. Step S15: Determine the time series based on the YUV color conversion space, and extract the spatiotemporal cube from the preprocessed video. The shape is: Where T is the number of time steps (frames), H×W is the spatial resolution, and C is the number of color channels.

[0026] In some embodiments, short-spatial-temporal convolutional features are determined based on the preprocessed data; local spatiotemporal information in the video is determined based on the time series; spatiotemporal nonlinear activation information is determined based on the local spatiotemporal information; spatiotemporal pooling information is determined based on the spatiotemporal nonlinear activation information; normalization information is determined based on the spatiotemporal pooling information; residual connection information is determined based on the normalization information; and short-spatial-temporal convolutional features are determined based on the residual connection information.

[0027] Step S2: Determine the short-temporal convolutional features of the target face video based on the effective region of the target face video;

[0028] Step S21: Determine the local spatiotemporal information in the video based on the time series, and set the 3D convolution kernel parameters as follows. Where, k t : The size of the convolutional kernel in the time dimension (e.g., 3 frames), k h k w : Spatial dimension of the convolution kernel size (e.g., 3×3), D s Output feature dimension (number of channels). For time step t, input local time window. The convolution output is The output spatial dimensions H′, W′ are determined by the convolution stride and padding, with typical parameters: k t =3,k h =k w =3, step size (1,1,1). Step S22: Determine the spatiotemporal nonlinear activation information based on the local spatiotemporal information. The purpose of introducing nonlinear activation is to enhance the model's expressive power. Activation function

[0029] Step S23: Determine spatiotemporal pooling information based on the spatiotemporal nonlinear activation information. Spatiotemporal pooling can reduce spatial resolution, enhance translation invariance, and reduce computational load. This invention employs max pooling along the spatial dimension, as shown in the formula. Pooling core size: 2×2, step size 2, space size halved.

[0030] Step S24: Determine normalization information based on the spatiotemporal pooling information, and normalize each feature channel using the formula. Step S25: Determine residual connection information based on the normalized information. Residual connections alleviate the gradient vanishing problem and preserve shallow detail information. If the input and output dimensions match, fuse features using skip connections. If the dimensions do not match, the number of channels needs to be adjusted using a 1×1×1 convolution.

[0031] Step S26: Determine short-spatial-temporal convolutional features based on the residual connection information, compressing multidimensional features into a time series for easier subsequent fusion. Global Average Pooling (GAP): Aggregate information along the spatial dimension H″×W″. Time alignment: Arrange the features at each time step in sequence to form a short-term feature sequence.

[0032] In some embodiments, long-term spatiotemporal convolutional features are determined based on the preprocessed data; dilated temporal convolutions in the video are determined based on the time series; temporal nonlinear activation information is determined based on the dilated temporal convolutions; temporal pooling information is determined based on the temporal nonlinear activation information; normalization information is determined based on the temporal pooling information; residual connection information is determined based on the normalization information; temporal attention mechanism information is determined based on the residual connection information; and long-term spatiotemporal feature information is determined based on the above information.

[0033] Step S3: Determine long-term spatiotemporal convolution features based on the preprocessed data.

[0034] Step S31: Determine the dilated temporal convolution in the video based on the aforementioned time series. Dilated convolution increases the temporal perception interval, capturing long-term periodic patterns (such as heart rhythm). One-dimensional dilated convolution is applied in the temporal dimension, ignoring the spatial dimension (spatial information has already been extracted by the short-term branch). Mathematical formula: Let the dilated convolution parameters be... Where: k t : Kernel size (e.g., 3), D l: Output feature dimension, r: dilation rate (e.g., r=4), for time point t, the actual time window covered by the convolution is: t start =tr×(k t -1) / 2,t end =t+r×(k) t -1) / 2, the output feature is calculated as

[0035] Step S32: Determine temporal nonlinear activation information based on the dilated temporal convolution; the nonlinear activation function enhances the nonlinear expressive power; the activation function...

[0036] Step S33: Determine temporal pooling information based on the temporal nonlinear activation information. The pooling process can reduce temporal resolution and enhance robustness to low-frequency periodic signals. Perform max pooling along the time dimension:

[0037] Step S34: Determine normalization information based on the time pooling information. The normalization process can stabilize the training process and accelerate convergence. Step S35: Determine residual connection information based on the normalization information. Residual connections are used to mitigate gradient vanishing and preserve the long-term trend of the original time series. If the input dimensions match, the initial input and convolution result are fused through skip connections. If the input dimensions do not match, use a 1×1×1 convolution to adjust the number of channels.

[0038] Step S36: Determine the time attention mechanism information based on the residual connection information, adaptively focus on key time steps (such as the peak heart rate), and introduce a time attention weight β. t ∈[0,1], in This represents the short-term branching characteristic, where σ is the Sigmoid function and W... a ,b a Weighted output for learnable parameters: Step S37: Based on the information described above, determine the long-term spatiotemporal feature information. The long-term spatiotemporal feature formula can be expressed as:

[0039] In some embodiments, dynamic spatiotemporal fusion features are determined based on the long and short spatiotemporal convolution features; feature alignment information is determined based on the short spatiotemporal convolution features and long spatiotemporal feature information; joint feature representation is determined based on the feature alignment information; attention weight information is determined based on the joint features; and gating fusion information and residual output information are determined based on the attention weight information.

[0040] Step 4: The long and short spatiotemporal convolution features are used to determine the dynamic spatiotemporal fusion features;

[0041] Step 41: Determine feature alignment information based on the short-term spatiotemporal convolutional features and long-term spatiotemporal features, and map the short and long-term features to the same dimension D through linear mapping:

[0042] Step 42: Determine the joint feature representation based on the feature alignment information, and perform the joint representation using a concatenation method.

[0043] Step 43: Determine attention weight information based on the joint features, and dynamically learn the fusion weights of short-term features at each time step through a self-attention mechanism to generate Query, Key, and Value.

[0044] Project the joint feature X onto the query, key, and value space:

[0045]

[0046]

[0047] Calculate attention scores, measure the correlation between time steps using dot products, and scale to prevent gradient explosion. Finally, normalized weights are calculated by applying Softmax to each row to obtain the normalized attention weight matrix: Where α i,j This indicates the degree of attention that time step i pays to j.

[0048] Step 44: Determine the gating fusion information and residual output information based on the attention weight information; aggregate based on contextual features; weight the values ​​using attention weights to generate context-aware global features. Weighted fusion of long-term and short-term features, applying gated weight β t ∈[0,1], characteristics of dynamic allocation of length: β t =σ(W g [L′ t ;S′ t ]+b g ),in σ is the Sigmoid function, and the final fused features are: In some embodiments, the rPPG signal is determined based on the dynamic spatiotemporal fusion characteristics. A 1D signal is determined based on the dynamic spatiotemporal fusion characteristics; a high-quality rPPG signal is then determined by bandpass filtering of the 1D signal.

[0049] Step 5: Determine the rPPG signal based on the dynamic spatiotemporal fusion characteristics.

[0050] Step 51: Determine the 1D signal based on the dynamic spatiotemporal fusion features, and decode the fusion features into a 1D signal through a fully connected layer: Connection layer

[0051] Step 52: Based on the 1D signal, perform bandpass filtering to determine a high-quality rPPG signal. The generated rPPG raw signal is bandpass filtered (0.5-4Hz, corresponding to 30-240 BPM) to remove high-frequency noise (such as motion artifacts) and baseline drift, resulting in a high-quality rPPG signal. filtered =Butterworth_Bandpass(rPPG) raw ).

Claims

1. A method for processing remote photoplethysmography (rPPG) signals based on long short-term spatiotemporal convolutional networks, characterized in that, Includes the following steps: Step S1: Acquire the target face video, and determine preprocessing data based on the face video; Step S2: Extract short-term spatiotemporal convolutional features from the preprocessed data. The short-term spatiotemporal convolutional features capture local spatiotemporal patterns through a 3D convolutional network. Step S3: Extract long-term spatiotemporal convolutional features from the preprocessed data. The long-term spatiotemporal convolutional features capture long-range periodic patterns through a 1D dilated convolutional network. Step S4: Perform dynamic spatiotemporal fusion of the short-term spatiotemporal convolutional features and the long-term spatiotemporal convolutional features to generate fused features. Step S5: Determine the rPPG signal based on the fusion characteristics.

2. The method according to claim 1, characterized in that, The preprocessing data determined in step S1 includes: extracting the face ROI region based on the MTCNN algorithm; calculating the motion points of the ROI region through the optical flow equation; performing noise reduction processing and YUV color space conversion on the ROI region to generate a time series signal.

3. The method according to claim 2, characterized in that, The noise reduction process includes: applying Gaussian blur or median filtering to the ROI region; and eliminating non-skin pixel interference based on a skin color mask.

4. The method according to claim 2, characterized in that, The extraction of short-term spatiotemporal convolutional features in step S2 specifically includes: performing 3D convolution operations on the time series to capture local spatiotemporal information; extracting high-dimensional features through spatiotemporal nonlinear activation functions and spatiotemporal pooling operations; and using residual connections to fuse shallow and deep features to generate a short-term spatiotemporal feature sequence.

5. The method according to claim 4, characterized in that, The kernel parameters for the 3D convolution operation are: kernel size in the time dimension. Spatial dimension convolution kernel size The step size is (1, 1, 1).

6. The method according to claim 2, characterized in that, The extraction of long-term spatiotemporal convolutional features in step S3 specifically includes: performing a 1D dilated convolution operation on the time series to expand the temporal receptive field, with an dilation rate r≥2; weighting the features of key time steps through a temporal attention mechanism; and generating a long-term spatiotemporal feature sequence by combining residual connections and normalization operations.

7. The method according to claim 6, characterized in that, The time attention mechanism generates weights using the Sigmoid function. ,satisfy: ,in As a long-term characteristic, It is a short-term characteristic. and σ is a learnable parameter, representing the Sigmoid function.

8. The method according to claim 1, characterized in that, The dynamic spatiotemporal fusion in step S4 includes: aligning short-spatiotemporal convolutional features and long-spatiotemporal convolutional features to the same dimension; calculating the spatiotemporal weight matrix through a self-attention mechanism; and adaptively fusing the long and short-spatiotemporal features using a gated residual structure, resulting in the final fused feature: ,in For gating weights, These are global features for self-attention aggregation.

9. The method according to claim 1, characterized in that, The step S5 of determining the rPPG signal includes: decoding the fused features into a 1D raw signal through a fully connected layer; performing bandpass filtering on the raw signal in the frequency range of 0.5 Hz to 4 Hz to filter out high-frequency noise and baseline drift, thereby generating a high-quality rPPG signal.

10. A remote photoplethysmography (rPPG) signal processing system based on a long short-term spatiotemporal convolutional network, characterized in that, include: The preprocessing module is used to perform step S1 as described in any one of claims 1-9; the short-spatial-temporal feature extraction module is used to perform step S2 as described in any one of claims 1-9; the long-spatial-temporal feature extraction module is used to perform step S3 as described in any one of claims 1-9; the dynamic fusion module is used to perform step S4 as described in any one of claims 1-9; and the signal generation module is used to perform step S5 as described in any one of claims 1-9.

Images