A non-contact heart rate measurement method and system based on spatiotemporal augmentation networks

Abstract

This invention belongs to the interdisciplinary field of computer vision and medical health monitoring, and discloses a non-contact heart rate measurement method and system based on spatiotemporal augmentation networks, solving the problems of inconvenience of traditional contact measurement and weak anti-interference of existing rPPG technology. The method includes: performing illumination normalization on the original facial video frames and constructing an input tensor; inputting an end-to-end spatiotemporal augmentation neural network (including 3D central difference convolution and multi-dimensional attention module) to reconstruct the rPPG time-series signal; and converting the heart rate through bandpass filtering and fast Fourier transform. This scheme achieves mean absolute errors of 0.24 and 0.40 on the UBFC-rPPG and PURE datasets, respectively, is resistant to illumination / motion interference, is highly efficient end-to-end, has only 0.99M network parameters, is compatible with servers (inference ≥35FPS) and embedded devices (inference ≥28FPS), and has excellent generalization performance.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision and medical health monitoring. Specifically, it relates to a non-contact heart rate measurement method and system based on spatiotemporal augmentation networks. Through an end-to-end deep learning network, it achieves robust extraction of remote photoplethysmography (rPPG) signals and accurate estimation of heart rate in facial videos. It can be applied to scenarios such as medical monitoring (e.g., newborns, burn patients), smart wearable devices, and health management. Background Technology

[0002] Heart rate, as a core physiological parameter reflecting the cardiovascular health status of the human body, is of great significance for disease early warning and health assessment when measured accurately. Traditional heart rate measurement relies on contact technologies, such as electrocardiogram (ECG), which requires electrodes to be attached to the skin, and photoplethysmography (PPG), which requires sensors to contact areas with dense blood vessels. These methods have drawbacks such as inconvenience in use, easy to cause skin allergies, and inability to be suitable for special populations such as burn patients or newborns, making it difficult to meet the needs of convenient and non-invasive monitoring.

[0003] To address these issues, remote photoplethysmography (rPPG) based on facial video has emerged. Its principle is to infer heart rate by analyzing the minute color changes in facial skin caused by the periodic pulsation of blood volume (the fluctuation amplitude of the RGB channel signal is only 1 / 100th of the light noise). However, existing rPPG technology still faces three major bottlenecks, and related technical solutions have significant limitations:

[0004] Traditional methods such as Independent Component Analysis (ICA) and Principal Component Analysis (POS) rely on the mathematical model assumption of "linear signal separation," which leads to a sharp drop in accuracy in complex scenarios. For example, in the UBFC-rPPG public dataset (KU Leuven, 2019), the mean absolute error (MAE) of the ICA method reaches 1.2, and in head motion scenarios, the MAE rises to 1.8, which is completely unable to meet the medical-grade accuracy requirements (MAE≤0.5).

[0005] The core technology of the non-contact heart rate measurement method that integrates ConvLSTM and CBAM in patent CN115024706A is "ConvLSTM temporal modeling + CBAM channel-spatial attention". However, ConvLSTM focuses on long-term temporal dependency learning and cannot accurately capture the microsecond-level color gradient changes of rPPG signals. Moreover, CBAM does not have multi-branch dilated convolution, and the spatial receptive field is limited to 3×3, making it difficult to suppress noise in the facial background and non-skin areas. In the PURE dataset (PhysioNet, 2015), the MAE of this method is 0.75, and the MAE increases to 1.1 in low light (<50 lux) scenes.

[0006] The patent CN114092855A describes a non-contact intelligent human heart rate prediction method based on deep learning. This method uses a non-end-to-end solution of "CNN feature extraction + SVR regression model". It requires manual cropping of the facial region of interest (ROI) and signal preprocessing, which is cumbersome and prone to human error. At the same time, it does not integrate spatiotemporal features, and the heart rate prediction error exceeds 20% in scenarios with violent head movements (displacement > 10cm).

[0007] The paper "3D Convolutional Neural Networks for Remote Photoplethysmography Signal Extraction" (IEEE ICIP 2020, DOI: 10.1109 / ICIP40778.2020.9191064) was the first to use 3D convolution for rPPG signal extraction. However, it uses traditional 3D convolution (kernel_size=5×5×5), which has the problems of "feature redundancy and large number of parameters" (4.2M parameters). Moreover, it only introduces channel attention and does not involve spatiotemporal co-optimization. In the scenario of dark-skinned people (melanin index>50), the signal extraction success rate drops by 40%.

[0008] 3D central difference convolution (3D-CDC) has traditionally been only seen in academic research on edge detection of dynamic images (such as the paper "3DCenterDifferenceConvolutionforVideoEdgeDetection", which clearly states that its application scenario is "edge gradient capture of dynamic video frames"). Those skilled in the art generally believe that it "can only process strong gradient edge signals and cannot adapt to weak physiological signals of rPPG".

[0009] Dilated convolutions are mostly used for large-scale semantic segmentation modeling (such as the paper "Dilated Residual Networks", CVPR 2017, DOI:10.1109 / CVPR.2017.634), and have never been combined with the "focusing of local physiological regions of the face" requirement of rPPG;

[0010] Meanwhile, existing deep learning solutions generally have more than 3M parameters (such as the 4.2M parameters of the model in the IEEE ICIP literature mentioned above), and the inference speed is less than 25FPS, making it difficult to deploy in embedded devices (such as smartwatches and portable monitors).

[0011] In summary, existing technologies cannot simultaneously meet the requirements of "high precision, strong robustness, and lightweight end-to-end" non-contact heart rate measurement, and innovative solutions that break through the above-mentioned technical bottlenecks are urgently needed. Summary of the Invention

[0012] In view of this, the present invention provides a non-contact heart rate measurement method and system based on spatiotemporal augmentation network. Through this drainage structure, the drainage capacity of the upper mold can be significantly improved, making it consistent with the capacity of the lower mold. Furthermore, by controlling the pressure parameters, it can achieve the production of products with better quality.

[0013] To achieve the above objectives, one of the present invention provides the following technical solution:

[0014] A non-contact heart rate measurement method based on spatiotemporal augmentation networks includes the following steps:

[0015] S1. Video Data Preprocessing: Perform illumination normalization on the original facial video frame sequence.

[0016]

[0017] in, This represents the original video frame image at frame t. This represents the average pixel value of all video frames within a predetermined time window centered at frame t. This represents the output video frame after normalization.

[0018] 180 normalized video frames are continuously extracted to construct an input tensor of size [3, 180, 128, 128].

[0019] S2. Physiological Signal Reconstruction: The input tensor is input into an end-to-end spatiotemporal augmentation neural network, and the output is a reconstructed one-dimensional rPPG temporal signal. The spatiotemporal augmentation neural network includes a spatial feature extraction module, multiple spatiotemporal feature enhancement modules, and a multidimensional attention optimization module. The spatiotemporal feature enhancement module is based on a three-dimensional central difference convolution with kernel_size=3×3×3, stride=1, and padding=1. The multidimensional attention optimization module includes a channel-space attention submodule and a temporal attention submodule. The channel-space attention submodule contains parallel 3D dilated convolutional layers with dilation rates of 1, 2, and 4, and the temporal attention submodule contains a 3D convolutional layer with kernel_size=5×1×1.

[0020] S3. Heart Rate Calculation: Perform a Fast Fourier Transform on the rPPG time-series signal to extract the dominant frequency with the strongest energy within the human physiological frequency range of 0.7Hz to 4.0Hz, and calculate it according to the formula. Converted to heart rate value The dominant frequency.

[0021] Furthermore, step S1 also includes: capturing facial video at 30fps and 640×480 resolution, extracting and aligning 68 key points of the face using the dlib library, cropping the facial region by 10% expansion of the face bounding box, and scaling it to 128×128 pixels using bilinear interpolation.

[0022] Furthermore, the spatial feature extraction module consists of a 2D convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence; the parameters of the 2D convolutional layer are in_channels=3, out_channels=16, kernel_size=5×5, stride=1, and padding=2, which extracts shallow spatial features of a single frame while keeping the temporal dimension unchanged.

[0023] Furthermore, the spatiotemporal feature enhancement module comprises six modules. After the input tensor is processed by the spatial feature extraction module, it is downsampled through a three-dimensional max pooling layer, wherein the convolution kernel size of the pooling layer is 2×2×2 and the stride is 2, thereby compressing the feature map to a size [16,180,64,64], and then sequentially input to the six spatiotemporal feature enhancement modules. The channel-spatial attention submodule is embedded after the fourth spatiotemporal feature enhancement module, and the temporal attention submodule is embedded after the sixth spatiotemporal feature enhancement module.

[0024] Furthermore, the processing flow of the channel-spatial attention submodule is as follows: the input feature map is subjected to global average pooling and global max pooling respectively, and the two generated feature vectors are fed into a two-layer multilayer perceptron with shared weights for processing. The dimension of the intermediate layer of the multilayer perceptron is set to one-sixteenth of the number of input channels; channel attention weights are generated by the Sigmoid activation function and then weighted and fused with the original feature map; the weighted feature map is subjected to average pooling and max pooling along the channel dimension, and the results are concatenated and then input into a set of parallel 3D dilated convolutional layers; its output is then subjected to a convolutional layer with a kernel size of 1×1×1 for dimensionality reduction, and spatial attention weights are generated by the Sigmoid function. Finally, the weights are multiplied element-wise with the feature map.

[0025] Furthermore, the processing flow of the temporal attention submodule is as follows: the input feature map is globally averaged in the spatial dimension H×W and compressed into a temporal feature sequence; the temporal context is modeled by a 3D convolutional layer with kernel_size=5×1×1, and the sigmoid function generates temporal attention weights, which are multiplied element-wise with the original feature map to strengthen key frames and weaken abnormal frames.

[0026] Furthermore, step S3 also includes: the rPPG timing signal is first detrended, and then subjected to a third-order Butterworth bandpass filter of 0.7Hz~4.0Hz to eliminate baseline drift and high-frequency noise.

[0027] Furthermore, the end-to-end spatiotemporal augmentation neural network training uses the PURE and UBFC-rPPG public datasets, with the loss function being a linear combination of the negative Pearson correlation coefficient and the mean squared error; the Adam optimizer has an initial learning rate of 9e-3, which is halved every 10 epochs; the batch size is 4, training lasts for 30 epochs, and the early stopping strategy is to stop training if the mean absolute error of the validation set does not decrease for 5 consecutive epochs.

[0028] The second aspect of this invention provides the following technical solution:

[0029] A non-contact heart rate measurement system based on spatiotemporal augmentation networks includes a hardware module and a software module. The hardware module includes an RGB camera and a computing unit. The RGB camera supports 30fps and 640×480 resolution to capture facial video. The computing unit is equipped with an NVIDIA GPU to perform neural network operations as described above for the non-contact heart rate measurement method. The software module has a built-in neural network as described above to realize an end-to-end process of video preprocessing, physiological signal reconstruction, and heart rate calculation.

[0030] Furthermore, the RGB camera is a Logitech C920HDPro, and the NVIDIA GPU of the computing unit is either a server-side RTX8000 or an embedded NVIDIA Jetson Xavier NX; the number of neural network parameters is ≤0.99M, the server-side inference speed is ≥35FPS, and the embedded speed is ≥28FPS.

[0031] The beneficial effects of this invention are as follows:

[0032] 1. High Precision: By introducing 3D central difference convolution, optimized modeling is specifically implemented for the minute-by-minute variations of rPPG signals, significantly improving the accuracy of heart rate measurement. The mean absolute errors on the UBFC-rPPG and PURE datasets are as low as 0.24 and 0.40, respectively.

[0033] 2. Strong robustness: It combines the dilated convolutional attention module and the temporal attention module, enabling the model to adaptively focus on effective physiological information in the three dimensions of channel, space and time, and suppress noise, and perform stably in complex scenes with head movement and lighting changes.

[0034] 3. Highly efficient end-to-end: Eliminating the need for cumbersome preprocessing steps (such as precise ROI selection or skin segmentation), it achieves end-to-end prediction from raw video frames to heart rate values, simplifying the process and improving efficiency.

[0035] 4. Good generalization ability: Cross-dataset test results show that the model trained by this invention can still maintain excellent performance on unseen datasets, demonstrating strong generalization ability.

[0036] 5. Lightweight model: The entire network has only 0.99M parameters, with relatively low computational overhead, which is conducive to deployment on embedded devices or mobile platforms.

[0037] In summary, this non-contact heart rate measurement scheme based on spatiotemporal augmentation networks achieves high accuracy (MAE of 0.24 for UBFC-rPPG and 0.40 for PURE dataset) and strong anti-interference (suppressing illumination / motion noise) and efficient end-to-end output by using three-dimensional central difference convolution and multi-dimensional attention modules. It also has excellent generalization ability, only 0.99M parameters, and is suitable for deployment on servers (inference ≥35FPS) and embedded devices (inference ≥28FPS). Attached Figure Description

[0038] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0039] Figure 1 A schematic diagram of the overall network structure of STE-rPPGNet provided in this embodiment of the invention;

[0040] Figure 2 : A schematic diagram of the spatial block structure provided in the embodiment of the present invention;

[0041] Figure 3 : A schematic diagram of the spatiotemporal block (integrated 3D-CDC) structure provided in this embodiment of the invention;

[0042] Figure 4 : A schematic diagram of the structure of the dilated convolutional attention module provided in this embodiment of the invention;

[0043] Figure 5 : A schematic diagram of the temporal attention module structure provided in this embodiment of the invention;

[0044] Figure 6(a): Scatter plot of model-predicted heart rate and actual heart rate and linear regression fitting line on the UBFC-rPPG dataset;

[0045] Figure 6(b): Scatter plot of model-predicted heart rate and actual heart rate and linear regression fitting line on the PURE dataset;

[0046] Figure 7(a): Comparison of the rPPG signal (blue curve) predicted on the test set using a model without any attention module with the real BVP signal (red curve);

[0047] Figure 7(b): Comparison of the rPPG signal (blue curve) predicted on the test set using a model with only a temporal attention module and the true BVP signal (red curve);

[0048] Figure 7(c): Comparison of the rPPG signal (blue curve) predicted on the test set using a model with only dilated convolutional attention modules and the true BVP signal (red curve);

[0049] Figure 7(d): Comparison of the rPPG signal predicted on the test set using the full model (blue curve) with the true BVP signal (red curve);

[0050] Figure 8 Performance comparison of ST-Block with different versions of convolution operations on heart rate estimation task (MAE and RMSE). Detailed Implementation

[0051] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0052] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0053] like Figure 1-8 As shown, this invention provides a non-contact heart rate measurement method based on a spatiotemporal augmentation network, the method comprising the following steps performed in sequence:

[0054] S1, Video Data Preprocessing

[0055] The input raw facial video frame sequence is subjected to illumination normalization processing to eliminate the interference of ambient light source color attributes and intensity changes on signal extraction, so that subsequent processing focuses on the relative color changes caused by changes in blood volume; the normalization processing is performed according to the following formula:

[0056]

[0057] in, This represents the original video frame image at frame t. This represents the average pixel value of all video frames within a predetermined time window centered at frame t. This represents the output video frame after normalization.

[0058] S2, Physiological signal reconstruction

[0059] The normalized video frame sequence obtained in step S1 is input into a pre-trained end-to-end spatiotemporal augmentation neural network. The network takes the aforementioned frame sequence as input and outputs a reconstructed one-dimensional remote photoplethysmography (PPG) time-series signal through its internal multi-level feature extraction and fusion mechanism.

[0060] S3, Heart Rate Calculation

[0061] The spectral analysis of the rPPG time-series signal reconstructed in step S2 includes:

[0062] S3.1 Frequency Domain Transformation: Perform a Fast Fourier Transform on the rPPG time-series signal to transform it from the time domain to the frequency domain and obtain the corresponding spectrum;

[0063] S3.2 Dominant Frequency Extraction: Within a preset range of human heart rate physiological frequency, identify the frequency component with the strongest energy in the spectrum as the dominant frequency;

[0064] S3.3, Heart Rate Conversion: Convert the dominant frequency (Unit: Hz) Convert to heart rate (HR) (unit: bpm) using the formula: HR = ×60.

[0065] The structure of an end-to-end spatiotemporal augmentation neural network includes the following core processing modules in sequence:

[0066] 1. Spatial Feature Extraction Module: This module consists of a two-dimensional convolutional layer, a batch normalization layer, and a ReLU activation function connected sequentially. It is configured to extract shallow spatial visual features from a single frame image while keeping the temporal dimension of the input data unchanged.

[0067] 2. Spatiotemporal Feature Enhancement Module: Its core operation is implemented using three-dimensional central difference convolution to replace traditional three-dimensional convolution. This convolution operation achieves accurate modeling of microsecond-level temporal color change patterns caused by pulse by explicitly aggregating the intensity gradient information between the center point on the feature map and its spatiotemporal neighbors, thereby significantly enhancing the network's ability to extract weak physiological temporal features.

[0068] 3. A multi-dimensional attention optimization module, which further includes:

[0069] Channel-Spatial Attention Submodule: This is an enhanced improvement upon the classic convolutional block attention module. Specifically:

[0070] Channel attention unit: By performing global average pooling and global max pooling on the input feature map respectively, two different channel description vectors are generated. After being processed and fused by a multilayer perceptron with shared weights, a channel attention weight map is generated.

[0071] Spatial Attention Unit: Parallel multi-branch dilated convolutional layers are introduced, with dilation rates set to 1, 2 and 4 respectively, to expand the receptive field and capture multi-scale spatial context information while keeping the feature map resolution unchanged. This allows the model to focus its attention more effectively on the effective physiological regions of the face and suppress noise interference from the background and non-skin regions.

[0072] Temporal Attention Submodule: This module first performs global average pooling on the input feature map in the spatial dimension, compressing it into a temporal feature sequence; then, it uses a 3D convolutional layer with a kernel size of 5×1×1 to model the temporal context of the sequence, and generates temporal attention weights through the Sigmoid activation function; finally, through weighting operations, it strengthens the feature responses of key video frames with high information content, while weakening the negative impact of abnormal frames caused by instantaneous violent motion or sudden changes in lighting.

[0073] The following is a specific embodiment: a non-contact heart rate measurement method based on a spatiotemporal augmentation network, the system of which is as follows:

[0074] S1. System Hardware and Software Environment Configuration

[0075] Hardware platform: Includes an RGB camera (such as Logitech C920HDPro) for video capture, and a computing server equipped with an NVIDIA GPU (such as RTX8000) for model computation.

[0076] Software environment: Operating system is Ubuntu 18.04 or Windows 10, with Python 3.8, PyTorch 1.12.1 and dependent libraries (such as OpenCV, dlib, NumPy) installed.

[0077] S2. Preprocessing of facial video data

[0078] Reference Figure 1 The specific steps for the input section are as follows:

[0079] 1. Video capture: Record the user's frontal face video using an RGB camera at a frame rate of 30fps and a resolution of 640×480.

[0080] 2. Face detection and alignment: 68 key points of the face are obtained using the dlib library.

[0081] 3. Facial region cropping and scaling: Based on the face bounding box, the facial region is cropped by expanding it by 10% and then scaled to 128×128 pixels.

[0082] 4. Illumination Normalization: For each frame of image ft, calculate the RGB three-channel average μ(ft) of all pixels within the preceding and following 5 frames. Normalize it according to the formula fnt=ft / μ(ft) to obtain fnt.

[0083] 5. Construct the input tensor: continuously extract normalized images of T=180 frames (corresponding to 6 seconds of video) to form an input tensor of size [3,180,128,128].

[0084] S3. Forward propagation process of the STE-rPPGNet network model

[0085] The network structure parameters of this embodiment are detailed in Appendix I of the specification, and the data flow is described below in conjunction with the accompanying drawings:

[0086] Appendix I: STE-rPPGNet Network Structure Parameter Table

[0087] Layer / Module Name Core configuration parameters Input dimensions [C,T,H,W] Output dimensions [C,T,H,W] Input tensor —— [3,180,128,128] [3,180,128,128] S-Block 2D convolution (k=5, s=1, p=2) + BN + ReLU [3,180,128,128] [16,180,128,128] 3DMaxPool Pooling kernel (1x2x2), step size (1x2x2) [16,180,128,128] [16,180,64,64] ST-Block-1 3D-CDC(k=3x3x3,s=1,p=1)+BN+ReLU [16,180,64,64] [32,180,64,64] ST-Block-2 3D-CDC(k=3x3x3,s=1,p=1)+BN+ReLU [32,180,64,64] [32,180,64,64] ST-Block-3 3D-CDC(k=3x3x3,s=1,p=1)+BN+ReLU [32,180,64,64] [64,180,64,64] ST-Block-4 3D-CDC(k=3x3x3,s=1,p=1)+BN+ReLU [64,180,64,64] [64,180,64,64] DCBAM Channel and spatial attention, with increased convolutional ratio [1,2,4]. [64,180,64,64] [64,180,64,64] ST-Block-5 3D-CDC(k=3x3x3,s=1,p=1)+BN+ReLU [64,180,64,64] [64,180,64,64] ST-Block-6 3D-CDC(k=3x3x3,s=1,p=1)+BN+ReLU [64,180,64,64] [64,180,64,64] TAM 3D convolution (k=5x1x1) + Sigmoid weighted [64,180,64,64] [64,180,64,64] 3DAdaptiveAvgPool Output size (1,1,1) [64,180,64,64] [64,1,1,1] Output layer (3DConv) 3D convolution (k=1x1x1), output channel=1 [64,1,1,1] [1,180,1,1]

[0088] Note:

[0089] [C,T,H,W] represent: number of channels, number of frames, height, and width, respectively.

[0090] Parameter abbreviations: k (kernel_size), s (stride), p (padding), BN (BatchNormalization); 3D-CDC is three-dimensional central difference convolution.

[0091] 1. Initial Spatial Feature Extraction (S-Block): such as Figure 2 As shown, the input tensor first passes through an S-Block. This module consists of a 2D convolution (in_channels=3, out_channels=16, kernel_size=5×5, stride=1, padding=2), a batch normalization layer, and a ReLU activation layer connected sequentially, with an output tensor size of [16, 180, 128, 128].

[0092] 2. Downsampling and Deep Spatiotemporal Feature Extraction: After a three-dimensional max-pooling layer, the feature map size becomes [16, 180, 64, 64]. Subsequently, the feature map passes through multiple ST-Blocks. For example... Figure 3As shown, the core of each ST-Block is a 3D central difference convolution, and this embodiment preferably uses the 3D-CDC_T version. Its parameters are: kernel_size=3×3×3, stride=1, padding=1. This convolutional layer is followed by a batch normalization layer and a ReLU layer. Deeper layers of the network (e.g., after ST-Block-4) embed dilated convolutional attention modules.

[0093] 3. Processing flow of the dilated convolutional attention module:

[0094] Reference Figure 4 Channel attention: The input feature map X is subjected to 3D global average pooling and max pooling respectively, resulting in two one-dimensional vectors. These vectors are fed into a shared two-layer MLP (the intermediate layer has a dimension of C / r, where r is the reduction rate, which can be 16). The outputs of the MLP are summed and then passed through the Sigmoid function to generate channel attention weights f. The weights f are multiplied element-wise with the original input X to obtain the intermediate feature f′.

[0095] Spatial attention: Average pooling and max pooling are performed on f′ along the channel dimension, and the results are concatenated to obtain feature Fc. Three parallel 3D dilated convolutional layers (dilation rates of 1, 2, and 4) are used to process Fc, and the outputs are concatenated to obtain feature D. A 1×1×1 convolutional layer is applied to D to reduce its dimensionality to one channel, and then the spatial attention weights F are generated using the sigmoid function. Finally, the weights F are multiplied element-wise with the intermediate feature f′ to obtain the output of DCBAM.

[0096] 4. Processing flow of the temporal attention module:

[0097] Reference Figure 5 In the later part of the network (e.g., after ST-Block-6), a temporal attention module is embedded;

[0098] Global average pooling is performed on the input feature map along the spatial dimensions (H and W) to obtain the feature Dct. A 3D convolutional layer with a kernel size of 5×1×1 is used to process Dct, and then the temporal attention weights Mweight are generated by the Sigmoid function.

[0099] Multiply Mweight element-wise with the original input to obtain the output of TAM.

[0100] 5. Signal generation: The network ends by sequentially using a 3D adaptive average pooling layer and a 1×1×1 3D convolutional layer to finally convert the feature map into a one-dimensional rPPG signal with a length of 180.

[0101] S4. Heart Rate Calculation

[0102] After detrending and bandpass filtering (e.g., 0.7Hz-4.0Hz) the rPPG signal output by the network, a Fast Fourier Transform is performed. The frequency fpeak corresponding to the peak power spectrum is found in the frequency domain, and the final heart rate HR = fpeak × 60 (unit: BPM).

[0103] S5. Model Training

[0104] Datasets: Training and validation were performed using the publicly available datasets PURE, UBFC-rPPG, and MMPD.

[0105] Loss function: A linear combination of negative Pearson correlation coefficient and mean squared error.

[0106] Optimizer: Using the Adam optimizer, the initial learning rate is... And a step size decay strategy is adopted.

[0107] Hyperparameters: Batch size set to 4, number of training epochs set to 30.

[0108] Through the specific embodiments described above, this invention constructs an end-to-end non-contact heart rate measurement system. This system effectively utilizes spatiotemporal information and attention mechanisms to robustly and accurately extract heart rate from facial videos captured by ordinary cameras, demonstrating significant practical value.

[0109] The method and system of the present invention can be used in special scenarios, including but not limited to neonatal monitoring, to achieve high-precision and stable non-contact heart rate measurement, solving the problems of contact stimulation and insufficient accuracy in the prior art, and has practical medical application value.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A non-contact heart rate measurement method based on spatiotemporal augmentation networks, characterized in that, Includes the following steps: S1. Video Data Preprocessing: Perform illumination normalization on the original facial video frame sequence. in, This represents the original video frame image at frame t. This represents the average pixel value of all video frames within a predetermined time window centered at frame t. This represents the output video frame after normalization. 180 normalized video frames are continuously extracted to construct an input tensor of size [3, 180, 128, 128]. S2. Physiological Signal Reconstruction: The input tensor is input into an end-to-end spatiotemporal augmentation neural network, and the output is a reconstructed one-dimensional rPPG temporal signal. The spatiotemporal augmentation neural network includes a spatial feature extraction module, multiple spatiotemporal feature enhancement modules, and a multidimensional attention optimization module. The spatiotemporal feature enhancement module is based on a three-dimensional central difference convolution with kernel_size=3×3×3, stride=1, and padding=1. The multidimensional attention optimization module includes a channel-space attention submodule and a temporal attention submodule. The channel-space attention submodule contains parallel 3D dilated convolutional layers with dilation rates of 1, 2, and 4, and the temporal attention submodule contains a 3D convolutional layer with kernel_size=5×1×1. S3. Heart Rate Calculation: Perform a Fast Fourier Transform on the rPPG time-series signal to extract the dominant frequency with the strongest energy within the human physiological frequency range of 0.7Hz to 4.0Hz, and calculate it according to the formula. Converted to heart rate value As the dominant frequency, The spatial feature extraction module consists of a 2D convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence. The parameters of the 2D convolutional layer are in_channels=3, out_channels=16, kernel_size=5×5, stride=1, and padding=2. It extracts shallow spatial features of a single frame while keeping the temporal dimension unchanged.

2. The non-contact heart rate measurement method based on spatiotemporal augmentation network according to claim 1, characterized in that, Step S1 also includes capturing facial video at 30fps and 640×480 resolution, extracting 68 key points of the face using the dlib library and aligning them, cropping the facial region by 10% based on the face bounding box, and scaling it to 128×128 pixels using bilinear interpolation.

3. The non-contact heart rate measurement method based on spatiotemporal augmentation network according to claim 1, characterized in that, The spatiotemporal feature enhancement module comprises six modules. After the input tensor is processed by the spatial feature extraction module, it is downsampled through a three-dimensional max pooling layer. The convolution kernel size of the pooling layer is 2×2×2 with a stride of 2, thereby compressing the feature map to a size of [16,180,64,64]. The feature map is then sequentially input into the six spatiotemporal feature enhancement modules. The channel-spatial attention submodule is embedded after the fourth spatiotemporal feature enhancement module, and the temporal attention submodule is embedded after the sixth spatiotemporal feature enhancement module.

4. The non-contact heart rate measurement method based on spatiotemporal augmentation network according to claim 1, characterized in that, The processing flow of the channel-spatial attention submodule is as follows: the input feature map is subjected to global average pooling and global max pooling respectively, and the two generated feature vectors are fed into a two-layer multilayer perceptron with shared weights for processing. The dimension of the intermediate layer of the multilayer perceptron is set to one-sixteenth of the number of input channels; channel attention weights are generated by the sigmoid activation function and then weighted and fused with the original feature map; the weighted feature map is subjected to average pooling and max pooling along the channel dimension, and the results are concatenated and then input into a set of parallel 3D dilated convolutional layers. Its output is then subjected to a dimensionality reduction by a convolutional layer with a kernel size of 1×1×1, and spatial attention weights are generated by the Sigmoid function. Finally, these weights are multiplied element-wise with the feature map.

5. The non-contact heart rate measurement method based on spatiotemporal augmentation network according to claim 1, characterized in that, The processing flow of the temporal attention submodule is as follows: the input feature map is globally averaged in the spatial dimension H×W and compressed into a temporal feature sequence; Temporal context is modeled by a 3D convolutional layer with kernel_size=5×1×1. The sigmoid function generates temporal attention weights, which are multiplied element-wise with the original feature map to enhance key frames and weaken anomalous frames.

6. The non-contact heart rate measurement method based on spatiotemporal augmentation network according to claim 1, characterized in that, Step S3 further includes: the rPPG timing signal is first detrended, and then subjected to a third-order Butterworth bandpass filter of 0.7Hz~4.0Hz to eliminate baseline drift and high-frequency noise.

7. The non-contact heart rate measurement method based on spatiotemporal augmentation network according to claim 1, characterized in that, The end-to-end spatiotemporal augmentation neural network was trained using the PURE and UBFC-rPPG public datasets. The loss function was a linear combination of the negative Pearson correlation coefficient and the mean squared error. The Adam optimizer had an initial learning rate of 9e-3, which was halved every 10 epochs. The batch size was 4, and the training lasted for 30 epochs. The early stopping strategy was to stop training if the mean absolute error of the validation set did not decrease for 5 consecutive epochs.

8. A non-contact heart rate measurement system based on spatiotemporal augmentation networks, characterized in that, It includes a hardware module and a software module; the hardware module includes an RGB camera and a computing unit. The RGB camera supports 30fps and 640×480 resolution to capture facial video. The computing unit is equipped with an NVIDIA GPU to perform neural network operations of the non-contact heart rate measurement method based on spatiotemporal augmentation network as described in any one of claims 1-7. The software module has a built-in spatiotemporal augmentation neural network as described in any one of claims 1-7 to realize an end-to-end process of video preprocessing, physiological signal reconstruction, and heart rate calculation.

9. A non-contact heart rate measurement system based on a spatiotemporal augmentation network according to claim 8, characterized in that, The RGB camera is a Logitech C920HDPro, and the NVIDIA GPU for the computing unit is either a server-side RTX 8000 or an embedded NVIDIA Jetson Xavier NX; the number of neural network parameters is ≤0.99M, the server-side inference speed is ≥35FPS, and the embedded speed is ≥28FPS.

Images