Non-contact heart rate measurement method and system based on space-time enhancement network

Abstract

The invention belongs to the crossing field of computer vision and medical health monitoring, discloses a non-contact heart rate measurement method and system based on a space-time enhancement network, and solves the problems that traditional contact measurement is inconvenient, and an existing rPPG technology is weak in interference resistance. The method comprises the following steps: performing illumination normalization on an original face video frame and constructing an input tensor; an end-to-end space-time enhancement neural network (including three-dimensional central difference convolution and a multi-dimensional attention module) is input to reconstruct an rPPG time sequence signal; and the heart rate is converted through band-pass filtering and fast Fourier transform. According to the scheme, average absolute errors in UBFC-rPPG and PURE data sets reach 0.24 and 0.40 respectively, illumination / motion interference resistance is achieved, end-to-end efficiency is achieved, the network parameter quantity is only 0.99 M, an adaptive server (reasoning is larger than or equal to 35 FPS) and embedded equipment (reasoning is larger than or equal to 28 FPS) are matched, and generalization is excellent.

Description

TECHNICAL FIELD

[0001] The present application belongs to the cross technical field of computer vision and medical health monitoring, and specifically relates to a non-contact heart rate measurement method and system based on a space-time enhanced network, which realizes robust extraction of a remote photoplethysmography (rPPG) signal in a facial video and accurate estimation of heart rate through an end-to-end deep learning network, and can be applied to medical monitoring (such as neonates and burn patients), intelligent wearable devices, health management and the like. BACKGROUND

[0002] As a core physiological parameter reflecting the cardiovascular health status of the human body, accurate measurement of heart rate is of great significance for disease early warning and health assessment. Traditional heart rate measurement relies on contact technology, such as electrocardiogram (ECG) which requires electrode attachment to the skin, and photoplethysmography (PPG) which requires sensor contact with blood vessel dense areas. Such methods have the defects of inconvenience, easy skin irritation, and inability to adapt to special populations such as burn patients or neonates, and are difficult to meet the needs of convenient and non-invasive monitoring.

[0003] To solve the above problems, remote photoplethysmography (rPPG) technology based on facial video has emerged. The principle is to analyze the micro color changes (the amplitude of RGB channel signal fluctuation is only 1 / 100 of the illumination noise) of the facial skin caused by the periodic pulsation of blood volume to infer heart rate. However, the existing rPPG technology still faces three major bottlenecks, and the related technical solutions have obvious limitations:

[0004] Traditional methods such as independent component analysis (ICA) and principal component analysis (POS) rely on the mathematical model assumption of "signal linear separation", and the accuracy drops sharply in complex scenes: for example, in the UBFC-rPPG public data set (University of Louvain, 2019), the mean absolute error (MAE) of the ICA method is 1.2, and the MAE rises to 1.8 in the head motion scene, which cannot meet the medical level precision requirement (MAE≤0.5).

[0005] The core technology of patent CN115024706A, a non-contact heart rate measurement method fusing ConvLSTM and CBAM, is "ConvLSTM time series modeling + CBAM channel-spatial attention", but ConvLSTM focuses on long-time time series dependent learning and cannot accurately capture microsecond-level color gradient changes of rPPG signals. And CBAM has no multi-branch dilated convolution, and the spatial receptive field is limited to 3x3, making it difficult to suppress noise from facial backgrounds and non-skin areas - in the PURE data set (PhysioNet, 2015), the MAE of this method is 0.75, and the MAE rises to 1.1 in low light (<50 lux) scenes.

[0006] Patent CN114092855A based on deep learning Non-contact intelligent human heart rate prediction method adopts "CNN feature extraction + SVR regression model" non-end-to-end scheme, which needs to manually crop the face region of interest (ROI) and perform signal preprocessing, the process is complicated and easy to introduce artificial error; At the same time, it does not fuse the space-time feature, and in the scene of violent head movement (displacement>10cm), the heart rate prediction error is more than 20%.

[0007] 3D Convolutional Neural Networks for Remote Photoplethysmography Signal Extraction (IEEE ICIP 2020, DOI:10.1109 / ICIP40778.2020.9191064) first used 3D convolution for rPPG signal extraction, but used traditional 3D convolution (kernel_size=5x5x5), which had the problems of "feature redundancy and large parameter quantity" (parameter quantity 4.2M); And only introduces channel attention, without involving space-time collaborative optimization, in the scene of dark skin population (melanin index>50), the signal extraction success rate decreases by 40%.

[0008] 3D center difference convolution (3D-CDC) is traditionally only found in academic research on dynamic image edge detection (such as the paper "3D Center Difference Convolution for Video Edge Detection", which clearly states its application scenario as "edge gradient capture of dynamic video frames", and the technical personnel in the field generally believe that it "can only process strong gradient edge signals and cannot adapt to rPPG weak physiological signals";

[0009] Dilated convolution is more commonly used in large-scale modeling of semantic segmentation (such as the paper "Dilated Residual Networks", CVPR 2017, DOI:10.1109 / CVPR.2017.634), and has never been combined with the "focus on facial local physiological area" requirement of rPPG;

[0010] At the same time, the parameter quantity of existing deep learning schemes is generally more than 3M (such as the model parameter quantity of 4.2M in the above IEEE ICIP literature), and the inference speed is less than 25FPS, which is difficult to deploy to embedded devices (such as smart watches, portable monitors).

[0011] In summary, the existing technology cannot simultaneously meet the non-contact heart rate measurement requirements of "high accuracy, strong robustness, and lightweight end-to-end", and there is an urgent need for innovative solutions to break through the above technical bottlenecks. SUMMARY

[0012] Therefore, the application provides a non-contact heart rate measurement method and system based on a space-time enhanced network.

[0013] To achieve the above-mentioned purpose, one of the application provides the following technical solutions:

[0014] A non-contact heart rate measurement method based on a space-time enhanced network comprises the following steps:

[0015] S1, video data preprocessing: performing illumination normalization on the original face video frame sequence

[0016]

[0017] wherein, represents the t-th frame of the original video frame image, represents the pixel average value of all video frames in a predetermined time window centered on the t-th frame, represents the output video frame after normalization processing,

[0018] 180 frames of normalized video frames are continuously intercepted to construct an input tensor with a size of [3, 180, 128, 128];

[0019] S2, physiological signal reconstruction: inputting the input tensor into an end-to-end space-time enhanced neural network to output a reconstructed one-dimensional rPPG time series signal; the space-time enhanced neural network comprises a spatial feature extraction module, a plurality of space-time feature enhancement modules and a multi-dimensional attention optimization module in sequence; the space-time feature enhancement module takes a three-dimensional center difference convolution with kernel_size=3×3×3, stride=1 and padding=1 as the core; the multi-dimensional attention optimization module comprises a channel-spatial attention submodule and a time series attention submodule, the channel-spatial attention submodule comprises parallel 3D dilated convolution layers with dilations of 1, 2 and 4, and the time series attention submodule comprises a 3D convolution layer with kernel_size=5×1×1;

[0020] S3, heart rate value calculation: performing fast Fourier transform on the rPPG time series signal to extract the dominant frequency with the strongest energy in the human physiological frequency range of 0.7Hz~4.0Hz, and converting the dominant frequency into a heart rate value according to the formula is the dominant frequency.

[0021] ​Further, step S1 further comprises: collecting a face video at 30fps, 640x480 resolution, extracting 68 key points of the face by the dlib library and aligning, cropping the face region by expanding 10% according to the face bounding box, and scaling to 128x128 pixels by bilinear interpolation.

[0022] Further, the spatial feature extraction module is sequentially connected by a 2D convolution layer, a batch normalization layer and a ReLU activation function; the 2D convolution layer has parameters of in_channels=3, out_channels=16, kernel_size=5x5, stride=1 and padding=2, extracts a single-frame shallow spatial feature and keeps the time dimension unchanged.

[0023] Further, the spatio-temporal feature enhancement module has a total of 6; after the input tensor is processed by the spatial feature extraction module, it is down-sampled by a three-dimensional maximum pooling layer, wherein the convolution kernel size of the pooling layer is 2x2x2 and the step is 2, so as to compress the feature map to the size [16, 180, 64, 64], and then sequentially input into the 6 spatio-temporal feature enhancement modules; wherein the channel-spatial attention submodule is embedded after the 4th spatio-temporal feature enhancement module, and the time sequence attention submodule is embedded after the 6th spatio-temporal feature enhancement module.

[0024] Further, the processing flow of the channel-spatial attention submodule is: the input feature map is respectively subjected to global average pooling and global maximum pooling, the two generated feature vectors are input into a two-layer multilayer perceptron sharing weights for processing, the dimension of the intermediate layer of the multilayer perceptron is set to one-sixteenth of the input channel number; the channel attention weight is generated by the Sigmoid activation function, and is fused with the original feature map; the weighted feature map is subjected to average pooling and maximum pooling along the channel dimension, and the results are spliced, and then input into a group of parallel 3D dilated convolution layers; the output is reduced in dimension by a convolution layer with a kernel size of 1x1x1, the spatial attention weight is generated by the Sigmoid function, and finally the weight is multiplied with the feature map element by element.

[0025] Further, the processing flow of the time sequence attention submodule is: the input feature map is globally averaged pooled in the spatial dimension HxW, and is compressed into a time sequence feature sequence; the 3D convolution layer with kernel_size=5x1x1 is used to model the temporal context, the time sequence attention weight is generated by Sigmoid, and is multiplied with the original feature map element by element to strengthen the key frame and weaken the abnormal frame.

[0026] Further, step S3 further comprises: the rPPG time sequence signal is first detrended, and then subjected to 0.7Hz-4.0Hz three-order Butterworth band-pass filtering to eliminate baseline drift and high-frequency noise.

[0027] Further, the end-to-end spatio-temporal enhancement neural network is trained: the PURE and UBFC-rPPG public datasets are adopted, the loss function is a linear combination of negative Pearson correlation coefficient and mean square error, the initial learning rate of the Adam optimizer is 9e-3, and is halved every 10 epochs, the batch size is 4, 30 epochs are trained, and the early stopping strategy is to stop when the average absolute error of the validation set does not decrease for 5 consecutive epochs.

[0028] The second aspect of the application provides the following technical scheme:

[0029] A non-contact heart rate measurement system based on a spatio-temporal enhancement network comprises a hardware module and a software module; the hardware module comprises an RGB camera and a computing unit, the RGB camera supports 30fps and 640*480 resolution to collect facial videos, and the computing unit is provided with an NVIDIA GPU to perform neural network operation of the non-contact heart rate measurement method as described above; the software module is built-in the neural network as described above, and realizes an end-to-end process of video preprocessing, physiological signal reconstruction and heart rate calculation.

[0030] Further, the RGB camera is a Logitech C920 HD Pro, the NVIDIA GPU of the computing unit is a server-side RTX8000 or an embedded-side NVIDIA Jetson Xavier NX; the parameter quantity of the neural network is less than or equal to 0.99M, the inference speed of the server side is greater than or equal to 35FPS, and the inference speed of the embedded side is greater than or equal to 28FPS.

[0031] The application has the following beneficial effects:

[0032] 1. High precision: by introducing 3D center difference convolution, the rPPG signal micro-time sequence change is optimized and modeled, and the accuracy of heart rate measurement is greatly improved. The average absolute error on the UBFC-rPPG and PURE datasets is as low as 0.24 and 0.40 respectively.

[0033] 2. Strong robustness: combined with the dilated convolution attention module and the time sequence attention module, the model can adaptively focus on effective physiological information in the channel, space and time dimensions, and suppress noise, and performs stably in complex scenes with head movement and light changes.

[0034] 3. Efficient end-to-end: without complicated preprocessing steps (such as accurate ROI selection or skin segmentation), end-to-end prediction from raw video frames to heart rate values is realized, the process is simplified, and the efficiency is improved.

[0035] 4. Good generalization: cross-dataset test results show that the model trained by the application can still maintain excellent performance on unseen datasets, showing strong generalization ability.

[0036] 5. Model lightweight: the entire network parameter quantity is only 0.99M, and the calculation overhead is relatively small, which is conducive to deployment to embedded devices or mobile platforms.

[0037] In summary, the non-contact heart rate measurement scheme based on the spatio-temporal enhancement network, by three-dimensional center difference convolution and multi-dimensional attention module, realizes high precision (UBFC-rPPG, PURE dataset MAE is 0.24, 0.40 respectively), strong anti-interference (suppresses light / motion noise), end-to-end efficient output, and good generalization, and the parameter quantity is only 0.99M, which is suitable for server (inference >= 35FPS) and embedded device (inference >= 28FPS) deployment. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to make the purpose, technical scheme and advantages of the present application clearer, the preferred detailed description of the present application will be made below in combination with the drawings, in which:

[0039] Figure 1 : The STE-rPPGNet overall network structure schematic diagram provided by the embodiment of the present application;

[0040] Figure 2 : The spatial block structure schematic diagram provided by the embodiment of the present application;

[0041] Figure 3 : The spatio-temporal block (integrating 3D-CDC) structure schematic diagram provided by the embodiment of the present application;

[0042] Figure 4 : The dilated convolution attention module structure schematic diagram provided by the embodiment of the present application;

[0043] Figure 5 : The timing attention module structure schematic diagram provided by the embodiment of the present application;

[0044] Fig. 6 (a): scatter plot and linear regression fitting line of model predicted heart rate and true heart rate on UBFC-rPPG dataset;

[0045] Fig. 6 (b): scatter plot and linear regression fitting line of model predicted heart rate and true heart rate on PURE dataset;

[0046] Fig. 7 (a): comparison of rPPG signal (blue curve) predicted by the model without any attention module and true BVP signal (red curve) on the test set;

[0047] Fig. 7 (b): comparison of rPPG signal (blue curve) predicted by the model containing only timing attention module and true BVP signal (red curve) on the test set;

[0048] Figure 7(c): Comparison of the rPPG signal predicted on the test set (blue curve) using the model with only the dilated convolution attention module and the true BVP signal (red curve);

[0049] Figure 7(d): Comparison of the rPPG signal predicted on the test set (blue curve) using the full model and the true BVP signal (red curve);

[0050] Figure 8 Performance comparison of ST-Block with different versions of convolution operations on the heart rate estimation task (MAE and RMSE). DETAILED DESCRIPTION

[0051] The present application can be implemented or applied in other different specific embodiments, and various modifications or changes can be made to the details of the present application based on different views and applications without departing from the spirit of the present application. It should be noted that the drawings provided in the following examples only illustrate the basic concept of the present application in a schematic manner, and the following examples and features in the examples can be combined with each other without conflict.

[0052] The drawings are only used for illustrative explanation, and the representation is only a schematic diagram, not a physical diagram, and should not be understood as a limitation of the present application; in order to better illustrate the embodiments of the present application, some components of the drawings may be omitted, enlarged or reduced, and do not represent the actual size of the product; for those skilled in the art, it is understandable that some known structures and their descriptions in the drawings may be omitted.

[0053] As shown in Figures 1-8 The present application provides a non-contact heart rate measurement method based on a space-time enhanced network, which comprises the following steps executed in the following order:

[0054] S1, video data preprocessing

[0055] The input original face video frame sequence is subjected to illumination normalization processing to eliminate the interference of environmental light source color properties and intensity changes on signal extraction, so that the subsequent processing focuses on the relative color changes caused by blood volume changes; the normalization processing is performed according to the following formula:

[0056]

[0057] wherein, represents the t-th original video frame image, represents the pixel average value of all video frames within a predetermined time window centered on the t-th frame, represents the normalized output video frame.

[0058] S2, physiological signal reconstruction

[0059] The normalized video frame sequence obtained in step S1 is input into a pre-trained end-to-end spatio-temporal enhancement neural network; the network takes the aforementioned frame sequence as input, and outputs a reconstructed one-dimensional remote photoplethysmography time series signal through the internal multi-level feature extraction and fusion mechanism.

[0060] S3, heart rate value calculation

[0061] The rPPG time series signal reconstructed in step S2 is subjected to spectral analysis, specifically including:

[0062] S3.1, frequency domain transformation: performing fast Fourier transform on the rPPG time series signal to convert it from time domain to frequency domain, obtaining the corresponding frequency spectrum;

[0063] S3.2, dominant frequency extraction: identifying the strongest energy frequency component in the frequency spectrum as the dominant frequency within the preset human heart rate physiological frequency range;

[0064] S3.3, heart rate conversion: converting the dominant frequency (unit: Hz) into a heart rate value HR (unit: bpm), and the conversion formula is: HR = f × 60.

[0065] The structure of the end-to-end spatio-temporal enhancement neural network successively includes the following core processing modules:

[0066] 1. Spatial feature extraction module: composed of a two-dimensional convolution layer, a batch normalization layer and a ReLU activation function connected in sequence; this module is configured to extract shallow spatial visual features from single-frame images while keeping the time dimension of the input data unchanged.

[0067] 2. Spatio-temporal feature enhancement module: its core operation adopts three-dimensional central difference convolution to replace the traditional three-dimensional convolution; this convolution operation explicitly aggregates the intensity gradient information between the center point and its spatio-temporal neighborhood points on the feature map, accurately models the microsecond-level time series color change pattern caused by the pulse, and significantly enhances the network's ability to extract weak physiological time series features.

[0068] 3. Multi-dimensional attention optimization module, which further includes:

[0069] Channel-spatial attention sub-module: this is an enhanced improvement of the classic convolution block attention module. Among them:

[0070] Channel attention unit: Two different channel description vectors are generated by performing global average pooling and global max pooling on the input feature map, respectively. After processing and fusion through a multi-layer perceptron with shared weights, the channel attention weight map is generated.

[0071] Spatial attention unit: Parallel multi-branch dilated convolution layers are introduced, with dilation rates of 1, 2, and 4, respectively. These layers are used to expand the receptive field and capture multi-scale spatial context information while maintaining the resolution of the feature map. This allows the model to focus its attention on the effective physiological regions of the face and suppress noise interference from the background and non-skin areas.

[0072] Temporal attention sub-module: This module first performs global average pooling on the input feature map in the spatial dimension, compressing it into a temporal feature sequence. Then, a 3D convolution layer with a kernel size of 5x1x1 is used to model the temporal context, and a Sigmoid activation function is used to generate the temporal attention weight. Finally, through weighted operation, the feature responses of key video frames with high information content are enhanced, while the negative effects of abnormal frames caused by instantaneous intense motion or sudden changes in illumination are weakened.

[0073] The following embodiment is specifically provided: a non-contact heart rate measurement method based on a spatio-temporal enhancement network, which adopts a system comprising:

[0074] S1. System hardware and software environment configuration

[0075] Hardware platform: includes an RGB camera (such as Logitech C920 HD Pro) for video acquisition, and a computing server equipped with NVIDIA GPU (such as RTX8000) for model operation.

[0076] Software environment: the operating system is Ubuntu 18.04 or Windows 10, and Python 3.8, PyTorch 1.12.1 and dependent libraries (such as OpenCV, dlib, NumPy) are installed.

[0077] S2. Preprocessing of facial video data

[0078] Referring to the input part of Figure 1 , the specific steps are as follows:

[0079] 1. Video acquisition: use an RGB camera to record the user's front face video at a frame rate of 30fps and a resolution of 640x480.

[0080] 2. Face detection and alignment: use the dlib library to obtain 68 key points of the face.

[0081] 3. Face region cropping and scaling: According to the face bounding box, the face region is cropped after being expanded by 10% and scaled to 128x128 pixels.

[0082] 4. Illumination normalization: For each frame image ft, the RGB three-channel average μ(ft) of all pixels in the previous and subsequent five frames is calculated. Normalization is performed according to the formula fnt=ft / μ(ft) to obtain fnt.

[0083] 5. Construction of input tensor: The normalized images of T=180 frames (corresponding to 6 seconds of video) are continuously intercepted to form an input tensor with a size of [3, 180, 128, 128].

[0084] S3. Forward propagation process of STE-rPPGNet network model

[0085] The network structure parameters of the present embodiment are shown in Table I in the specification, and the data flow is described as follows in combination with the accompanying drawings:

[0086] Table I: STE-rPPGNet network structure parameter table

[0087] Layer / module name Core configuration parameters Input dimensions [C, T, H, W] Output dimensions [C, T, H, W] Input tensor —— [3,180,128,128] [3,180,128,128] S-Block 2D convolution (k=5, s=1, p=2) + BN + ReLU [3,180,128,128] [16,180,128,128] 3D MaxPool Pooling kernel (1x2x2), stride (1x2x2) [16,180,128,128] [16,180,64,64] ST-Block-1 3D-CDC (k=3x3x3, s=1, p=1) + BN + ReLU [16,180,64,64] [32,180,64,64] ST-Block-2 3D-CDC (k=3x3x3, s=1, p=1) + BN + ReLU [32,180,64,64] [32,180,64,64] ST-Block-3 3D-CDC (k=3x3x3, s=1, p=1) + BN + ReLU [32,180,64,64] [64,180,64,64] ST-Block-4 3D-CDC (k=3x3x3, s=1, p=1) + BN + ReLU [64,180,64,64] [64,180,64,64] DCBAM Channel and spatial attention, dilation convolution rate [1, 2, 4] [64,180,64,64] [64,180,64,64] ST-Block-5 3D-CDC (k=3x3x3, s=1, p=1) + BN + ReLU [64,180,64,64] [64,180,64,64] ST-Block-6 3D-CDC (k=3x3x3, s=1, p=1) + BN + ReLU [64,180,64,64] [64,180,64,64] TAM 3D convolution (k=5x1x1) + Sigmoid weighting [64,180,64,64] [64,180,64,64] 3D AdaptiveAvgPool Output size (1, 1, 1) [64,180,64,64] [64,1,1,1] Output layer (3DConv) 3D convolution (k=1x1x1), output channels = 1 [64,1,1,1] [1,180,1,1]

[0088] Notes:

[0089] [C, T, H, W] respectively represent: channel number, frame number, height, and width.

[0090] Parameter abbreviations: k (kernel size, convolution kernel size), s (stride, step), p (padding, padding), BN (Batch Normalization, batch normalization); 3D-CDC is three-dimensional central difference convolution.

[0091] 1. Initial spatial feature extraction (S-Block): As shown in Figure 2 , the input tensor is first passed through an S-Block. This module is composed of a 2D convolution (in_channels=3, out_channels=16, kernel_size=5x5, stride=1, padding=2), a batch normalization layer, and a ReLU activation layer connected in sequence, and the output tensor size is [16, 180, 128, 128].

[0092] 2. Down-sampling and deep spatio-temporal feature extraction: After passing through a three-dimensional maximum pooling layer, the feature map size becomes [16, 180, 64, 64]. Subsequently, the feature map is sequentially passed through multiple ST-Blocks. As shown in Figure 3As shown, the core of each ST-Block is 3D center difference convolution, and the preferred embodiment is 3D-CDC_T version. Its parameters are: kernel_size=3x3x3, stride=1, padding=1. After this convolution layer, a batch normalization layer and a ReLU layer are also connected. In the deep part of the network (for example, after ST-Block-4), an expanded convolution attention module is embedded.

[0093] 3. Processing flow of expanded convolution attention module:

[0094] Referring to Figure 4 Channel attention: Perform three-dimensional global average pooling and maximum pooling on the input feature map X respectively, and obtain two one-dimensional vectors. Send them to a shared two-layer MLP (the intermediate layer dimension is C / r, r is the reduction rate, which can be 16), add the outputs of the MLP, and pass them through the Sigmoid function to generate channel attention weights f. Multiply the weights f and the original input X element by element to obtain the intermediate feature f';

[0095] Spatial attention: Perform average pooling and maximum pooling on f' in the channel dimension respectively, and concatenate the results to obtain the feature Fc. Use three parallel 3D dilated convolution layers (dilation rates are 1, 2, and 4 respectively) to process Fc, and concatenate the outputs to obtain the feature D. Use a 1x1x1 convolution layer to reduce D to 1 channel, and then pass it through the Sigmoid function to generate spatial attention weights F. Finally, multiply the weights F and the intermediate feature f' element by element to obtain the output of DCBAM.

[0096] 4. Processing flow of time series attention module:

[0097] Referring to Figure 5 In the later part of the network (for example, after ST-Block-6), a time series attention module is embedded;

[0098] Perform global average pooling on the input feature map in the spatial dimension (H and W) to obtain the feature Dct. Use a 3D convolution layer with a convolution kernel size of 5x1x1 to process Dct, and then pass it through the Sigmoid function to generate time series attention weights Mweight;

[0099] Multiply MweightMweight and the original input element by element to obtain the output of TAM.

[0100] 5. Signal generation: At the end of the network, use a three-dimensional adaptive average pooling layer and a 1x1x1 3D convolution layer in sequence to finally convert the feature map into a one-dimensional rPPG signal with a length of 180.

[0101] S4. Heart rate calculation

[0102] After detrending and band-pass filtering (e.g. 0.7 Hz-4.0 Hz) of the rPPG signal output by the network, a fast Fourier transform is performed. The frequency fpeak corresponding to the power spectrum peak is found in the frequency domain, and the final heart rate HR = fpeak x 60 (unit: BPM) is obtained.

[0103] S5. Training of the model

[0104] Dataset: The public datasets PURE, UBFC-rPPG and MMPD are used for training and validation.

[0105] Loss function: A linear combination of negative Pearson correlation coefficient and mean square error is adopted.

[0106] Optimizer: The Adam optimizer is used with an initial learning rate of and a step decay strategy.

[0107] Hyperparameters: The batch size is set to 4 and the number of training epochs is 30.

[0108] The present application constructs an end-to-end non-contact heart rate measurement system through the above specific embodiments. The system can effectively utilize the spatio-temporal information and attention mechanism to robustly and accurately extract the heart rate from the facial video captured by an ordinary camera, and has important practical value.

[0109] The method and system of the present application can be used to realize high-precision and stable non-contact heart rate measurement in special scenarios including but not limited to neonatal care, and solve the problems of contact stimulation and insufficient precision in the prior art, and have practical medical application value.

[0110] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and all should be covered in the scope of the claims of the present application.

Claims

1. A non-contact heart rate measurement method based on spatiotemporal augmentation networks, characterized in that, Includes the following steps: S1. Video Data Preprocessing: Perform illumination normalization on the original facial video frame sequence. in, This represents the original video frame image at frame t. This represents the average pixel value of all video frames within a predetermined time window centered at frame t. This represents the output video frame after normalization. 180 normalized video frames are continuously extracted to construct an input tensor of size [3, 180, 128, 128]. S2. Physiological Signal Reconstruction: The input tensor is input into an end-to-end spatiotemporal augmentation neural network, and the output is a reconstructed one-dimensional rPPG temporal signal. The spatiotemporal augmentation neural network includes a spatial feature extraction module, multiple spatiotemporal feature enhancement modules, and a multidimensional attention optimization module. The spatiotemporal feature enhancement module is based on a three-dimensional central difference convolution with kernel_size=3×3×3, stride=1, and padding=1. The multidimensional attention optimization module includes a channel-space attention submodule and a temporal attention submodule. The channel-space attention submodule contains parallel 3D dilated convolutional layers with dilation rates of 1, 2, and 4, and the temporal attention submodule contains a 3D convolutional layer with kernel_size=5×1×1. S3. Heart Rate Calculation: Perform a Fast Fourier Transform on the rPPG time-series signal to extract the dominant frequency with the strongest energy within the human physiological frequency range of 0.7Hz to 4.0Hz, and calculate it according to the formula. Converted to heart rate value The dominant frequency.

2. The non-contact heart rate measurement method based on spatiotemporal augmentation network according to claim 1, characterized in that, Step S1 also includes capturing facial video at 30fps and 640×480 resolution, extracting 68 key points of the face using the dlib library and aligning them, cropping the facial region by 10% based on the face bounding box, and scaling it to 128×128 pixels using bilinear interpolation.

3. The non-contact heart rate measurement method based on spatiotemporal augmentation network according to claim 1, characterized in that, The spatial feature extraction module consists of a 2D convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence. The parameters of the 2D convolutional layer are in_channels=3, out_channels=16, kernel_size=5×5, stride=1, and padding=2. It extracts shallow spatial features of a single frame while keeping the temporal dimension unchanged.

4. The non-contact heart rate measurement method based on spatiotemporal augmentation network according to claim 1, characterized in that, The spatiotemporal feature enhancement module comprises six modules. After the input tensor is processed by the spatial feature extraction module, it is downsampled through a three-dimensional max pooling layer. The convolution kernel size of the pooling layer is 2×2×2 with a stride of 2, thereby compressing the feature map to a size of [16,180,64,64]. The feature map is then sequentially input into the six spatiotemporal feature enhancement modules. The channel-spatial attention submodule is embedded after the fourth spatiotemporal feature enhancement module, and the temporal attention submodule is embedded after the sixth spatiotemporal feature enhancement module.

5. The non-contact heart rate measurement method based on spatiotemporal augmentation network according to claim 1, characterized in that, The processing flow of the channel-spatial attention submodule is as follows: the input feature map is subjected to global average pooling and global max pooling respectively, and the two generated feature vectors are fed into a two-layer multilayer perceptron with shared weights for processing. The dimension of the intermediate layer of the multilayer perceptron is set to one-sixteenth of the number of input channels; channel attention weights are generated by the sigmoid activation function and then weighted and fused with the original feature map; the weighted feature map is subjected to average pooling and max pooling along the channel dimension, and the results are concatenated and then input into a set of parallel 3D dilated convolutional layers. Its output is then subjected to a dimensionality reduction by a convolutional layer with a kernel size of 1×1×1, and spatial attention weights are generated by the Sigmoid function. Finally, these weights are multiplied element-wise with the feature map.

6. The non-contact heart rate measurement method based on spatiotemporal augmentation network according to claim 1, characterized in that, The processing flow of the temporal attention submodule is as follows: the input feature map is globally averaged in the spatial dimension H×W and compressed into a temporal feature sequence; Temporal context is modeled by a 3D convolutional layer with kernel_size=5×1×1. The sigmoid function generates temporal attention weights, which are multiplied element-wise with the original feature map to enhance key frames and weaken anomalous frames.

7. The non-contact heart rate measurement method based on spatiotemporal augmentation network according to claim 1, characterized in that, Step S3 further includes: the rPPG timing signal is first detrended, and then subjected to a third-order Butterworth bandpass filter of 0.7Hz~4.0Hz to eliminate baseline drift and high-frequency noise.

8. The non-contact heart rate measurement method based on spatiotemporal augmentation network according to claim 1, characterized in that, The end-to-end spatiotemporal augmentation neural network was trained using the PURE and UBFC-rPPG public datasets. The loss function was a linear combination of the negative Pearson correlation coefficient and the mean squared error. The Adam optimizer had an initial learning rate of 9e-3, which was halved every 10 epochs. The batch size was 4, and the training lasted for 30 epochs. The early stopping strategy was to stop training if the mean absolute error of the validation set did not decrease for 5 consecutive epochs.

9. A non-contact heart rate measurement system based on spatiotemporal augmentation networks, characterized in that, It includes a hardware module and a software module; the hardware module includes an RGB camera and a computing unit. The RGB camera supports 30fps and 640×480 resolution to capture facial video. The computing unit is equipped with an NVIDIA GPU to perform neural network operations of the non-contact heart rate measurement method based on spatiotemporal augmentation network as described in any one of claims 1-8. The software module has a built-in spatiotemporal augmentation neural network as described in any one of claims 1-8 to realize an end-to-end process of video preprocessing, physiological signal reconstruction, and heart rate calculation.

10. A non-contact heart rate measurement system based on a spatiotemporal augmentation network according to claim 9, characterized in that, The RGB camera is a Logitech C920HDPro, and the NVIDIA GPU for the computing unit is either a server-side RTX 8000 or an embedded NVIDIA Jetson Xavier NX; the number of neural network parameters is ≤0.99M, the server-side inference speed is ≥35FPS, and the embedded speed is ≥28FPS.

Images