Non-contact heart rate detection method based on mixed multistage feature fusion

Abstract

The invention provides a non-contact heart rate detection method based on mixed multi-level feature fusion, which takes a face video as input and a heart rate value as output to realize direct mapping from the face video to the heart rate. Comprising five steps of data preprocessing and enhancement, multi-scale attention fusion module processing, cascade multi-layer feature fusion processing, model-independent meta-learning (MAML) training strategy, and mapping of multi-dimensional fusion features into one-dimensional BVP signals by using a signal estimator. And the generalization ability, the detail feature capturing ability and the dependency on the marked data of the model are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of biomedical signal processing, and in particular relates to a non-contact heart rate detection method based on hybrid multi-level feature fusion. Background Art

[0002] Heart rate is an important clinical indicator for assessing cardiopulmonary and hemodynamic stability. Given its clinical relevance in triage decisions, diagnosis, and prognosis, continuous heart rate monitoring is particularly important for people with lung and respiratory diseases. Traditional heart rate detection, primarily obtained from electrocardiogram (ECG) and photoplethysmograph (PPG) signals, relies on specific sensors contacting the subject's skin, and this method presents numerous inconveniences.

[0003] Contactless heart rate monitoring is a low-invasive method that overcomes the shortcomings of traditional heart rate monitoring. With the continuous advancement of image processing technology and deep learning algorithms, remote photoplethysmography (rPPG) signals based on facial videos have shown great potential for remote health monitoring. However, the robustness of existing rPPG technology is susceptible to illumination variations and motion artifacts. In recent years, researchers have focused on reducing or eliminating these adverse effects to improve system robustness and reliability. Common rPPG signal extraction methods include model-based methods, unsupervised data-driven methods, and supervised deep learning methods. These methods can provide reliable signal extraction under relatively stable environmental conditions. However, the performance of existing algorithms fluctuates with illumination changes, subject motion, and other factors, and their applicability and robustness in complex environments need further improvement. Existing transformer-based methods, while demonstrating powerful feature learning capabilities for rPPG signal extraction, still face challenges such as an inability to effectively capture long-term dependencies and subtle changes, reliance on large amounts of data, and susceptibility to changes in external factors. Further optimization is needed. Summary of the Invention

[0004] The purpose of the present invention is to provide a non-contact heart rate detection method based on hybrid multi-level feature fusion, which takes facial video as input and heart rate value as output, realizes direct mapping from facial video to heart rate, optimizes the model to cope with the errors of detection results caused by different lighting, motion and facial changes, and improves the applicability and robustness of signal extraction in complex environments.

[0005] The objective of the present invention is achieved through the following technical solution: a non-contact heart rate detection method based on hybrid multi-level feature fusion, which takes facial video as input and heart rate value as output, and realizes direct mapping from facial video to heart rate, comprising the following steps: Step 1: Data preprocessing and enhancement: Identify the face area from the face video, roughly segment the video irrelevant information, and then perform face alignment to obtain facial video data. For the processing of facial color video data, only the green channel is selected for spatial feature reconstruction. The single-channel k-SVD algorithm is used for data enhancement to retain feature information related to the rPPG signal. Step 2: Multi-scale attention fusion module processing: The data feature information processed in step 1 is processed by the multi-scale attention fusion module, and the spatial attention mechanism is used to refine and fuse the central differential convolution features of the middle and low layers to obtain fusion features at different levels. , where represents the Hadamard product, represents the original features, and represent average pooling and maximum pooling respectively, represents a 3D convolutional layer, represents the sigmoid activation function, Indicates the number of vectors, low indicates low-level features, mid indicates mid-level features, represents the fine fusion feature, Represents the basic vector features; Step 3: Cascade multi-layer feature fusion processing: The fused features of step 2 are pre-processed by module encoding, and the local information in the image is converted into vector form, so that it can be input into the subsequent transformer module to obtain: = In the formula is the region of interest taken out from the multi-scale fusion module, The convolution kernel size is , the step length is The 3D convolutional layer, Each mapped tile Flatten into a one-dimensional vector to get the final module encoding result ,Will The input is sent to the cascaded multi-layer feature fusion network constructed by connecting 12 transformer blocks in series, which realizes the effective fusion of features at different levels and the comprehensive flow of information, and outputs a multi-level mixed feature fusion map. ; Step 4: Model-agnostic meta-learning (MAML) training strategy: Based on the multi-scale attention fusion module, we introduce MAML training strategy optimization. By optimizing the initial parameter settings of the multi-scale attention fusion module, we find the model initial parameters that can quickly adapt to these changes in conditions to cope with different lighting, motion, and facial changes. Step 5: Use the signal estimator to map the multi-dimensional fusion features into a one-dimensional BVP signal: The multi-level mixed feature fusion map output by the cascaded multi-layer feature fusion network Two consecutive average pooling operations are performed, and then two independent channel-level convolution filters are used on the multi-channel spatiotemporal representation to extract features and reduce the dimensionality of the one-dimensional spatiotemporal representation stream, so that the output becomes a smooth one-dimensional sequence to obtain the rPPG signal. The heart rate information is extracted by analyzing the rPPG signal spectrum.

[0006] In the above technical solution, the step 2: the multi-scale attention fusion module uses multiple three-dimensional center difference convolution layers to process the face video sequence, and uses batch normalization and ReLU activation function after each layer to extract deeper features. By using jump connections and spatial attention modules between consecutive three-dimensional center difference convolution blocks, the network can maintain the key information of the original video input and enhance the information of the spatial area of ​​interest.

[0007] In the above technical solution, in step 3, The input is then fed into a cascaded multi-layer feature fusion network consisting of 12 transformer blocks connected in series. The specific process is as follows: , where is the output of the transformer block in the previous layer, To perform residual block operation on the current transformer layer, is the output of the transformer block in the current layer. Then, two consecutive upsampling block operations are performed, where each upsampling block consists of four layers. The first layer is the upsampling operation, and the second layer is the convolution kernel size. The 3D convolution is followed by a normalization and ELU activation operation.

[0008] In the above technical solution, in step 4, the specific steps of introducing the MAML training strategy optimization are as follows: Step 1: Set up the task set ,in Indicates the number of task sets, each task Corresponding to different scenarios and having corresponding data sets , which contains the video frames reconstructed with enhanced spatial features and the corresponding true rPPG signal As target output; Step 2: Fusion of multi-scale attention modules The initial parameters are expressed as , the cascaded multi-layer feature fusion module is represented as , the initial parameters are expressed as ; Step 3: For each task , from the dataset corresponding to the task Small batches of data are extracted from the dataset and a series of gradient descent steps are performed to optimize the model parameters to minimize the loss, so that it can quickly adapt to the needs of each specific environment and improve the response and effect on specific tasks: , where is the intra-task learning rate, which is used to control the speed and amplitude of parameter updates, It's a task The loss function is It is through the task The specific parameters obtained by training, is a parameter gradient; Step 4: In the task set After completing the task update for all tasks, the learning experience is integrated and the global initial parameters of the model are updated to optimize the average performance on all tasks, so that the model can quickly adjust when encountering new tasks, improving the generalization ability and adaptability of the model on unseen tasks: , where is the meta-learning rate, which is used to control the step size of global parameter updates, is the initial parameter, is the number of tasks, Input to subsequent modules; Step 5: Fusion of the multi-scale attention module optimized by MAML Output features As a subsequent module Input, in the Input After that, continue with the regular end-to-end training and optimize Module parameters : , where yes The learning rate of the module, is the loss function.

[0009] In the above technical solution, in step 5, extracting heart rate information by analyzing the rPPG signal spectrum includes converting the signal from the time domain to the frequency domain using Fourier transform, and then analyzing the specific frequency components in the spectrum to detect the peak value from the specific frequency. , and then calculate the average heart rate value: , where F is the signal sampling frequency and M is the number of Fourier transform points.

[0010] In the above technical solution, in step 5, the one-dimensional spatiotemporal representation stream is subjected to feature extraction and dimensionality reduction, so that the output becomes a smooth one-dimensional sequence to obtain the rPPG signal: , where For the input features Averaging along the third dimension, The one-dimensional convolution operations with kernel sizes of 5 and 3 are performed respectively to generate the final one-dimensional output rPPG signal.

[0011] In the above technical solution, the obtained rPPG signal segment is divided into two segments and , and then construct the frequency consistency loss function of adjacent signals between the two : , where It represents the calculation of the power spectral density of the rPPG signal segment, where M is the number of frequency points.

[0012] In the above technical solution, in step 3, the output multi-level hybrid feature fusion map The denoising diffusion probability model is used for reconstruction. The reconstructed image has a high similarity with the original video frame, and their similarity is calculated using cross entropy loss: , where and They are the original video frame and the feature map reconstructed by the denoising diffusion probability model. The pixel value at position, and Represents the height and width of the image respectively.

[0013] Compared with the existing technology, the present invention has the following advantages: the present invention effectively improves the generalization ability, detail feature capture ability and dependence on labeled data of the model through data enhancement, multi-layer (multi-level) feature fusion and MAML training strategy. Specifically, (1) by using the single-channel k-SVD algorithm for data enhancement, it can not only capture and enhance global and local features, improve data diversity and robustness, but also better retain and enhance feature information related to rPPG signals through sparse representation and dictionary learning; (2) by using the spatial attention module to refine and fuse the low-level center difference convolution features, thereby enhancing the model's ability to capture spatiotemporal information and improving the robustness to subtle changes and noise in the image, so as to more accurately extract rPPG signals; (3) by introducing MAML, taking the multi-scale attention fusion module as the base learning module, by optimizing the initial parameter setting of the module, the model initial parameters that can quickly adapt to these changes in conditions are found, and the network can respond quickly and effectively to new and unseen lighting and motion states, thereby enhancing the applicability and robustness of the network to rPPG signal extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a system structure diagram of non-contact heart rate detection based on hybrid multi-level feature fusion of the present invention; Figure 2 Evaluation of heart rate prediction under different video lengths; Figure 3 Error distribution diagram between predicted value and true value; Figure 4 Scatter plot of heart rate prediction and true value; Figure 5 Test results of heart rate prediction in various video scenes of the VIPL-HR dataset; Figure 6 Results of rPPG signal and multi-layer fusion feature visualization for motion scene estimation in the VIPL-HR dataset; Figure 7 Bland-Altman plot of true and estimated heart rates; Figure 8 Scatter plot of true and estimated heart rates; Figure 9 Visualization of the estimated rPPG signal and multi-layer fusion features in bright scenes of the VIPL-HR dataset; Figure 10 Visualization of the estimated rPPG signal and multi-layer fusion features in dark scenes of the VIPL-HR dataset; Figure 11 The results of facial moles, hair, and glasses occlusion samples for people with different skin colors, the red line is the signal estimated by the method of the present invention, and the blue line is the true signal; Figure 12 Visualize rPPG signals and heart rate estimation for facial expression change samples in the UBFC-RPPG dataset. DETAILED DESCRIPTION

[0015] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0016] This method, based on a three-dimensional spatiotemporal convolutional network, takes a three-channel facial video as input and generates an estimated heart rate value as output, thereby achieving a direct mapping from video to heart rate. The method comprises five steps: data preprocessing and enhancement, multi-scale attention fusion module processing, cascaded multi-layer feature fusion processing, a model-independent meta-learning (MAML) training strategy, and mapping the multi-dimensional fused features into a one-dimensional BVP signal using a signal estimator. Through data enhancement, cascaded multi-layer feature fusion, and MAML training, the method effectively improves the model's generalization ability, ability to capture detailed features, and reliance on labeled data.

[0017] like Figure 1 As shown, the present invention provides a non-contact heart rate detection method based on hybrid multi-level feature fusion, which takes facial video as input and heart rate value as output, and realizes direct mapping from facial video to heart rate, including the following steps: Step 1: Data preprocessing and enhancement: Identify the face area from the face video, roughly segment the video irrelevant information, and then perform face alignment to obtain facial video data. For the processing of facial color video data, only the green channel is selected for spatial feature reconstruction. The single-channel k-SVD algorithm is used for data enhancement to retain feature information related to the rPPG signal. Step 2: Multi-scale attention fusion module processing: The data feature information processed in step 1 is processed by the multi-scale attention fusion module, and the spatial attention mechanism is used to refine and fuse the central differential convolution features of the middle and low layers to obtain fusion features at different levels. , where represents the Hadamard product, represents the original features, and represent average pooling and maximum pooling respectively, represents a 3D convolutional layer, represents the sigmoid activation function, Indicates the number of vectors, low indicates low-level features, mid indicates mid-level features, represents the fine fusion feature, Represents the basic vector features; Step 3: Cascade multi-layer feature fusion processing: The fused features of step 2 are pre-processed by module encoding, and the local information in the image is converted into vector form, so that it can be input into the subsequent transformer module to obtain: = In the formula is the region of interest taken out from the multi-scale fusion module, The convolution kernel size is , the step length is The 3D convolutional layer, Each mapped tile Flatten into a one-dimensional vector to get the final module encoding result ,Will The input is sent to the cascaded multi-layer feature fusion network constructed by connecting 12 transformer blocks in series, which realizes the effective fusion of features at different levels and the comprehensive flow of information, and outputs a multi-level mixed feature fusion map. ; Step 4: Model-agnostic meta-learning (MAML) training strategy: Based on the multi-scale attention fusion module, we introduce MAML training strategy optimization. By optimizing the initial parameter settings of the multi-scale attention fusion module, we find the model initial parameters that can quickly adapt to these changes in conditions to cope with different lighting, motion, and facial changes. Step 5: Use the signal estimator to map the multi-dimensional fusion features into a one-dimensional BVP signal: The multi-level mixed feature fusion map output by the cascaded multi-layer feature fusion network Two consecutive average pooling operations are performed, and then two independent channel-level convolution filters are used on the multi-channel spatiotemporal representation to extract features and reduce the dimensionality of the one-dimensional spatiotemporal representation stream, so that the output becomes a smooth one-dimensional sequence to obtain the rPPG signal. The heart rate information is extracted by analyzing the rPPG signal spectrum.

[0018] This embodiment will introduce the details of each step in detail according to the order of the steps of the above method.

[0019] Step 1: Considering that the light absorption of facial skin is very weak and some backgrounds with strong reflections may affect the prediction results, the face area is first identified from the face video, and after roughly segmenting the irrelevant information in the video, face alignment is performed. , where represents the original video frame, Represents the roughly cropped video frame, after normalizing the face video frame to After the size of Frames are aggregated into a video stream as network input ,in is the number of channels (in this experiment, express three channels), is the number of input video frames, and are the width and height of the video frame, respectively. Since hemoglobin has different absorption characteristics in the red and near-infrared spectral regions, and green light, which lies between red and blue, absorbs relatively little of it, the green channel signal intensity is typically higher. Therefore, when processing facial color video data to extract rPPG information, to improve computational efficiency, only the green channel is selected for spatial feature reconstruction. Data augmentation using the single-channel k-SVD algorithm not only captures and enhances global and local features, improving data diversity and robustness, but also better preserves and enhances feature information related to the rPPG signal through sparse representation and dictionary learning.

[0020] Step 2: The multi-scale attention fusion module uses multiple three-dimensional center difference convolution layers to process the face video sequence, and each layer is followed by batch normalization and ReLU activation function to extract deeper features. By using jump connections and spatial attention modules between consecutive three-dimensional center difference convolution blocks, the network can maintain the key information of the original video input and enhance attention to the spatial area of ​​interest. The multi-scale attention fusion module will correspond to low-level center difference convolution features and medium-level center difference convolution features. By connecting them together, a refined fusion feature can be formed. : , where represents the Hadamard product, represents the original features, and represent average pooling and maximum pooling respectively, represents a 3D convolutional layer, represents the sigmoid activation function, Indicates the number of vectors, low indicates low-level features, mid indicates mid-level features, represents the fine fusion feature, Represents the basis vector features.

[0021] Step 3: Perform module encoding preprocessing operations, divide the output feature map after the signal feature extraction module into a series of non-overlapping small blocks according to a fixed window size, aggregate the spatiotemporal information nearby, and convert the local information in the image into a vector form, so that it can be input into the subsequent transformer module, so that the transformer can perform self-attention calculations on each local area of ​​the image, thereby better capturing the global and local relationships of the image: = In the formula is the region of interest taken out from the multi-scale fusion module, The convolution kernel size is , the step length is The 3D convolutional layer, Each mapped tile Flatten into a one-dimensional vector to get the final module encoding result ,Will The input is fed into a cascaded multi-layer feature fusion network consisting of 12 transformer blocks in series, which gradually learns more complex and abstract feature representations from shallow to deep layers, performs deeper feature extraction, and realizes effective fusion of features at different levels and comprehensive flow of information. , where is the output of the transformer block in the previous layer, To perform residual block operation on the current transformer layer, is the output of the transformer block in the current layer. Then, two consecutive upsampling block operations are performed, where each upsampling block consists of four layers. The first layer is the upsampling operation, and the second layer is the convolution kernel size. 3D convolution, which is followed by a normalization and ELU activation operation; the final output is a multi-level mixed feature fusion map In step 3, the output multi-level hybrid feature fusion map The denoising diffusion probability DDPM model is used for reconstruction. The reconstructed image has a high similarity with the original video frame, and their similarity is calculated using cross entropy loss: , where and They are the original video frame and the feature map reconstructed by DDPM. The pixel value at position, and Represents the height and width of the image respectively.

[0022] In step 4, the specific steps of introducing MAML training strategy optimization are as follows: Step 1, set the task set ,in Indicates the number of task sets, each task Corresponding to different scenarios and having corresponding data sets , which contains the video frames reconstructed with enhanced spatial features and the corresponding true rPPG signal As target output; Step 2: Fusion of multi-scale attention modules The initial parameters are expressed as , the cascaded multi-layer feature fusion module is represented as , the initial parameters are expressed as ; Step 3: For each task , from the dataset corresponding to the task Small batches of data are extracted from the dataset and a series of gradient descent steps are performed to optimize the model parameters to minimize the loss, so that it can quickly adapt to the needs of each specific environment and improve the response and effect on specific tasks: , where is the intra-task learning rate, which is used to control the speed and amplitude of parameter updates, It's a task The loss function is It is through the task The specific parameters obtained by training, is a parameter gradient; Step 4: In the task set After completing the task update for all tasks, the learning experience is integrated and the global initial parameters of the model are updated to optimize the average performance on all tasks, so that the model can quickly adjust when encountering new tasks, improving the generalization ability and adaptability of the model on unseen tasks: , where is the meta-learning rate, which is used to control the step size of global parameter updates, is the initial parameter, is the number of tasks, Input to subsequent modules; Step 5: Fusion of the multi-scale attention module optimized by MAML Output features As a subsequent module Input, in the Input After that, continue with the regular end-to-end training and optimize Module parameters : , where yes The learning rate of the module, is the loss function.

[0023] In step 5, in order to reduce the dimension of the feature map, extract higher-level semantic information, and reduce the amount of calculation, the features output by the cascaded multi-layer feature fusion network are Perform two consecutive average pooling operations, and then use two independent channel-level convolution filters on the multi-channel spatiotemporal representation to extract features and reduce the dimension of the one-dimensional spatiotemporal representation stream, so that the output becomes a smooth one-dimensional sequence and the rPPG signal is obtained: , where For the input features Averaging along the third dimension, The one-dimensional convolution operations with kernel sizes of 5 and 3 are performed respectively to generate the final one-dimensional output rPPG signal.

[0024] Then the obtained rPPG signal segment is divided into two segments and , and then construct the frequency consistency loss function of adjacent signals between the two : , where It represents the calculation of the power spectral density of the rPPG signal segment, where M is the number of frequency points. The rPPG signal contains multiple physiological signals, such as heart rate and respiratory rate, which are expressed in different frequency bands.

[0025] In the frequency domain, heart rate usually shows obvious peaks, so heart rate information can be extracted by analyzing the rPPG spectrum. First, the signal is converted from the time domain to the frequency domain using Fourier transform, and then the specific frequency components in the spectrum are analyzed to detect the peaks. , and then calculate the average heart rate value: , where F is the signal sampling frequency and M is the number of Fourier transform points.

[0026] The present invention sets up an experiment to verify the description of the above invention content, and selects videos with lengths of 80 frames, 160 frames, 240 frames, 320 frames and 480 frames for the experiment. In order to intuitively reflect the impact of facial video length on heart rate prediction, Figure 2 The prediction error of heart rate for different input facial video lengths in the VIPL-HR dataset is plotted, and the video length predicted by the best model is determined. The accuracy of heart rate estimation is compared with the state-of-the-art methods, as shown in Table 1 below.

[0027] Table 1 shows the comparison of heart rate measurement results on the VIPL-HR dataset.

[0028] In order to more intuitively show the error distribution of heart rate estimation per minute in the test set, the present invention examines and visualizes the error distribution of heart rate estimation of the proposed network model. The scatter plot of predicted HR and true HR values ​​is shown in Figure 2. Figure 3 As shown, Figure 4 The correlation between the two sets of data, namely the true heart rate value and the predicted heart rate value, is shown. The results show a strong positive correlation. The predicted heart rate value is very close to the actual value, and the overall prediction performance of the model is good.

[0029] In order to verify the robustness of the proposed method in the face of different environmental changes, the nine scenarios mentioned above were tested, and the mean absolute error of the heart rate measurement results in some typical experimental scenarios was analyzed. Figure 5 From the test results of predicting heart rate in various scenes of the VIPL dataset, it can be observed that the average error values ​​are 2.57 bpm and 3.16 bpm when exercising and the phone is moving, respectively. This shows that the proposed method performs well in processing scenes with facial changes caused by head rotation and speaking, and can also better adapt to videos recorded by mobile phone cameras for heart rate prediction.

[0030] In cross-dataset validation, in order to further verify the generalization ability and robustness of the proposed method, the network was trained using the VIPL-HR dataset and then tested and evaluated on the UBFC-RPPG dataset. The test results are shown in Table 2. From the data in the table, it can be seen that the average error of the model trained on the VIPL-HR dataset when tested on the UBFC-RPPG dataset is 0.53 bpm and the RMSE is 1.49 bpm, indicating that the performance of the proposed method is significantly better than other methods.

[0031] Table 2 shows the comparison of existing methods on the UBFC-RPPG dataset.

[0032] After conducting a series of ablation experiments, the present invention also performed a visual display to verify the robustness of the present invention to motion artifacts and uneven illumination. Among them, the BVP signal reconstruction results and multi-level fusion feature maps on the VIPL-HR dataset are as follows: Figure 6As shown. The results show that in motion scenes: 1) the true value (blue line) and the predicted BVP signal (red line) are significantly positively correlated; 2) the heart rates calculated from the predicted and true signals in the first group of videos are 57.6 bpm and 57.0 bpm, respectively, and the heart rates calculated from the predicted and true signals in the second group of videos are 54.4 bpm and 54.0 bpm, respectively, proving that the method of the present invention can accurately predict BVP signals and HR; 3) the single-channel regularized k-SVD data enhancement method can effectively separate periodic heart rate signals from random noise or artifacts caused by head movement; 4) the proposed cascade transformer model for central difference convolution feature fusion can continue to focus on the key areas of the face even when the head rotates, capturing local detail changes while paying attention to macroscopic spatiotemporal relationships, and distinguishing effective signals related to heart rate from motion interference.

[0033] At the same time, in order to verify the correlation between the heart rate calculated by the predicted BVP and the true value, a scatter plot and a Bland-Altman plot were drawn, as shown in Figure 7 and Figure 8 The results show that most of the prediction results in motion scenes fall within the consistency limit, and the average difference is close to zero. The present invention can effectively deal with the problems caused by motion artifacts on rPPG measurement.

[0034] Some results on the VIPL-HR dataset in bright and dark light scenes are as follows Figure 9 and Figure 10 As shown, the experimental results show that: 1) the true value (blue line) and the predicted BVP signal (red line) are significantly positively correlated; 2) in bright scenes, the heart rates calculated from the predicted and true signals are 79.2 bpm and 80.0 bpm respectively; in dim scenes, the heart rates calculated from the predicted and true signals are both 80.0 bpm, proving that the present invention can accurately predict BVP signals and HR. 3) Single-channel regularized k-SVD data enhancement can distinguish the heart rate signal contained in pixel value fluctuations under changing illumination from the noise caused by uneven illumination, and suppress the noise; 4) The proposed cascade transformer model ensures that features from local to global can be fully captured and utilized, and the fusion of the central difference convolution feature model can ensure that these features will not be distorted due to the interference of changes in external lighting conditions.

[0035] Figure 11 This figure shows samples from the VIPL dataset with facial moles, hair, and glasses occluded. The blue line represents the true signal, and the red line represents the predicted BVP signal. The heart rate is then calculated based on the maximum value of the BVP signal. Experimental results demonstrate that the proposed method can adapt to facial features and occlusions, achieving accurate measurement results.

[0036] Figure 12 The results of training on the VIPL-HR dataset and testing on the UBFC-RPPG dataset are shown. The heart rate calculated from the estimated maximum power spectral density of the rPPG signal is 66.0 bpm, and the actual test value is 65.0 bpm. The experimental results show that the method of the present invention can still obtain accurate measurement results when the head moves and the facial expression changes.

[0037] Experimental results show that the detection method of the present invention outperforms existing methods (such as spatiotemporal attention networks) in BVP signal measurement, with the mean absolute error and root mean square error reduced by 5.4% and 20.9%, respectively.

Claims

1. A non-contact heart rate detection method based on hybrid multi-level feature fusion, which takes facial video as input and heart rate value as output, and realizes direct mapping from facial video to heart rate, characterized by: The steps include: Step 1: Data preprocessing and enhancement: Identify the face area from the face video, roughly segment the video irrelevant information, and then perform face alignment to obtain facial video data. For the processing of facial color video data, only the green channel is selected for spatial feature reconstruction. The single-channel k-SVD algorithm is used for data enhancement to retain feature information related to the rPPG signal. Step 2: Multi-scale attention fusion module processing: The data feature information processed in step 1 is processed by the multi-scale attention fusion module, and the spatial attention mechanism is used to refine and fuse the central differential convolution features of the middle and low layers to obtain fusion features at different levels. , where represents the Hadamard product, represents the original features, and represent average pooling and maximum pooling respectively, represents a 3D convolutional layer, represents the sigmoid activation function, Indicates the number of vectors, low indicates low-level features, mid indicates mid-level features, represents the fine fusion feature, Represents the basic vector features; Step 3: Cascade multi-layer feature fusion processing: The fused features of step 2 are pre-processed by module encoding, and the local information in the image is converted into vector form, so that it can be input into the subsequent transformer module to obtain: = In the formula is the region of interest taken out from the multi-scale fusion module, The convolution kernel size is , the step length is The 3D convolutional layer, Each mapped tile Flatten into a one-dimensional vector to get the final module encoding result ,Will The input is sent to the cascaded multi-layer feature fusion network constructed by connecting 12 transformer blocks in series, which realizes the effective fusion of features at different levels and the comprehensive flow of information, and outputs a multi-level mixed feature fusion map. ; Step 4: Model-agnostic meta-learning (MAML) training strategy: Based on the multi-scale attention fusion module, we introduce MAML training strategy optimization. By optimizing the initial parameter settings of the multi-scale attention fusion module, we find the model initial parameters that can quickly adapt to these changes in conditions to cope with different lighting, motion, and facial changes. Step 5: Use the signal estimator to map the multi-dimensional fusion features into a one-dimensional BVP signal: The multi-level mixed feature fusion map output by the cascaded multi-layer feature fusion network Two consecutive average pooling operations are performed, and then two independent channel-level convolution filters are used on the multi-channel spatiotemporal representation to extract features and reduce the dimensionality of the one-dimensional spatiotemporal representation stream, so that the output becomes a smooth one-dimensional sequence to obtain the rPPG signal. The heart rate information is extracted by analyzing the rPPG signal spectrum.

2. The non-contact heart rate detection method based on hybrid multi-level feature fusion according to claim 1, characterized in that: The second step: the multi-scale attention fusion module uses multiple three-dimensional center difference convolution layers to process the face video sequence, and uses batch normalization and ReLU activation function after each layer to extract deeper features. By using jump connections and spatial attention modules between consecutive three-dimensional center difference convolution blocks, the network can maintain the key information of the original video input and enhance the information of the spatial region of interest.

3. The non-contact heart rate detection method based on hybrid multi-level feature fusion according to claim 1, characterized in that: In the step three, The input is then fed into a cascaded multi-layer feature fusion network consisting of 12 transformer blocks connected in series. The specific process is as follows: , where is the output of the transformer block in the previous layer, To perform residual block operation on the current transformer layer, is the output of the transformer block in the current layer. Then, two consecutive upsampling block operations are performed, where each upsampling block consists of four layers. The first layer is the upsampling operation, and the second layer is the convolution kernel size. The 3D convolution is followed by a normalization and ELU activation operation.

4. The non-contact heart rate detection method based on hybrid multi-level feature fusion according to claim 1, characterized in that: In step 4, the specific steps of introducing the MAML training strategy optimization are as follows: Step 1: Set up the task set ,in Indicates the number of task sets, each task Corresponding to different scenarios and having corresponding data sets , which contains the video frames reconstructed with enhanced spatial features and the corresponding true rPPG signal As target output; Step 2: Fusion of multi-scale attention modules The initial parameters are expressed as , the cascaded multi-layer feature fusion module is represented as , the initial parameters are expressed as ; Step 3: For each task , from the dataset corresponding to the task Small batches of data are extracted from the dataset and a series of gradient descent steps are performed to optimize the model parameters to minimize the loss, so that it can quickly adapt to the needs of each specific environment and improve the response and effect on specific tasks: , where is the intra-task learning rate, which is used to control the speed and amplitude of parameter updates, It's a task The loss function is It is through the task The specific parameters obtained by training, is a parameter gradient; Step 4: In the task set After completing the task update for all tasks, the learning experience is integrated and the global initial parameters of the model are updated to optimize the average performance on all tasks, so that the model can quickly adjust when encountering new tasks, improving the generalization ability and adaptability of the model on unseen tasks: , where is the meta-learning rate, which is used to control the step size of global parameter updates, is the initial parameter, is the number of tasks, Input to subsequent modules; Step 5: Fusion of the multi-scale attention module optimized by MAML Output features As a subsequent module Input, in the Input After that, continue with the regular end-to-end training and optimize Module parameters : , where yes The learning rate of the module, is the loss function.

5. The non-contact heart rate detection method based on hybrid multi-level feature fusion according to claim 1, characterized in that: In step 5, extracting heart rate information by analyzing the rPPG signal spectrum includes converting the signal from the time domain to the frequency domain using Fourier transform, and then analyzing the specific frequency components in the spectrum to detect the peak value from the specific frequency. , and then calculate the average heart rate value: , where F is the signal sampling frequency and M is the number of Fourier transform points.

6. The non-contact heart rate detection method based on hybrid multi-level feature fusion according to claim 1, characterized in that: In step 5, the one-dimensional spatiotemporal representation stream is subjected to feature extraction and dimensionality reduction, so that the output becomes a smooth one-dimensional sequence to obtain the rPPG signal: , where For the input features Averaging along the third dimension, The one-dimensional convolution operations with kernel sizes of 5 and 3 are performed respectively to generate the final one-dimensional output rPPG signal.

7. The non-contact heart rate detection method based on hybrid multi-level feature fusion according to claim 6, characterized in that: The obtained rPPG signal segment is divided into two segments and , and then construct the frequency consistency loss function of adjacent signals between the two : , In the formula It represents the calculation of the power spectral density of the rPPG signal segment, where M is the number of frequency points.

8. The non-contact heart rate detection method based on hybrid multi-level feature fusion according to claim 1, characterized in that: In step 3, the output multi-level hybrid feature fusion map The denoising diffusion probability model is used for reconstruction. The reconstructed image has a high similarity with the original video frame, and their similarity is calculated using cross entropy loss: , where and They are the original video frame and the feature map reconstructed by the denoising diffusion probability model. The pixel value at position, and Represents the height and width of the image respectively.

Images