A non-contact heart rate detection device and method based on face video
By synthesizing video VrPPG for simulating real face videos and using deep learning models with encoded decoder structure, the problem of insufficient central rate detection accuracy and generalization performance in the prior art is solved, and a pulse wave signal with high signal-to-noise ratio is achieved Testing significantly improves the accuracy of heart rate detection.
Patent Information
- Application Number
- CN202210332556.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-31
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-03-31
AI Technical Summary
The existing contactless heart rate detection method based on deep learning has shortcomings in detection accuracy and generalization performance, especially due to insufficient data and noise introduction, which leads to large errors in some scenarios.
By obtaining face videos without skin color changes, a video VrPPG is synthesized to simulate real face videos, and a deep learning model with encoded codec structure is used for training, adding data synthesis links to improve the generalization performance of the model, and through Signal amplification improves the signal-to-noise ratio of the pulse wave signal.
It significantly improves the accuracy and generalization performance of heart rate detection, reduces estimation errors, and improves the accuracy of detection results, especially in disturbing environments.
Smart Images

Figure CN114722869B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of heart rate detection, and in particular to a non-contact heart rate detection device and method based on face video. Background Art
[0002] Heart rate is a very important physiological parameter of the human body, which can reflect the health level of the human body in a timely manner. The most direct way to obtain heart rate is mainly through contact sensors, such as electrocardiographs and pulse wave detectors. These methods can accurately monitor human heart rate and pulse information, but they require external hardware devices and direct contact with human skin to obtain the measured information. These devices are expensive and not easy to carry, and the form of contact detection often brings a great burden to the person being tested. Therefore, the detection cost is very high, which is not conducive to the user's real-time detection of their own physiological information.
[0003] Traditional non-contact heart rate detection methods are mainly based on remote estimation of heart rate by establishing mathematical models. Based on the principle of rPPG (remote photoplethysmography), that is, changes in the hemoglobin content in the blood during the human cardiac cycle will cause slight changes in skin color, and the human heart rate is analyzed by this feature. In the early video-based non-contact heart rate estimation methods, signal separation, color channel analysis and skin reflection models were commonly used. Such as blind source separation method ICA based on principal component analysis, chromaticity-based method CHROM and mathematical model methods such as building a light reflection model POS. These methods have good generalization performance, but in actual situations, they have a relatively low signal-to-noise ratio, and the accuracy of the predicted heart rate is greatly limited.
[0004] With the development of deep learning in recent years, some deep learning models have been used to more accurately detect heart rate from face videos. Convolutional neural networks have shown excellent performance in processing image and video data, especially in detecting subtle features such as changes in facial skin color. There are many existing deep learning-based methods that require careful design of manual feature representations of face videos to ensure successful model training. For example, before model training, multiple frames of images need to be processed into one frame, but this will introduce additional noise and the final detection accuracy is also very limited. At the same time, the data set containing physiological information and labels is very limited, which further limits the generalization performance of deep learning models, causing this method to obtain large errors in some scenarios. Summary of the invention
[0005] In view of the technical problems existing in the prior art, one of the objectives of the present invention is to provide a non-contact heart rate detection device based on face video, which can improve the detection accuracy and generalization performance.
[0006] In view of the technical problems existing in the prior art, the second purpose of the present invention is to provide a non-contact heart rate detection method based on face video.
[0007] In order to achieve the above object, the present invention adopts the following technical solution:
[0008] A non-contact heart rate detection method based on face video, comprising the following steps:
[0009] Get a face video V without skin color change, and synthesize multiple videos V for simulating real face videos based on the face video V rPPG ;
[0010] Send multiple videos V rPPG The deep learning model is trained as a training sample so that the deep learning model can amplify the signal of the face image to be detected for heart rate, obtain data with a high signal-to-noise ratio from the amplified face image, and then calculate the heart rate.
[0011] Furthermore, the method for obtaining the face video V without skin color change is to select a basic image c and a driving video V d , through the first-order motion model, the basic image c and the driving video V d Fit it into a face video V without skin color changes.
[0012] Further, after obtaining the face video V, the following steps are also included: the face video V is processed using the function Generate a pulse wave signal and Add video V to the composition rPPG In, it is used for training deep learning models.
[0013] ;
[0014] in, represents the scale of the signal, represents the signal frequency; t represents the length of the signal; and is a constant that determines the signal waveform;
[0015] ;
[0016] in, represents the increased signal strength in the j channel, represents the pixel value of the image in the face video V generated by the first-order motion model, Indicates the pixel value of the image after adding the skin color change signal.
[0017] Further, the synthesized video VrPPG The implementation method includes constructing a BVP curve to simulate the skin color change of the face, sampling the constructed BVP curve, the sampling frequency is consistent with the frame rate of the face video V, the sampling value is used as the skin color signal strength and added to the face video V after dimensional expansion, and a video V for simulating a real face video is generated rPPG .
[0018] Further, the synthesized video V rPPG When the video V rPPG For the corresponding label, use V rPPG When training the deep learning model, the same video V is used as a training sample. rPPG The corresponding labels are used to calculate the loss;
[0019] The loss function used is, ;
[0020] in, represents the total loss, represents the regularization loss, represents the structural similarity loss.
[0021] Furthermore, the training method of the deep learning model includes the following steps: first, a database containing multiple face videos V is constructed, and a face video V is randomly selected from the database during each training iteration; then, skin color changes are added to the selected face video V, and the time phase of the function therein is randomly adjusted, and the code of the data synthesis step is integrated into the data loading process of network training, so that the data synthesis and network training processes are carried out synchronously.
[0022] Furthermore, during the deep learning model training process, the parameters of the data synthesis algorithm are adjusted to increase the types of synthesized data.
[0023] Furthermore, the implementation method of calculating the heart rate includes: capturing video of the facial area where the heart rate is to be detected, amplifying the signal of each frame in the video through a trained deep learning model, calculating the mean of the amplified facial image, performing peak detection on the calculated curve, analyzing the time interval between the peaks and using this to calculate the average heart rate during that period of time.
[0024] A non-contact heart rate detection device based on face video, comprising:
[0025] The video processing module is used to synthesize multiple videos V for simulating real face videos based on the face video V without skin color change through the RVG data synthesis algorithm rPPG ;
[0026] Deep learning model for using multiple videos V rPPGIt is used as a training sample for training, and the signal of the face image to be detected for heart rate is amplified;
[0027] The heart rate calculation module is used to obtain high signal-to-noise ratio data from the enlarged face image and then calculate the heart rate.
[0028] Furthermore, the deep learning model adopts a codec structure, including a bridge block and an encoder and a decoder respectively connected to the bridge block, the encoder is used to receive the video V rPPG , the decoder is used to receive and video V rPPG The generated tags are synchronized, and the bridge area is equipped with an amplification factor to adjust the signal amplification factor. .
[0029] In general, the present invention has the following advantages:
[0030] The added data synthesis link can ensure that the deep learning model has sufficient data to train the target features, so as to improve the generalization performance of the deep learning model provided by the present invention. Through signal amplification, the signal-to-noise ratio of the detected pulse wave signal can be greatly improved, and the detection accuracy is higher. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 Schematic diagram of the steps of the RVG data synthesis algorithm of the present invention.
[0032] Figure 2 Schematic diagram of quality assessment of video generated by RVG data synthesis algorithm of the present invention Figure 1 .
[0033] Figure 3 Schematic diagram of quality assessment of video generated by RVG data synthesis algorithm of the present invention Figure 2 .
[0034] Figure 4 It is a structural schematic diagram of the ED-RPPG model of the present invention.
[0035] Figure 5 This is a flow chart of the algorithm for RVG data synthesis of the present invention.
[0036] Figure 6 The signal-to-noise ratio calculated by the present invention varies with the amplification factor Schematic diagram of the changes in .
[0037] Figure 7 The present invention has different magnification factors. The following is a schematic diagram of the calculated pulse wave.
[0038] Figure 8 It is a schematic diagram of the heart rate detection effect under interference conditions of the present invention. DETAILED DESCRIPTION
[0039] The present invention will be described in further detail below.
[0040] A non-contact heart rate detection device based on face video, including a video processing module, which is used to extract the face area from the video, reduce the noise pollution caused by the background area, and the detected face image is used for analyzing the heart rate pulse wave of the human body;
[0041] The signal amplification module includes a trained deep learning model, which is used to amplify the subtle skin color change signal of the face, thereby increasing the ratio between the target signal and the noise, thereby enabling more accurate heart rate detection;
[0042] The heart rate calculation module is used to calculate the pixel mean from the signal amplified data and estimate the heart rate of the subject, reducing the estimation error by using techniques such as peak detection and Fourier transform.
[0043] Face image acquisition is achieved by locating the facial feature points and segmenting the face and background areas contained in the image based on the feature points.
[0044] The present invention provides a codec model for amplifying target features and a data synthesis algorithm capable of enhancing the robustness of the model, including:
[0045] Step 1: Based on the sample images, use the RVG data synthesis algorithm to synthesize the training samples and corresponding labels required by the model.
[0046] Step 2: Use synthetic data samples to train the proposed encoder-decoder ED-RPPG model, and adjust the parameters of the RVG algorithm during the training process to increase the types of synthetic data.
[0047] Step 3: Use the trained ED-RPPG model to amplify the target signal. High signal-to-noise ratio data can be obtained from the amplified image, from which the pulse wave and heart rate can be calculated.
[0048] Preferably, the step 1 includes, through a basic picture and drive video To generate a video without skin color changes This process is implemented using a first-order motion model, and the generated video removes the background area through face skin detection. Generate a pulse wave signal,
[0049] ;
[0050] in, The signal scale is limited, we set it to [0,1]; represents the signal frequency; t represents the length of the signal; and It is a constant that determines the signal waveform.
[0051] Preferably, step one further comprises constructing the pulse wave signal Add to generated video In, used for model training,
[0052] ;
[0053] Among them, represents the increased signal strength in the j channel, represents the pixel value of the image generated by the first-order motion model, Indicates the pixel value after adding the skin color change signal. U indicates that for each selected pixel Add a skin tone value .
[0054] It is worth noting that we set the intensity in the green channel The intensity is higher than that set in the R and B channels. High. Because according to research, hemoglobin has a stronger absorption capacity for green light. In this way, we synthesized synthetic data V with skin color change characteristics. rPPG By calculating the peak signal-to-noise ratio (PNSR) and structural similarity (SSIM) values between the synthetic image and the original image, our synthetic data has good consistency with the original data, which also provides a good foundation for subsequent data training.
[0055] Preferably, step 2 includes constructing and training the encoder-decoder model ED-RPPG. Based on the requirements of reducing the noise introduced by data preprocessing and improving the signal-to-noise ratio, the model construction goal is to directly amplify the signal of the skin color change of the face and output the amplified skin color change signal containing the original face image.
[0056] The coding layer of the network consists of 2 3D convolutional layers and 3 3D residual blocks. The input of the coding block is a 4D tensor containing 64 frames of images, with a size of 64×64×64×3. The size of each frame is 64×64, the number of channels is 3, and the sampling frequency is 15Hz. The coding block is designed to extract the skin color change information contained in the face video and filter out various noises contained in the image. The coding block finally outputs a 64×1×1×3 tensor and inputs the tensor into the bridge block.
[0057] The bridge block contains two 3D residual modules and the external magnification factor The bridge block is designed to set the network's amplification factor for the target signal by adjusting To achieve this goal, change the size of The value of can obtain target signals with different signal-to-noise ratios.
[0058] The input of the decoding block is the tensor amplified by the bridge block. The goal of this module is to restore the signal extracted from the encoder. The decoding block consists of three 3D residual blocks and four 3D convolutional layers. The network input is a 64×1×1×3 tensor and the output is a 64×64×64×3 tensor. Since the various noises and non-target signal features in the input video have been filtered out in the encoding block, the output of this module is only the change features of the restored target signal.
[0059] The essence of the encoder and decoder is a high-order differentiation and integration process. Because in the actual process, noise often manifests as a signal that presents low-order changes in time, while the target signal (pulse wave) is a high-order signal. When the noise and the target signal are mixed together, we can filter out the low-order noise by taking multiple partial derivatives of the mixed signal with respect to time. Then, by restoring the target signal through the integral term, we can obtain a clean signal of skin color changes. This process can be expressed by the following formula:
[0060] Original image definition: ,in, Represents the pixel value at position x at time t; Indicates the original color of the image; Indicates skin color changes caused by changes in blood volume; Indicates skin color changes caused by various external noises.
[0061] Take multiple partial derivatives of the target signal: , among which, due to It is a constant in the time dimension, so it is eliminated in the process of partial derivatives, and the same is true for low-order noise.
[0062] The target signal is filtered out To restore: ,in represents the recovered target signal, and we hope that it can be as close as possible to ; After the above processing, we can express the video after signal amplification as: .
[0063] Preferably, step three includes calculating the pixel mean of the recovered target signal and using peak and trough detection to obtain the average heart rate over a period of time. This method can greatly improve the signal-to-noise ratio of the detected pulse wave signal, thereby improving the accuracy of the heart rate detected from the video. At the same time, it also solves the problem of insufficient data faced by deep learning methods, thereby improving the generalization performance.
[0064] Specifically, Figure 1 As shown, the present invention provides an algorithm RVG for artificially synthesizing facial skin color change features, which can synthesize an unlimited number of facial videos V containing heart rate information and corresponding labels. This greatly facilitates the training of deep learning models, and also greatly improves the training effect and detection accuracy of deep learning models, including the following steps:
[0065] (1) First, generate a face video V without skin color changes.
[0066] Select a basic image c and a driving video V from the training sample d , using the first-order motion model of image animation proposed by Aliaksandr Siarohin et al., the basic image c is fitted into a segment that is consistent with the driving video V d A similar face video V, which has no skin color change. Then, the face video V is detected using the dlib module of OpenCV to extract the face in the video and the face skin area, so that the data obtained can reduce the interference of noise and improve the accuracy.
[0067] (2) The BVP signal is generated and added to the face video V.
[0068] Next, construct the BVP curve To simulate the skin color change of the face. Sample the constructed curve, and the sampling frequency is consistent with the frame rate of the face video V. The sampled value is used as the skin color signal strength and added to the face video V after dimensional expansion to generate the video V rPPG To simulate real face videos. We use peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) to evaluate the generated video V rPPG quality.
[0069] (3) Video V rPPG Assess the quality of
[0070] In the RVG algorithm we proposed, the impact video V rPPG The parameters of the effect are mainly parameters and U, so we calculate The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) of the generated video under the value of U are used to evaluate the synthetic video V rPPGThe mass of Figure 2 , Figure 3 As shown. Figure 2 , Figure 3 It can be seen from the curve that the parameters Less than 1.2, parameters When the ratio is between 1% and 2%, the synthesized video V rPPG The quality is best.
[0071] Figure 5 The RVG algorithm flow chart is shown in Figure 1. The input of the algorithm includes a basic face image c and a driving video. The algorithm first initializes the signal frequency , and then use the first-order motion model to generate a face video V without skin color change signal, such as Figure 5 Steps 1 and 2 in the above example. Next, the BVP signal is simulated using the Fourier series and normalized, as shown in Figure 5 Steps 3 and 4 in the video. Next, add the corresponding skin color change signal to each color channel in each frame of the video, such as Figure 5 Steps 5 to 11 in the above. Finally, the algorithm can output a video V that is similar to the real face video. rPPG .
[0072] like Figure 4 As shown, the signal amplification module adopts a codec structure, Encoder represents an encoder, Decoder represents a decoder, Bridge represents a bridge block, and input and output represent network input and output respectively. The present invention provides a deep learning model of a codec structure for amplifying skin color change signals in a human face, and is trained in the following manner:
[0073] (1) Obtaining original training samples from the training set and feeding them into the data synthesis algorithm, the data synthesis algorithm is used to construct training data containing skin color changes and corresponding labels for the training samples.
[0074] (2) The synthesized training data is fed into the constructed encoder-decoder, and the synthesized labels are used as the labels corresponding to the training data to calculate the loss.
[0075] (3) The data synthesis and training processes are performed synchronously, and by randomly adjusting the parameters in the data synthesis process, different data can be generated for model training, thereby improving the robustness of the encoder-decoder.
[0076] (4) During the training process, the corresponding magnification factors of the encoder and decoder are set according to the parameters used in data synthesis to ensure that the model achieves optimal performance.
[0077] The specific steps include:
[0078] (1) Joint training of the ED-RPPG network model and the RVG data synthesis algorithm.
[0079] Video V synthesized by RVG algorithm rPPG By cropping the size and setting the number of frames, the video can be sent to the ED-RPPG model for training. We synchronize the data synthesis with the network training process by using random numbers. First, we need to build a database of face videos V without skin color changes, and randomly select face videos V from it during each training iteration. Since the generated training samples do not contain any physiological information about skin color changes, we use functions to artificially synthesize skin color change information and add it to the generated training samples. The synthesized skin color change information also serves as a label to guide the training and learning process of the encoder and decoder. And randomly adjust the time phase of the function to enrich the number of training samples. So that enough different types of data can be used for training during the training process. We integrate the code of the data synthesis step into the data loading process of network training, so that data synthesis and network training can be carried out simultaneously, greatly improving the training efficiency and the effect of the final model training of the network.
[0080] (2) Use the ED-RPPG network model for heart rate detection.
[0081] The trained ED-RPPG can be used to recognize the heart rate of face videos. First, use the dlib module of OpenCV to collect the face area, and the collection frequency is set to 15 frames per second. The collected face image is sent to the ED-RPPG model for signal amplification. The model can process 64 frames of images at a time. The model will output a face video with an amplified BVP signal. At this time, the target signal in the video has a high signal-to-noise ratio, so the calculation of the heart rate becomes very convenient and accurate.
[0082] The ED-RPPG encoder-decoder model can amplify subtle skin color changes on the face to a level visible to the naked eye, and does not require manual feature design for the face. It mainly uses the idea of differentiation and integration to filter out low-order noise in the video and retain high-order signals such as heart rate, thereby greatly improving the signal-to-noise ratio of the target signal.
[0083] After zooming in on the video, we calculate the mean of each frame and can clearly see the curve of skin color change. The effect after zooming in is as follows: Figure 7 As shown. We perform peak detection on the calculated curve, analyze the time interval between the peaks and use it to calculate the average heart rate during that period. We use the data in the PURE dataset to evaluate our proposed method. The following table shows the calculation results.
[0084]
[0085] Table 1
[0086] Table 1 lists the effect of ED-RPPG proposed in the present invention on improving the signal-to-noise ratio of the calculated BVP curve. ICA, POS and CHROM represent several traditional non-contact heart rate calculation methods. We calculate the signal-to-noise ratio of the original video and the video after signal amplification using these methods. It can be seen that the signal-to-noise ratio calculated by the video after signal amplification has been significantly enhanced. Figure 7 It can also be clearly seen in. Figure 7 The signal-to-noise ratio of the BVP signal calculated from the videos at different magnification factors was also calculated.
[0087]
[0088] Table 2
[0089] Table 2 lists the Pearson correlation coefficients of the average heart rate calculated from the PURE dataset. , mean absolute error MAE, root mean square error RMSE and standard deviation Std. We calculated the error of the average heart rate within 30 seconds before and after signal amplification. It can be seen that the accuracy of the average heart rate calculated by the video after signal amplification has been significantly improved. In order to further verify the effect of our proposed method, we also used the method proposed in this invention to improve the effect of heart rate estimation under the conditions of lighting changes, speaking and exercise. Figure 8 As shown in Figure 2, it can be seen that our method can significantly improve the accuracy of heart rate estimation under these interference environments.
[0090] The skin color mentioned in this application is only used for heart rate detection and does not involve any emotional concerns.
[0091] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.
Claims
1. A non-contact heart rate detection method based on face video, characterized in that: The following steps are included: Get a face video V without skin color change, and synthesize multiple videos V for simulating real face videos based on the face video V rPPG ; Send multiple videos V rPPG The deep learning model is trained as a training sample, so that the deep learning model amplifies the signal of the face image to be detected for heart rate, obtains data with a high signal-to-noise ratio from the amplified face image, and then calculates the heart rate; The method of obtaining a face video V without skin color change is to select a basic image c and a driving video V d , through the first-order motion model, the basic image c and the driving video V d Fitting into a face video V without skin color change; After obtaining the face video V, the following steps are also included: using the function Generate a pulse wave signal and Add video V to the composition rPPG In, it is used for training deep learning models. ; in, represents the scale of the signal, represents the signal frequency; t represents the length of the signal; and is a constant that determines the signal waveform; ; in, represents the increased signal strength in the j channel, represents the pixel value of the image in the face video V generated by the first-order motion model, Indicates the pixel value of the image after adding the skin color change signal; The deep learning model adopts a codec structure, including a bridge block and an encoder and a decoder respectively connected to the bridge block, the encoder is used to receive the video V rPPG , the decoder is used to receive and video V rPPG The generated tags are synchronized, and the bridge area is equipped with an amplification factor to adjust the signal amplification factor. .
2. A non-contact heart rate detection method based on face video according to claim 1, characterized in that: According to the face video V, multiple videos V for simulating real face videos are synthesized rPPG The implementation method includes constructing a BVP curve to simulate the skin color change of the face, sampling the constructed BVP curve, the sampling frequency is consistent with the frame rate of the face video V, the sampling value is used as the skin color signal strength and added to the face video V after dimensional expansion, and a video V for simulating a real face video is generated rPPG .
3. A non-contact heart rate detection method based on face video according to claim 1, characterized in that: Synthetic Video V rPPG When the video V rPPG For the corresponding label, use V rPPG When training the deep learning model, the same video V is used as the training sample. rPPG The corresponding labels are used to calculate the loss; The loss function used is, ; in, represents the total loss, represents the regularization loss, represents the structural similarity loss.
4. A non-contact heart rate detection method based on face video according to claim 1, characterized in that: The training method of the deep learning model includes the following steps: first, a database containing multiple face videos V is constructed, and a face video V is randomly selected from the database during each training iteration; then, skin color changes are added to the selected face video V, and the time phase of the function therein is randomly adjusted, and the code of the data synthesis step is integrated into the data loading process of network training, so that the data synthesis and network training processes are carried out synchronously.
5. A non-contact heart rate detection method based on face video according to claim 1, characterized in that: During the deep learning model training process, adjust the parameters of the data synthesis algorithm to increase the types of synthesized data.
6. A non-contact heart rate detection method based on face video according to claim 1, characterized in that: The implementation method of calculating the heart rate includes: collecting video of the face area where the heart rate is to be detected, amplifying the signal of each frame image in the collected video through a trained deep learning model, calculating the mean of the amplified face image, performing peak detection on the calculated curve, analyzing the time interval between the peaks and using this to calculate the average heart rate during this period.
7. The non-contact heart rate detection device used in the non-contact heart rate detection method based on face video according to any one of claims 1 to 6, characterized in that: include The video processing module is used to synthesize multiple videos V for simulating real face videos based on the face video V without skin color change. rPPG ; Deep learning model for using multiple videos V rPPG It is used as a training sample for training, and the signal of the face image to be detected for heart rate is amplified; The heart rate calculation module is used to obtain high signal-to-noise ratio data from the enlarged face image and then calculate the heart rate.
