Method and device for heart rate estimation based on face video

Abstract

The application discloses a heart rate estimation method and device based on a face video, which estimates a heart rate value through analysis of the face video, comprises the following steps: pre-processing a face video to obtain a video data set, and dividing the video data set into a training set, a verification set and a test set; constructing a fusion network comprising a ResNet network and a ConvLSTM network, the ResNet network comprising a bottleneck residual module to improve the performance of the ResNet network; inputting the training set into the fusion network and training to obtain a network training model; inputting the verification set into the network training model for verification, and calculating a loss through a loss function to obtain a final network training model; inputting the test set into the final network training model to obtain a heart rate estimation value; and cross-dataset testing to improve the precision of the model. The application increases the bottleneck residual module in the ResNet network, improves the calculation efficiency of the ResNet network, enables the ResNet network to better train a neural network with a large depth, and through the fusion of the ConvLSTM network, global features can be fully utilized to improve the accuracy of the heart rate estimation value.

Description

Technical Field

[0001] This invention relates to a method and apparatus for estimating heart rate based on facial video, belonging to the field of image processing technology. Background Technology

[0002] Heart rate (also known as heart rate or heart rate beat) is one of the most important vital signs in humans. Since ancient times, heart rate has been used to diagnose and assess disease. In recent years, remote heart rate monitoring has become a significant need and a core research area. Applications include telemetry monitoring of the elderly, and real-time monitoring of premature infants, critically ill patients, and athletes participating in sports activities. In such cases, even highly trained doctors cannot be present at all times to monitor the patient's heart rate. To address this issue, researchers have begun to study estimating heart rate from facial videos. Specifically, although the human eye cannot see changes in blood flow in the facial area, these changes can be captured in video, allowing for the measurement of heart rate through facial video sequences.

[0003] Currently, a deep learning-based, end-to-end approach has been proposed for heart rate detection to address the non-contact problem. This approach uses a multi-layered network structure to extract low-level features, forming an abstract high-level representation to discover spatiotemporal correlations in the data. The heart rate of the target is estimated based on subtle facial color changes caused by heartbeats.

[0004] Heart rate is estimated by analyzing the frequency of facial color changes caused by heartbeats over a period of time. In a series of inputs prior to a certain moment, each input is related to the previous one and influences the estimation result to varying degrees. Feedforward neural networks (RNNs) only depend on the current input for each output, neglecting the interaction of inputs at different times, making them unsuitable for handling spatiotemporal sequence problems. Therefore, a recurrent neural network (RNN) specifically designed for time series data was proposed. However, this network struggles with excessively long sequences, as it only has short-term memory and lacks long-term memory, leading to reduced accuracy in heart rate estimation. Furthermore, while traditional ResNet networks can address the training difficulties of excessively deep networks, they require additional parameters, such as transform gates and carry gates, increasing the computational burden and reducing efficiency. Summary of the Invention

[0005] The purpose of this invention is to provide a method and apparatus for heart rate estimation based on facial video, which can solve the problems of low accuracy and low computational efficiency in existing heart rate estimation techniques.

[0006] To achieve the above objectives, this invention provides a heart rate estimation method based on facial video, which estimates heart rate values ​​through analysis of facial video, including the following steps:

[0007] Step 1: Preprocess the face video to obtain a video dataset, and divide the video dataset into a training set, a validation set, and a test set;

[0008] Step 2: Construct a fusion network, which includes a ResNet network and a ConvLSTM network. The ResNet network includes a bottleneck residual module to improve the performance of the ResNet network.

[0009] Step 3: Input the training set into the fusion network and train it to obtain the network training model;

[0010] Step 4: Input the validation set into the network training model for validation, and calculate the loss using the loss function. If the loss is not within the error range, modify the network training model and return to step 3; if the loss is within the error range, the final network training model is obtained.

[0011] Step 5: Input the test set into the final network training model to obtain the heart rate estimate;

[0012] Step 6: Perform cross-dataset testing to improve the accuracy of the final network training model and the accuracy of the heart rate prediction.

[0013] As a further improvement of the present invention, the preprocessing in step 1 includes segmenting and cropping the face video. The face video is segmented using the color change information of the face video to retain the facial region in the face video. The segmented face video is then cropped to obtain several video segments with the same number of frames.

[0014] As a further improvement of the present invention, the preprocessing in step 1 also includes calculating the spatiotemporal graph of the video segment. Specifically, facial regions are aligned between different frames of the video segment, and the facial regions are converted into the YUV color space. Simultaneously, the facial regions are divided into several Regions of Interest (ROIs), wherein the average color value of the i-th ROI in the t-th frame is:

[0015] Where C(x,y,t) represents the (x,y) value at the t-th frame from different dimensions of the YUV color space, |ROI i | represents the area of ​​the i-th ROI block. The average color values ​​of ROI blocks at the same location but in different frames are concatenated to obtain the time series of that location. Several of the time series are arranged to obtain the spatiotemporal map.

[0016] As a further improvement of the present invention, the YUV color space transformation specifically includes:

[0017] As a further improvement of the present invention, the time series is normalized and the value of the time series is scaled to [0, 255]. The training set, validation set and test set each include several spatiotemporal graphs.

[0018] As a further improvement of the present invention, the fusion network in step 2 uses a ResNet network as the backbone and incorporates a ConvLSTM network into the ResNet network. The ResNet network includes a max pooling layer, a module layer, a fully connected layer, and an average pooling layer. The ConvLSTM network is located between the module layer and the fully connected layer. The fusion network also includes a gated recurrent unit located between the ConvLSTM network and the fully connected layer. The module layer has four modules, each of which includes the bottleneck residual module.

[0019] As a further improvement of the present invention, the gated loop unit includes a reset gate and an update gate, and the bottleneck residual module includes three convolutional layers with kernel sizes of 1×1, 3×3 and 1×1, respectively.

[0020] As a further improvement of the present invention, the calculation formula of the ConvLSTM network is as follows:

[0021]

[0022]

[0023]

[0024]

[0025]

[0026] Where * represents a convolution operation; σ represents the Hadamard operation; f is the forget gate; i is the input gate; o is the output gate; C is the memory unit; σ is the sigmoid neural network layer; b is the bias term; t represents time step; X is the input; W is the weight; and H is the hidden layer input.

[0027] As a further improvement of the present invention, the calculation of loss using the loss function in step 4 is specifically as follows:

[0028]

[0029]

[0030] During backpropagation, the smoothing loss L smooth Compared to the input hr t The partial derivatives are calculated as follows:

[0031]

[0032] The loss function L is: L = L l1 +λL smooth ;

[0033] Among them, hr t For heart rate estimates, hr m Let L be the average heart rate estimate from T consecutive measurements. smooth The loss function is smoothing, sng(x) is the sign function, and L is the loss function. l1 Let λ represent the loss function of L1, and λ be the balancing parameter.

[0034] To achieve the above objectives, the present invention also provides a heart rate estimation device, including a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor executes the program instructions to implement the aforementioned heart rate estimation method based on face video.

[0035] The beneficial effects of this invention are:

[0036] 1. This invention is based on a fusion network that combines an improved ResNet network and a ConvLSTM network. The input is a face video, and the output is the heart rate estimate at that moment, making the result more intuitive.

[0037] 2. This invention improves the computational efficiency of the ResNet network by adding a bottleneck residual module to each layer, enabling it to better train deeper neural networks;

[0038] 3. By incorporating a ConvLSTM network, this invention can fully utilize global features to capture the spatiotemporal correlation between global features, making the learned global information more reasonable and improving the accuracy of heart rate estimation. Attached Figure Description

[0039] Figure 1 This is a flowchart of the steps of the heart rate estimation method based on facial video of the present invention.

[0040] Figure 2 This is a network structure diagram of the heart rate estimation method based on face video of the present invention.

[0041] Figure 3 yes Figure 1 A flowchart for preprocessing facial images in Chinese videos.

[0042] Figure 4 yes Figure 2 Network structure diagram of the integrated network.

[0043] Figure 5 yes Figure 4 A schematic diagram of the bottleneck residual module.

[0044] Figure 6 yes Figure 4 Diagram of the ConvLSTM network structure.

[0045] Figure 7 yes Figure 6 A schematic diagram of the coding and prediction networks in a ConvLSTM network. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Please see Figure 1 and Figure 2 As shown, this invention provides a heart rate estimation method and apparatus based on facial video. The method preprocesses the facial video and inputs the preprocessed video into a fusion network. The fusion network calculates the heart rate estimate based on the color change information between each frame of the facial video. The fusion network includes a ResNet network and a ConvLSTM network. Specifically, the fusion network uses a ResNet network as the backbone, with a ConvLSTM network integrated into the ResNet network. By adding a bottleneck residual module to the ResNet network, computational efficiency is improved, enabling better training of deeper neural networks. By integrating the ConvLSTM network into the ResNet network, global features can be fully utilized to capture the spatiotemporal correlation between global features, making the learned global information more reasonable and improving the accuracy of the heart rate estimate.

[0048] Specifically, LSTM (Long Short-Term Memory) networks are a variant of RNNs that combine short-term and long-term memory through sophisticated gating. ConvLSTM networks are an optimization of LSTM, incorporating convolutional structures in the input-to-state and state-to-state transitions, allowing them to better capture spatiotemporal correlations. ResNet networks address the training difficulties of excessively deep networks. Therefore, a fusion of these two technologies can improve accuracy while better capturing spatiotemporal correlations, resulting in more accurate heart rate estimations.

[0049] The specific steps of this invention are as follows:

[0050] Step 1: Preprocess the face videos to obtain a video dataset, and divide the video dataset into a training set, a validation set, and a test set.

[0051] Please see Figure 2 and Figure 3As shown, the preprocessing includes segmentation and cropping of the face video. Specifically, the heartbeat causes subtle changes in the color of the facial area. The face video is segmented using the color change information to preserve the facial area and reduce the noise impact of non-facial areas on the heart rate estimation, such as the eye area and background area.

[0052] The segmented face video is cropped to obtain several video segments with the same number of frames. Specifically, each video segment includes w frames with a step size of 0.5 seconds.

[0053] The preprocessing also includes calculating the spatiotemporal graph of the aforementioned video segment. Specifically, first, the detected facial regions are aligned between different frames of the video segment. Then, the facial regions are converted into the YUV color space. Finally, the facial regions are divided into several ROI blocks, the average color value is calculated, and they are arranged to obtain the spatiotemporal graph of the video segment.

[0054] Specifically, the transformation of the facial region into the YUV color space can be represented as:

[0055]

[0056] A video segment is defined with T frames and a C color space, i.e., the YUV color space. The average color value of the i-th ROI block in the t-th frame can be expressed as: Where C(x,y,t) represents the (x,y) value at the t-th frame from different dimensions of the YUV color space, |ROI i | represents the area of ​​the i-th ROI block.

[0057] By concatenating the average color values ​​of the same location across different frames, a time series for that location can be obtained. The time series of several ROI blocks can be represented as Y1, U1, V1, Y2, U2, V2, ... Y n U n and V n To better utilize facial color variations, the time series data is normalized and scaled to [0, 255]. The time series data are then arranged in rows to form a spatiotemporal graph of size T×n×C. Specifically, the video dataset includes several spatiotemporal graphs; that is, the training set, validation set, and test set each consist of several spatiotemporal graphs.

[0058] Step 2: Construct a fusion network, which includes a ResNet network and a ConvLSTM network. The ResNet network includes a bottleneck residual module to improve the performance of the ResNet network.

[0059] Please see Figure 2 and Figure 4As shown, the fusion network uses a ResNet network as its backbone and incorporates a ConvLSTM network within it. Specifically, the ResNet network includes convolutional layers, max pooling layers, module layers, fully connected layers, and average pooling layers. The ConvLSTM network is positioned between the module layers and the fully connected layers to capture the spatiotemporal correlation of global features.

[0060] The fusion network also includes a gated recurrent unit located between the ConvLSTM network and the fully connected layer. The gated recurrent unit includes a reset gate and an update gate, which are used to feed the features extracted by the ConvLSTM network to the fully connected layer.

[0061] The first layer in the fusion network is a convolutional layer, which takes the spatiotemporal map as input and performs convolution operations on it before inputting it into a max pooling layer. The max pooling layer divides the input spatiotemporal map into several rectangular regions, outputs the maximum value for each sub-region, and inputs the maximum value into the module layer. The module layer performs calculations on the spatiotemporal map, extracts global features from the spatiotemporal map through a ConvLSTM network, and inputs the extracted features into a gated recurrent unit to model the temporal relationship between subsequent measurements. The extracted features are then passed through the gated recurrent unit and enter a fully connected layer for regression to obtain the heart rate estimate for a single video segment. The average pooling layer calculates the average of the heart rate estimates for multiple single video segments to obtain the final heart rate estimate.

[0062] Specifically, there are four module layers, and each module layer includes a bottleneck residual module. By setting the bottleneck residual module in the module layer, the ResNet network can train neural networks with greater depth.

[0063] Please see Figure 5 As shown, in existing technologies, conventional convolutional neural network training typically uses a highway network: y = F(x, w f )·Γ(x,w t )+x·C(x,w c );

[0064] Where: Γ(x,w) t The transformation is nonlinear and is called a "transform gate," which controls the strength of the transformation. C(x,w) c The transform is nonlinear and called a "carry gate," which controls the strength of the original input signal retained. In other words, y is a function of F(x,w). f The weighted combination of x and Γ, where Γ and C control the weights of the two items respectively.

[0065] Subsequently, a simplified highway network was proposed based on the existing highway network: y = F(x, w f )·Γ(x,wt )+x·(1-Γ(x,w t This means that the nonlinear transformation has been removed.

[0066] When both the "transformation gate" and the "carry gate" of the highway network are identity mappings (i.e., let Γ = 1, C = 1), the bottleneck residual network is obtained: y = F(x, w) + x;

[0067] Multiple bottleneck residual modules are stacked to form a bottleneck residual network. Compared with the highway network, the shortcut connection in the bottleneck residual network is completed directly through a simple identity mapping, without the need for complex "transformation gates" and "carry gates". Therefore, no additional parameters are needed, which reduces the computational burden of the network.

[0068] The bottleneck residual module includes three convolutional layers with kernel sizes of 1×1, 3×3 and 1×1. The 1×1 kernel can increase or decrease the dimensionality of the channel count, while the 3×3 kernel can perform convolution operations with a relatively low-dimensional input to improve computational efficiency.

[0069] Please see Figure 2 and combined Figure 6 and Figure 7 As shown, since the heart rate estimate based on face video can be described as a prediction of a spatiotemporal sequence, where both the input value and the predicted value are spatiotemporal sequences, a ConvLSTM network with convolutional structures in both the input-to-state and state-to-state transitions is proposed by extending the fully connected LSTM network. This enables the ConvLSTM network to better capture spatiotemporal correlations.

[0070] Please see Figure 6 As shown, the formula for the ConvLSTM network structure is as follows:

[0071]

[0072]

[0073]

[0074]

[0075]

[0076] Where * represents a convolution operation; σ represents the Hadamard operation; σ is the sigmoid neural network layer, outputting values ​​between 0 and 1; the tanh layer outputs values ​​between -1 and 1; f is the forget gate, determining the extent to which the previous layer's output is forgotten; i is the input gate, controlling the extent to which the current calculated state is updated in the memory cells; o is the output gate, controlling the extent to which the current output depends on the current memory cell; b is the bias term; t is the current time step; t-1 is the previous time step; X is the input; W is the weight; H is the hidden layer input; C is the memory cell. It can be seen that the cell state is calculated by combining the weights, input, the hidden layer input of the previous layer, the memory cell state of the previous layer, and the input gate. The hidden layer state of this layer is determined by the output gate and the memory cell state.

[0077] Please see Figure 7 As shown, multiple ConvLSTM networks are stacked to form a network as shown in the figure. Figure 7 The encoding and prediction networks are shown.

[0078] The initial state and output of the prediction network are copied from the final state of the encoding network. Since the target to be predicted has the same dimension as the input, all the states in the prediction network are concatenated and fed into a 1x1 convolutional layer to generate the final prediction result.

[0079] Step 3: Input the training set into the aforementioned fusion network and train it to obtain the network training model.

[0080] Step 4: Input the validation set into the aforementioned network training model for validation, and calculate the loss using the loss function. If the loss is not within the error range, modify the network training model and return to step 3; if the loss is within the error range, the final network training model is obtained.

[0081] The error range is specifically defined as the acceptable error range for medical heart rate testing.

[0082] Because the subjects' heart rate changes very little over a short period of time, a smoothing loss function is introduced to constrain the smoothness of adjacent heart rate measurements. The average heart rate estimate for T consecutive measurements is:

[0083] Among them, hr t For heart rate estimates, hr m Let T be the average heart rate estimate from T consecutive measurements.

[0084] The smoothing loss function is defined as:

[0085]

[0086] During backpropagation, the smoothing loss L smooth Compared to the input hrt The partial derivatives can be calculated as follows:

[0087]

[0088] The loss function L is: L = L l1 +λL smooth ;

[0089] Among them, L smooth The loss function is smoothing, sng(x) is the sign function, and L is the loss function. l1 Let λ represent the L1 loss function, used to measure the difference between the predicted heart rate estimate and the actual heart rate estimate, and λ be the balance parameter.

[0090] Step 5: Input the test set into the final network training model to obtain the heart rate estimate.

[0091] Step 6: Cross-dataset testing to improve the accuracy of the final network training model and the accuracy of heart rate estimation. Replace the face videos with different ones and input them into the final network training model, repeating steps 3 through 6 to improve the accuracy of the final network training model and the accuracy of heart rate estimation.

[0092] The present invention also provides a heart rate estimation device for performing heart rate estimation calculations based on facial video. The heart rate estimation device includes a memory and a processor coupled to each other. The memory stores program instructions, and the processor executes the program instructions to implement the aforementioned heart rate estimation method based on facial video, thereby further realizing heart rate estimation through facial video.

[0093] Specifically, we used the PyTorch library in a Python environment on a GeForce GTX 1080 Ti processor. For network optimization, we used the Adam optimizer and the non-linear activation function ReLU in our experiments. During network training, we set the number of iterations to 100 and the learning rate to 0.001. To achieve accurate convergence, we also set the stride to 8.

[0094] In summary, this invention provides a heart rate estimation method based on face video. By fusing an improved ResNet network and a ConvLSTM network, the input is a face video, and the output is the heart rate estimate at that moment, making the result more intuitive. By adding a bottleneck residual module to each layer of the ResNet network, the computational efficiency of the ResNet network is improved, enabling it to better train deeper neural networks. By incorporating a ConvLSTM network, global features can be fully utilized to capture the spatiotemporal correlation between global features, making the learned global information more reasonable and improving the accuracy of the heart rate estimate. By segmenting the face video into multiple ROI regions, all color change information caused by heartbeats can be more fully utilized. By segmenting the face video to preserve the facial region, the noise influence of non-facial regions on the heart rate estimate is reduced.

[0095] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A heart rate estimation method based on facial video, which estimates heart rate values ​​by analyzing facial videos, characterized in that... Includes the following steps: Step 1: Preprocess the face video to obtain a video dataset, and divide the video dataset into a training set, a validation set, and a test set; the preprocessing includes segmenting and cropping the face video, segmenting the face video using the color change information of the face video to retain the facial region in the face video; cropping the segmented face video to obtain several video segments with the same number of frames; Step 2: Construct a fusion network, which includes a ResNet network and a ConvLSTM network. The ResNet network includes a bottleneck residual module to improve the performance of the ResNet network. Step 3: Input the training set into the fusion network and train it to obtain the network training model; Step 4: Input the validation set into the network training model for validation, and calculate the loss using the loss function. If the loss is not within the error range, modify the network training model and return to step 3; if the loss is within the error range, the final network training model is obtained. Step 5: Input the test set into the final network training model to obtain the heart rate estimate; Step 6: Perform cross-dataset testing to improve the accuracy of the final network training model and the accuracy of the heart rate prediction. The fusion network described in step 2 uses a ResNet network as its backbone and incorporates a ConvLSTM network into the ResNet network. The ResNet network includes a max pooling layer, a module layer, a fully connected layer, and an average pooling layer. The ConvLSTM network is located between the module layer and the fully connected layer. The fusion network also includes a gated recurrent unit located between the ConvLSTM network and the fully connected layer. The module layer has four modules, each of which includes the bottleneck residual module. The gated loop unit includes a reset gate and an update gate, and the bottleneck residual module includes three convolutional layers with kernel sizes of 1×1, 3×3 and 1×1, respectively. Step 1 preprocessing also includes calculating the spatiotemporal graph of the video segment. Specifically, facial regions are aligned between different frames of the video segment, and the facial regions are converted to the YUV color space. Simultaneously, the facial regions are divided into several Regions of Interest (ROIs), where the average color value of the i-th ROI in the t-th frame is: Where C(x,y,t) represents the (x,y) value at the t-th frame from different dimensions of the YUV color space, |ROI i | represents the area of ​​the i-th ROI block. The average color values ​​of ROI blocks at the same location but in different frames are concatenated to obtain the time series of that location. Several of the time series are arranged to obtain the spatiotemporal map.

2. The heart rate estimation method based on face video according to claim 1, characterized in that: The YUV color space transformation specifically involves:

3. The heart rate estimation method based on face video according to claim 1, characterized in that: The time series is normalized and its values ​​are scaled to [0, 255]. The training set, validation set, and test set each include several spatiotemporal graphs.

4. The heart rate estimation method based on face video according to claim 1, characterized in that: The calculation formula for the ConvLSTM network is as follows: Where * represents a convolution operation; σ represents the Hadamard operation; f is the forget gate; i is the input gate; o is the output gate; C is the memory unit; σ is the sigmoid neural network layer; b is the bias term; t represents time step; X is the input; W is the weight; and H is the hidden layer input.

5. The heart rate estimation method based on face video according to claim 1, characterized in that: Step 4 involves calculating the loss using the loss function as follows: During backpropagation, the smoothing loss L smooth Compared to the input hr t The partial derivatives are calculated as follows: The loss function L is: L = L l1 +λL smooth ; Among them, hr t For heart rate estimates, hr m Let L be the average heart rate estimate from T consecutive measurements. smooth The loss function is smoothing, sng(x) is the sign function, and L is the loss function. l1 Let λ represent the loss function of L1, and λ be the balancing parameter.

6. A heart rate estimation device, characterized in that: The method includes a memory and a processor coupled to each other, the memory storing program instructions, and the processor executing the program instructions to implement the heart rate estimation method based on face video as described in any one of claims 1-5.

Images