Remote heart rate monitoring method based on face video analysis in random frame loss scene
By using the knowledge distillation method of attention migration in 3DCNN network to process the face image sequence after the frame is lost, the robust face pulse wave signal is extracted, and the accuracy reduction of remote heart rate measurement in random frame loss scenarios is solved, and the accuracy of remote heart rate measurement is reduced in random frame loss scenarios is improved. The robustness and adaptability of measurements are suitable for complex environments.
Patent Information
- Application Number
- CN202410941392.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-01
AI Technical Summary
The existing remote heart rate measurement methods cannot effectively process incomplete face image sequences in random frame drop scenarios, resulting in a serious decrease in heart rate accuracy and cannot meet the high-precision and high robustness requirements of telemedicine scenarios.
The robust 3DCNN network is adopted to train the model through the knowledge distillation method of attention transfer, and combine the teacher model and the student model to process the lost face image sequence, extract the robust face pulse wave signal, thereby calculating the heart rate.
In random frame drop scenarios, the robustness and adaptability of heart rate measurements are significantly improved, and the robust heart rate calculation can be performed in the case of unstable network transmission, which is suitable for more complex actual environments.
Smart Images

Figure CN118887718A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a heart rate measurement algorithm based on face video analysis, in particular to a remote heart rate monitoring method based on face video analysis in a random frame loss scenario. Background Art
[0002] By monitoring the human heart rate, we can effectively judge the load state of the cardiovascular system and roughly judge the health status of the human body. It is a very important physiological indicator. In daily life or medical monitoring, people can adjust their living habits and behaviors according to the heart rate value. When the heart rate is too high or too low, it can be detected and treated early.
[0003] Since the skin is translucent, when light hits the surface of the human face, part of the light will penetrate the epidermis and enter the blood vessels. This part of the light will be absorbed by the hemoglobin in the blood and then diffusely reflected and captured by the camera. With the periodic beating of the heart, the light absorbed by the blood under the skin also changes periodically, so the human pulse wave can be obtained by extracting the color information of the face, and the heart rate can be extracted from the frequency of the pulse wave changes.
[0004] However, factors such as light changes, face movement, and video compression rate in real scenes will greatly affect the calculation of heart rate indicators based on face video extraction. For example, in the case of network fluctuations, the image frames collected by the camera are likely to be lost when transmitted to the server, that is, the face image sequence received by the server is incomplete. Using the current heart rate measurement method to process the incomplete image sequence is likely to cause serious measurement errors, which does not meet the high-precision and high-robustness requirements of future telemedicine scenarios. Therefore, with the aging of the population and the increase in the number of people with potential cardiovascular sub-health, designing a more robust face pulse wave extraction algorithm suitable for a variety of complex scenarios has become an urgent need in the healthcare industry.
[0005] The invention patent with authorization announcement number CN112819790A and invention name "A heart rate detection method and device" selects an ROI area that meets the standard scenario, obtains the estimated heart rate value based on the peak state of the signal, and obtains the final heart rate value after aligning and fitting with the reference heart rate value.
[0006] The invention patent with authorization announcement number CN112200162A and invention name "Non-contact heart rate measurement method, system and device based on end-to-end network" uses a dual-branch 3D convolutional network to extract significant features from the face to predict the average heart rate value over a period of time.
[0007] However, the invention patent mentioned above only considers the face image sequence when no random frame loss occurs as input, and the data sets used are all complete sequences. If the existing method is applied to process the incomplete face image sequence in the case of random frame loss, the accuracy of the measured heart rate will be seriously reduced. Therefore, the existing remote heart rate measurement method is not suitable for application in the random frame loss scenario. Summary of the invention
[0008] The purpose of the present invention is to provide a remote heart rate monitoring method based on face video analysis in a random frame loss scenario in response to the problems existing in the prior art. The method calculates the heart rate using a facial pulse wave signal extracted by a 3DCNN network that is robust to random frame loss scenarios. The training process of the 3DCNN network that is robust to random frame loss scenarios is as follows: a sequence of facial images is recorded using an ordinary camera, and the collected sequence is transmitted to a server, during which a poor channel and random frame loss are simulated; an incomplete facial image sequence is received at the receiving end of the server, and the incomplete facial image sequence and the frame number where the frame loss occurs are known; first, a 3DCNN is trained using a complete image sequence as a teacher model to extract a facial pulse wave signal composed of skin chromaticity information of the face, and then a student model is trained using an incomplete facial image sequence in a knowledge distillation manner to restore a robust facial pulse wave signal. The method of the present invention can reliably measure the heart rate of a human body when the pulse wave signal is disturbed by frame loss.
[0009] The present invention is implemented by the following technical solutions:
[0010] A remote heart rate monitoring method based on face video analysis in a random frame loss scenario, the method uses a trained 3DCNN network to extract the face pulse wave signal in the random frame loss scenario, and calculates the human heart rate based on the face pulse wave signal; the 3DCNN network acquisition method consists of two main parts: generating random frame loss data and training a knowledge distillation network based on attention transfer. The 3DCNN that is robust to random frame loss scenarios is obtained in the network training step, which can be further expanded to more complex real-world measurement environments.
[0011] The method for obtaining the 3DCNN network includes the following specific steps:
[0012] Step 1: Use a camera to collect a continuous sequence of face images and divide it into a fixed length (which can be selected according to needs, such as 256 frames) of face image sequences;
[0013] Step 2: Randomly generate lost frames in the fixed-length face image sequence, which can be expressed as follows:
[0014]
[0015] Among them, I raw (t) represents the initial face image collected at time t, I drop (t) represents the face image after the lost frame is generated and received by the server at time t.
[0016] Step 3: Use the knowledge distillation method based on attention transfer to train the 3DCNN network, which can extract robust facial pulse wave signals in random frame loss scenarios;
[0017] Step 3.1: Use the initial face image sequence I raw The teacher network is trained with the real fingertip pulse wave signal to obtain a standard facial pulse wave extraction network;
[0018] Step 3.2: Initial face image sequence I raw Input the standard facial pulse wave extraction network, and obtain the intermediate layer features of the standard facial pulse wave extraction network after forward propagation and average pooling. in,
[0019] Represents the middle layer features of the i-th layer of the standard facial pulse wave extraction network after average pooling;
[0020] Step 3.3: Generate face image sequence I after missing frames drop Input the student network, forward propagation and average pooling to obtain the intermediate layer features of the student network And output facial pulse wave signal; represents the intermediate layer features of the i-th layer of the student network after average pooling;
[0021] Step 3.4: Calculate the knowledge distillation loss and signal regression loss, back-propagate and update the parameters of the student model to obtain the trained 3DCNN network.
[0022] When remote heart rate monitoring is required, the 3DCNN network trained in step 3 can be used to extract a robust facial pulse wave signal y(t) in a random frame loss scenario, and the human heart rate can be calculated based on the facial pulse wave signal y(t).
[0023] In the above technical solution, further, in the step S2, a lost frame is randomly generated in the fixed-length facial image sequence. Specifically, each frame image in the fixed-length facial image sequence is numbered in sequence, and the sequence number of the lost frame is randomly selected from the sequence numbers. The specific steps of generating random lost frame data are as follows:
[0024] a. Given a sequence of face images Among them, I raw(T) represents the face image collected at time T, T represents the length of the face image sequence, C represents the number of image channels, W and H represent the face
[0025] Image width and height;
[0026] b. Set the frame loss intensity missing rate∈[0,1] to simulate the probability of loss of the initial face image during network transmission.
[0027] The greater the frame loss intensity, the higher the probability of frame loss in the image at time t.
[0028] c. Generate the server receiving mask sequence m(t) represents the reception probability at time t, which is a random variable that obeys the uniform distribution U(0,1). The larger the m(t), the more likely the image is at time t.
[0029] Received by the server;
[0030] d. Compare the reception probability in the reception mask sequence m with the frame loss intensity missing rate one by one. If m(t)>missing rate, it means that the t-th frame face image can be received by the server and m(t) is assigned to 1. If m(t)≤missing rate, it means that the t-th frame face image is lost and m(t) is assigned to 0. The calculation formula of this process can be expressed as follows:
[0031]
[0032] e. Receive the mask sequence m and the initial face image sequence I raw Perform logical AND operation to obtain the face image sequence I received by the server after the lost frame is generated drop .
[0033] Furthermore, the knowledge distillation method based on attention transfer is used to train the 3DCNN network (i.e., the knowledge distillation network training part based on attention transfer), and the specific steps are as follows:
[0034] a. Using the initial face image sequence I raw and fingertip real pulse wave signal GT Training standard facial pulse wave extraction
[0035] Network (i.e. teacher network);
[0036] b. Initial face image sequence I raw Input the facial pulse wave extraction network, and after forward propagation and average pooling, obtain the intermediate layer features of the standard facial pulse wave extraction network.
[0037] c. Generate face image sequence I after missing frames drop Input the student network, forward propagation and average pooling to obtain
[0038] Middle-layer features of the student network And output facial pulse wave signal;
[0039] d. Calculate the knowledge distillation loss according to the attention transfer formula. The specific calculation formula is as follows:
[0040]
[0041] in, represents the number of intermediate layer features, and ‖·‖2 represents L2 regularization.
[0042] e. Calculate the signal regression loss of the facial pulse wave signal output by the student network and the real fingertip pulse wave signal.
[0043] The calculation formula is as follows:
[0044]
[0045] where y GT is the real pulse wave signal of the fingertip, and y is the facial pulse wave signal output by the student network.
[0046] f. Based on the total loss function Back propagation trains the student model until the model converges. β is the hyperparameter set during model training.
[0047] Furthermore, the step of calculating the human heart rate based on the facial pulse wave signal, i.e. the heart rate calculation part, comprises the following specific steps:
[0048] a. Extract the facial pulse wave signal from the trained 3DCNN network Perform Fourier transform to obtain the pulse wave signal amplitude spectrum Y;
[0049] b. Select the peak frequency component in the pulse wave signal amplitude spectrum and record it as the frequency component f of the human heart rate HR ;
[0050] c. HR Multiply by 60 to get the human heart rate.
[0051] The beneficial effects of the present invention are:
[0052] The present invention proposes a method for constructing a heart rate dataset containing random frame loss scenarios, which can generate a heart rate dataset with random frame loss based on an existing face video. Based on the random frame loss dataset, the present invention uses the knowledge distillation method based on attention transfer to train 3DCNN for the first time, so that in the scenario where the face video contains random frame loss input, the trained 3DCNN can effectively capture and transmit important features, further improving the performance of the model in the random frame loss scenario. By constructing a heart rate dataset with random frame loss and using advanced model training technology, the present invention improves the robustness and adaptability of remote heart rate calculation, enables the model to adapt to more complex actual environments, provides a guarantee for robust heart rate calculation under unstable network transmission conditions, and promotes the development of telemedicine technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 The present invention is an overall flow chart of the remote heart rate monitoring method based on face video analysis in a random frame loss scenario.
[0054] Figure 2 This is the network structure diagram for training 3DCNN based on knowledge distillation. DETAILED DESCRIPTION
[0055] The specific implementation modes of the present invention are further described below in conjunction with the accompanying drawings.
[0056] The whole process of the remote heart rate monitoring method based on face video analysis in the random frame loss scenario described in the present invention is described in Figure 1 , a 3DCNN network that is trained to be robust to random frame loss scenarios is used to extract the facial pulse wave signal in the random frame loss scenario, and the human heart rate is calculated based on the facial pulse wave signal.
[0057] The facial pulse wave signal extracted by the 3DCNN network is transformed by Fourier transform to finally calculate the heart rate. The specific method is as follows: First, the facial pulse wave signal extracted by the trained 3DCNN network is transformed by Fourier transform. Perform Fourier transform to obtain the pulse wave signal amplitude spectrum Y; then, select the peak frequency component in the pulse wave signal amplitude spectrum and record it as the frequency component f of the human heart rate HR ; Finally, f HR Multiply by 60 to get the human heart rate.
[0058] The acquisition process of a 3DCNN network that is robust to random frame loss scenarios is as follows: Figure 2 As shown, the specific steps include:
[0059] First, a facial image sequence is collected using an ordinary camera and divided into facial image sequences of 256 frames in length. Where T = 256 represents the length of the face image sequence, C represents the number of image channels, and W and H represent the width and height of the face image. Set the frame loss intensity missing rate∈[0,1] to simulate the probability of loss of the image collected by the mobile phone in the network transmission signal.
[0060] Generate the server receiving mask sequence according to the missing frame intensity Where m(t) is a random variable that follows uniform distribution U(0,1), which indicates the probability of receiving a face image at time t. The receiving probabilities in the receiving mask sequence m are compared with the frame loss intensity missing rate one by one: if m(t)>missing rate, it means that the frame image can be received by the server at time t and m(t) is assigned a value of 1; if m(t)≤missing rate, it means that the frame is lost at time t and m(t) is assigned a value of 0. raw Perform logical AND operation to obtain the face image sequence I received by the server after the lost frame is generated drop .
[0061] Next, the initial face image sequence I is used raw and fingertip real pulse wave signal GT Train the teacher network, where the teacher network is trained on a complete sequence of face images. The spatial attention of the teacher network can extract features with lower loss, and the intermediate layer features of the teacher model can serve as a guide for the spatial attention of the student model. raw Input the teacher network, forward propagation and average pooling to obtain the intermediate layer features of the teacher network Represents the middle layer features of the i-th layer of the standard facial pulse wave extraction network after average pooling; the face image sequence I after the lost frame is generated drop Input the student network, forward propagation and average pooling to obtain the intermediate layer features of the student network And the output facial pulse wave signal, Represents the intermediate layer features of the i-th layer of the student network after average pooling. Knowledge distillation loss based on attention transfer And the signal regression loss of the output pulse wave and the real pulse wave of the fingertip The student model is trained and iterated until the network converges, thereby obtaining a 3DCNN network that is robust to frame loss scenarios.
[0062] Using the trained 3DCNN network for inference, when the input face image sequence contains randomly lost frames, the 3DCNN network can extract robust facial pulse wave signals.
[0063] The method of the present invention is verified on the public datasets PURE and UBFC for face heart rate measurement. As shown in Table 1, the face image sequence with frame loss directly trained by 3DCNN is tested on the PURE dataset and the evaluation index of the real heart rate calculation is shown in the left column of Table 1. The face image sequence with frame loss trained by 3DCNN based on knowledge distillation of attention transfer is compared with the evaluation index of the real heart rate as shown in the right column of Table 1. The evaluation index is higher than the left column as a whole. As shown in Table 2, in the UBFC dataset test, the evaluation index is also higher than the left column as a whole. The experimental results show that the method of the present invention has good noise resistance in calculating heart rate in random frame loss scenarios, and there is an opportunity to expand the heart rate measurement based on face video to more complex scenarios in reality.
[0064] Table 1 Evaluation indicators of heart rate calculation in PURE dataset with frame loss intensity ranging from 0.1 to 0.9
[0065]
[0066] Table 2 Evaluation indicators of heart rate calculation in UBFC dataset with frame loss intensity of 0.1 to 0.9
[0067]
[0068]
[0069] The above are only specific implementation methods of the present invention, which cannot be used to limit the scope of the present invention. Equivalent changes made by ordinary technicians in this technical field based on this application, as well as changes known to technicians in this field, should still fall within the scope of the present invention.
Claims
1. A remote heart rate monitoring method based on face video analysis in a random frame loss scenario, characterized in that: The trained 3DCNN network is used to extract the facial pulse wave signal in the random frame loss scenario, and the human heart rate is calculated based on the facial pulse wave signal; the acquisition method of the 3DCNN network has the following specific steps: S1: Use a camera to collect a continuous sequence of face images and divide it into fixed-length face image sequences; S2: randomly generate lost frames in the fixed-length face image sequence, expressed by the following formula: Among them, I raw (t) represents the initial face image collected at time t, I drop (t) represents the face image after the missing frame is generated at time t; S3: Use the knowledge distillation method based on attention transfer to train the 3DCNN network; S31: Using the initial face image sequence I raw The teacher network is trained with the real fingertip pulse wave signal to obtain a standard facial pulse wave extraction network; S32: Initial face image sequence I raw Input the standard facial pulse wave extraction network, and obtain the intermediate layer features of the standard facial pulse wave extraction network after forward propagation and average pooling; S33: Generate face image sequence I after missing frames drop Input the student network, and after forward propagation and average pooling, obtain the intermediate layer features of the student network and the output facial pulse wave signal; S34: The knowledge distillation loss is calculated using the standard facial pulse wave extraction network and the intermediate layer features of the student network. The signal regression loss is calculated using the facial pulse wave signal obtained based on the student network and the real fingertip pulse wave signal. The parameters of the student model are updated by backpropagation to obtain a trained 3DCNN network.
2. The remote heart rate monitoring method based on face video analysis in a random frame loss scenario according to claim 1, characterized in that: The step S2 specifically includes the following steps: a. Given a sequence of face images Among them, I raw (T) represents the face image collected at time T, T represents the length of the face image sequence, C represents the number of image channels, W and H represent the width and height of the face image; b. Set the frame loss intensity missing rate∈[0, 1] to simulate the probability of one frame in the initial face image sequence being lost during network transmission; c. Generate the server receiving mask sequence m(t) represents the probability of receiving a face image at time t, which is a random variable that follows a uniform distribution U(0,1); d. Compare the reception probability in the reception mask sequence m with the frame loss intensity missing rate one by one. If m(t)>missing rate, it means that the t-th frame face image can be received by the server and m(t) is assigned to 1. If m(t)≤missing rate, it means that the t-th frame face image is lost and m(t) is assigned to 0. The calculation formula of this process is expressed as follows: e. Perform a logical AND operation on the received mask sequence and the initial face image sequence to obtain a face image sequence after generating the lost frame.
3. The remote heart rate monitoring method based on face video analysis in a random frame loss scenario according to claim 1, characterized in that: The step S34 specifically includes the following steps: a. Intermediate layer features based on standard facial pulse wave extraction network and student model and The knowledge distillation loss is calculated according to the attention transfer formula. The specific calculation formula is as follows: in, represents the number of intermediate layer features, ||·||2 represents L2 regularization, Represents the middle layer features of the i-th layer of the standard facial pulse wave extraction network after average pooling, represents the intermediate layer features of the i-th layer of the student network after average pooling; b. Calculate the signal regression loss of the facial pulse wave signal output by the student network and the real fingertip pulse wave signal. The specific calculation formula is as follows: Among them, YG T is the real pulse wave signal of the fingertip, y is the facial pulse wave signal output by the student network; c. Based on the total loss function β is a hyperparameter set during model training; the student model is trained by back propagation until the model converges to obtain the trained 3DCNN network.
4. The remote heart rate monitoring method based on face video analysis in a random frame loss scenario according to claim 1, characterized in that: The specific steps of calculating the human heart rate based on the facial pulse wave signal are as follows: a. Extract the facial pulse wave signal from the trained 3DCNN network Perform Fourier transform to obtain the pulse wave signal amplitude spectrum Y; b. Select the peak frequency component in the pulse wave signal amplitude spectrum and record it as the frequency component f of the human heart rate HR ; c. f HR Multiply by 60 to get the human heart rate.
