Heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition
Through the method based on facial feature point recognition and empirical modal decomposition, the problems of low ROI extraction accuracy and weak anti-interference ability in contactless heart rate recognition are solved, real-time and accurate heart rate recognition are achieved, and the signal center is improved Rate extraction effect.
Patent Information
- Application Number
- CN202310150162.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-22
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-02-22
AI Technical Summary
In the non-contact heart rate recognition, the facial ROI area extraction accuracy is low and the calculation speed is slow, the anti-interference ability is weak, and the time domain signal resolution is poor, making it difficult to achieve real-time and accurate heart rate recognition.
Using a method based on facial feature point recognition and empirical modal decomposition, the 3D feature points on the face are inferred through the face mesh network, the mean of the ROI region is calculated, and the sliding window strategy is used to perform empirical modal decomposition and time-frequency domain analysis, combining the time domain and Frequency domain information performs heart rate estimation, adaptive noise reduction and reduce interference.
It realizes effective noise reduction and fast accuracy of heart rate extraction of rPPG signals, improves recognition effect, can run in real time on low computing power platforms, and provides stable heart rate estimation.
Smart Images

Figure CN116189264B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of deep learning and signal processing, and in particular to a heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition. Background Art
[0002] Photoplethysmography (PPG), also known as light volumetric method, can identify physiological signals through the light information of the skin area. PPG is generally divided into contact cPPG and non-contact rPPG (r means remote). Non-contact rPPG uses an RGB camera for heart rate recognition, which generally includes two parts: the first is ROI area capture and information statistics, and the second is to analyze the proposed information in the time domain to obtain heart rate data.
[0003] Common implementation solutions and their limitations are introduced as follows:
[0004] ROI area extraction scheme:
[0005] 1) Using the facial skin segmentation algorithm, the extraction area is the largest and most accurate, and the reflected light change information of the facial skin area is fully captured. However, due to the calculation speed of the segmentation algorithm, it is impossible to perform real-time operations on a low-computing power platform. In addition, due to the three-dimensional characteristics of the face, changes in illumination in different areas will reduce the signal-to-noise ratio and weaken the recognition ability.
[0006] 2) Using head recognition and rough facial feature point recognition algorithms, the accuracy of the extracted area is average, but it is impossible to obtain all exposed skin areas. However, it is fast and can achieve real-time recognition and information extraction.
[0007] Time domain heart rate extraction solution:
[0008] 1) Use frequency domain analysis, perform FFT transformation, and then find the area with the largest energy to obtain the heart rate. This method is the simplest to implement, but the effect is extremely poor in actual use, because environmental interference, respiratory interference, and signal modulation interference make it difficult to capture the heart rate peak in the frequency domain, and the anti-interference ability is extremely low.
[0009] 2) Sliding the time domain signal window and folding multiple windows, using a multi-layer perceptron or a convolutional neural network for recognition. This method requires a large amount of data calibration, and it is difficult to calibrate continuous heartbeat signals, and the time domain resolution is poor. Summary of the invention
[0010] The purpose of the present invention is to provide a heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition. The present invention can effectively reduce the noise of rPPG signals, and the heart rate extraction is fast and accurate.
[0011] The technical solution of the present invention: a heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition comprises the following steps:
[0012] Step 1: Take the mean value of the facial ROI of a single frame of the RGB video stream as the rPPG signal sampling point, and sample to obtain the rPPG signal stream;
[0013] Step 2: For the rPPG signal stream in step 1, use the sliding window strategy to decompose the heartbeat signal using the empirical mode decomposition algorithm in a single window;
[0014] Step 3: Perform heart rate analysis in the time domain and frequency domain based on the heartbeat signal to obtain a heart rate estimate.
[0015] The above-mentioned heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition uses a face detection model to infer facial 3D feature points from a single frame of an RGB video stream and calculates the mean value of the facial ROI area surrounded by the feature points.
[0016] In the aforementioned heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition, the face detection model is a face mesh network; the face mesh network performs 2D position recognition and 3D depth estimation of facial feature points to obtain 11 facial feature points, namely 5 on the forehead and 3 on each of the left and right cheeks, corresponding to three facial ROI areas; then the 3D information estimated from the feature points is used to calculate the normal vector of the ROI area, and the ROI area with a smaller angle between the normal vector and the camera's line of sight is selected to count the RGB pixel mean, and random ambient light fast sampling is used to eliminate ambient light floating interference to obtain facial ROI mean statistics.
[0017] The aforementioned heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition reduces the calls of the facial tracking algorithm by adjusting the facial tracking and feature point extraction strategies of the facemesh network.
[0018] The aforementioned heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition subtracts the mean signal stream of the background area in the RGB video stream from the rPPG signal stream to remove the drift caused by ambient light.
[0019] In the aforementioned heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition, in step 2, the rPPG signal stream is analyzed by sliding window with W sampling points as the time window, the signal of each window is subjected to empirical mode decomposition, energy statistics and FFT analysis are performed on all components, and the signal with an energy peak of 0.5-3Hz is taken as the heartbeat signal, which is sent to a filter with a cutoff frequency of 0.5-3Hz for filtering to obtain the heartbeat signal.
[0020] In the aforementioned heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition, in step 3, the heart rate analysis includes the following steps:
[0021] Step 3.1, in the heartbeat signal restored by the W sampling point window, the head and tail parts of the sampling point window are truncated, and the remaining window is used as a stable estimation segment to reduce the impact of the endpoint effect caused by signal decomposition and filtering, and the time when the heartbeat pulse occurs is counted in the stable estimation segment;
[0022] Step 3.2, repeat step S3.1 in multiple sliding windows to obtain a series of estimated values of heartbeat pulse moments, and count the heartbeat pulses with moments close to 0.3s for detection and counting; then set a detection count threshold to exclude heartbeat pulses with detection times less than the threshold, and calculate the average of the retained heartbeat pulses as the estimate of the heartbeat pulse moment;
[0023] Step 3.3, between different sliding windows in step S3.2, observe the occurrence time of the heartbeat pulse peak value in the time overlapping part, set the effective detection number threshold of the heartbeat pulse peak value, and obtain the time domain heart rate statistics;
[0024] Step 3.4, use fast Fourier transform to obtain the main frequency of the heartbeat pulse of a single sliding window, and obtain the median of the main frequency between the sliding windows as the frequency domain heart rate statistics;
[0025] Step 3.5: Use the average of the time domain heart rate statistics and the frequency domain heart rate statistics as the final heart rate estimation to improve the stability of the heart rate estimation.
[0026] The aforementioned heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition sets the sliding window length to contain at least 10 heartbeat pulses to ensure the quality of a single window, while increasing the sliding window step to reduce computing power consumption, and evenly adjusting the detection count threshold to obtain a heart rate inference result with a higher confidence rate.
[0027] Compared with the prior art, the present invention innovatively uses an improved empirical mode decomposition method to decompose and adaptively reduce noise on rPPG signals. It is not necessary to know the interference frequency band in advance, and can adaptively suppress interference on rPPG signals. At the same time, a combined analysis method of time domain and frequency domain is used for heart rate recognition, providing data such as heart rate variability, which effectively improves the effect of heart rate extraction in rPPG signals. The present invention can construct a set of heart rate recognition algorithms that can run in real time on terminals and PCs, and realize real-time remote photoplethysmography heart rate recognition on general PCs. At the same time, the present invention also has the following characteristics:
[0028] (1) Versatility
[0029] The empirical mode decomposition method is an adaptive signal decomposition algorithm that can adaptively decompose the signal into different modes, thereby distinguishing heartbeat, breathing and other high-frequency interference. Combined with the signal energy relationship, it can better find and restore the heartbeat signal.
[0030] (2) Efficiency
[0031] The original quality of the remote optical volume method signal is very poor, and the environmental interference is very strong. If only the frequency domain analysis method is used, it may be impossible to obtain accurate values due to too many missed heartbeat pulses. Therefore, the present invention marks the pulses in the time domain by sliding windows, and uses the accumulated number of detected pulses as a threshold to determine whether a pulse is a real heartbeat or interference. The heart rate statistics are then supplemented by longer frequency domain information to further improve the effect.
[0032] (3) Timeliness
[0033] The present invention can uniformly adjust the algorithm time consumption, and ensure a processing speed of more than 20FPS on a common PC (common here refers to a standard pressure CPU, without adding a graphics card or a powerful CPU or other computing unit), and ensure a processing speed of more than 15fps on a notebook with a non-standard pressure processor. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 The flow chart of video stream sampling rPPG signal is shown.
[0035] Figure 2 The figure shows how Facemesh divides the facial key points into three ROI areas.
[0036] Figure 3 The flowchart of obtaining heart rate from rPPG signal is shown.
[0037] Figure 4 The algorithm result and the reference heart rate comparison chart are shown. The horizontal axis is the sampling point number and the vertical axis is the heart rate per minute (bmp). DETAILED DESCRIPTION
[0038] The present invention is further described below in conjunction with the accompanying drawings and embodiments, but they are not intended to limit the present invention.
[0039] Embodiment: A heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition comprises the following steps:
[0040] Step 1: Take the mean value of facial ROI in a single frame of RGB video stream as the rPPG signal sampling point, and sample to obtain the rPPG signal stream; Figure 1 As shown, in this embodiment, face recognition and facial feature region feature (ROI) extraction and rPPG signal establishment are specifically as follows:
[0041] Step 1.1: Use a fast face detection model (the open source mediapipefacedetector algorithm is used in this embodiment) to quickly locate the face position in the RGB video stream, crop the external frame, and send it to the next level network;
[0042] Step 1.2: Use the face mesh network (the open source mediapipe face mesh algorithm is used in this embodiment) to perform 2D position recognition and 3D depth estimation of facial feature points, and finally obtain 11 facial feature points, including 5 on the forehead and 3 on each of the left and right cheeks, corresponding to three facial ROI areas, such as Figure 2 As shown;
[0043] Step 1.3: Use the 3D information estimated by the feature points to calculate the normal vector of the ROI area, discard the ROI area where the angle between the normal vector and the lens angle exceeds 45° to prevent interference and improve the signal-to-noise ratio, and finally perform full-pixel mean statistics on the RGB values of the retained ROI area as a sampling point of rPPG;
[0044] Step 1.4: Repeat steps 1.1 to 1.3 for each frame of the video stream, and subtract the mean signal stream of the background area from the signal stream to remove the drift caused by ambient lighting, and finally obtain the rPPG signal stream in the time domain.
[0045] Step 2: For the rPPG signal stream in step 1, use the sliding window strategy to decompose the heartbeat signal using the empirical mode decomposition algorithm in a single window, such as Figure 3 As shown, the specific steps are as follows:
[0046] The rPPG signal stream is analyzed by sliding window with 250 sampling points as the time window. The signal in each window is subjected to empirical mode decomposition (EMD). Energy statistics and FFT analysis are performed on all components. The signal with energy peak value of 0.5-3Hz is taken as the heartbeat signal, which is sent to a 30th-order elliptic filter with a cutoff frequency of 0.5-3Hz for filtering, and the remaining components are subtracted to restore the heartbeat signal, which is then sent to the next stage.
[0047] Step 3: Perform heart rate analysis in the time domain and frequency domain based on the heartbeat signal to obtain a heart rate estimate. Figure 3 As shown, the specific steps are as follows:
[0048] Step 3.1, in the heartbeat signal restored in the 250 sampling point window, take the section of 50-200 sampling points as the stable estimation section to reduce the impact of the endpoint effect caused by signal decomposition and filtering, and count the time when the heartbeat pulse occurs in the stable estimation section;
[0049] Step 3.2, repeat step 3.1 in multiple sliding windows to obtain a series of estimated values of heartbeat pulse times, count the heartbeat pulses with a time close to 0.3s for detection and counting, and set a threshold to exclude heartbeat pulses with a detection frequency of less than 3 times;
[0050] Step 3.3, between different sliding windows in step S3.2, observe the time when the peak value of the heartbeat pulse in the time overlapping part appears, set the effective detection number threshold of the heartbeat pulse peak value, and obtain the time domain heart rate statistics; also perform statistical analysis on the remaining heartbeat pulses to obtain data such as heart rate and heart rate variability, that is, obtain the average heart rate estimation in the time-frequency domain;
[0051] Step 3.4: For each sampling window, perform zero-crossing rate statistics to calculate the pulse frequency and obtain the time-domain average heart rate estimate under the window. At the same time, use FFT (Fast Fourier Transform) to calculate the maximum frequency point (main frequency) to obtain the frequency-domain average heart rate estimate, and perform weighted fusion with the time-frequency domain average heart rate estimate result obtained in step 3.3 to obtain the final average heart rate estimate to improve the stability of the heart rate estimate. After testing, the heart rate inference result of the present invention is compared with the real heart rate label. Figure 4 As shown. Figure 4 It can be seen that the algorithm result of the present invention is consistent with the reference heart rate in terms of fluctuation, which shows that the present invention has good heart rate extraction accuracy.
[0052] In addition, this embodiment also provides the deployment and real-time operation of the method of the present invention, and the specific steps are as follows:
[0053] S1: According to the difference in computing power of the terminals used, the algorithm scheduling pipeline is designed to control the face detection network to be called when initializing and losing the face, control the face mesh network to be called in each frame, and adjust the sliding window step between 1-10 sampling points to adapt to different terminals;
[0054] S2: For the algorithm pipeline of S1, use multiple processes to distribute computing power to improve the efficiency of algorithm operation.
[0055] In summary, the present invention innovatively uses an improved empirical mode decomposition method to decompose and adaptively reduce noise on rPPG signals. It is not necessary to know the interference frequency band in advance, and can adaptively suppress interference on rPPG signals. At the same time, a combined analysis method of time domain and frequency domain is used for heart rate recognition, providing data such as heart rate variability, which effectively improves the effect of heart rate extraction in rPPG signals. The present invention can construct a set of heart rate recognition algorithms that can run in real time on terminals and PCs, and realize real-time remote photoplethysmography heart rate recognition on general PCs.
Claims
1. A heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition, characterized in that: The steps include: Step 1: Take the mean value of the facial ROI of a single frame of the RGB video stream as the rPPG signal sampling point, and sample to obtain the rPPG signal stream; Step 2: For the rPPG signal stream in step 1, use the sliding window strategy to decompose the heartbeat signal using the empirical mode decomposition algorithm in a single window; Step 3: Perform heart rate analysis in time domain and frequency domain according to the heartbeat signal to obtain a heart rate estimation value; In step 2, the rPPG signal stream is analyzed by sliding window with W sampling points as the time window, the signal of each window is subjected to empirical mode decomposition, energy statistics and FFT analysis are performed on all components, and the signal with energy peak value of 0.5-3Hz is taken as the heartbeat signal, which is sent to the filter with cutoff frequency of 0.5-3Hz for filtering to obtain the heartbeat signal; In step 3, the heart rate analysis includes the following steps: Step 3.1, in the heartbeat signal restored by the W sampling point window, the head and tail parts of the sampling point window are truncated, and the remaining window is used as a stable estimation segment to reduce the impact of the endpoint effect caused by signal decomposition and filtering, and the time when the heartbeat pulse occurs is counted in the stable estimation segment; Step 3.2, repeat step S3.1 in multiple sliding windows to obtain a series of estimated values of heartbeat pulse moments, and count the heartbeat pulses with moments close to 0.3s for detection and counting; then set a detection count threshold to exclude heartbeat pulses with detection times less than the threshold, and calculate the average of the retained heartbeat pulses as the estimate of the heartbeat pulse moment; Step 3.3, between different sliding windows in step S3.2, observe the occurrence time of the heartbeat pulse peak value in the time overlapping part, set the effective detection number threshold of the heartbeat pulse peak value, and obtain the time domain heart rate statistics; Step 3.4, use fast Fourier transform to obtain the main frequency of the heartbeat pulse of a single sliding window, and obtain the median of the main frequency between the sliding windows as the frequency domain heart rate statistics; Step 3.5: Use the average of the time domain heart rate statistics and the frequency domain heart rate statistics as the final heart rate estimation to improve the stability of the heart rate estimation.
2. The heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition according to claim 1, characterized in that: Use the face detection model to infer the facial 3D feature points from a single frame of the RGB video stream and calculate the mean of the facial ROI area surrounded by the feature points.
3. The heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition according to claim 2, characterized in that: The face detection model is a face mesh network; the face mesh network performs 2D position recognition and 3D depth estimation of facial feature points to obtain 11 facial feature points, including 5 on the forehead and 3 on each of the left and right cheeks, corresponding to three facial ROI areas; then the 3D information estimated from the feature points is used to calculate the normal vector of the ROI area, and the ROI area with a smaller angle between the normal vector and the camera's line of sight is selected to count the RGB pixel mean, and random ambient light fast sampling is used to eliminate ambient light floating interference to obtain facial ROI mean statistics.
4. The heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition according to claim 3, characterized in that: By adjusting the facial tracking and feature point extraction strategies of the face mesh network, the calls to the facial tracking algorithm can be reduced.
5. The heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition according to claim 2, characterized in that: The rPPG signal stream is subtracted from the mean signal stream of the background area in the RGB video stream to remove the drift caused by ambient lighting.
6. The heart rate recognition method based on facial feature point recognition and signal empirical mode decomposition according to claim 1, characterized in that: The sliding window length is set to include at least 10 heartbeat pulses to ensure the quality of a single window. At the same time, the sliding window step is increased to reduce computing power consumption, and the detection count threshold is adjusted evenly to obtain a heart rate inference result with a higher confidence rate.
Citation Information
Patent Citations
Method and device for non-contact detecting vital sign signals
CN105662375A
Non-contact heart rate detection method based on multivariable singular spectrum analysis
CN115153473A
Heart rate estimation based on facial color variance and micro-movement
KR102215557B1
Method of heart rate estimation based on corrected image and apparatus thereof
KR102468654B1
Heart Rate Estimation Method and Apparatus, and Electronic Device Applying Same
US20220409079A1
