Head rotation motion adaptive non-contact robust heart rate detection method

Abstract

The present application belongs to the field of biological medical monitoring and computer vision, and discloses a head rotation motion adaptive non-contact robust heart rate detection method. The steps include: collecting a face video through an imaging device and detecting facial feature points in real time, generating a pulse wave original signal frame by frame based on a face region of interest selected based on specific face landmarks; using a detrended filter and a wavelet filter algorithm to process the original signal, selecting a power spectrum specific range peak-to-peak value to estimate the heart rate; finally, calculating the yaw and pitch Euler angles of the head in the motion of the subject by using the two-dimensional and three-dimensional feature points of the face, using the angle measurement as a new signal quality index to construct an adaptive Kalman filter and provide a single robust heart rate estimation. The present application can overcome the motion artifact noise caused by the non-rigid motion of the head rotation of the subject, effectively improve the accuracy of heart rate detection in the real scene of spontaneous motion, and expand the use scene of the imaging photoelectric plethysmography technology in the field of video health monitoring.

Description

Technical Field

[0001] This invention belongs to the fields of biomedical monitoring and computer vision, and specifically relates to a non-contact robust heart rate detection method that adapts to head rotation movement. Background Technology

[0002] Cardiovascular and cerebrovascular diseases have become major threats to human health. Based on the need for early detection, early treatment and active intervention in the prevention and control of cardiovascular and cerebrovascular diseases [1], it is necessary to seek a measurement technology and method that is simple to operate, accurate in results, has good repeatability and can be applied to daily detection of large-scale populations. Imaging photoplethysmography (IPPG) is a non-contact physiological parameter detection technology based on imaging equipment. It overcomes the defects of traditional contact technology and is gradually being applied to fields such as remote medical monitoring. IPPG heart rate measurement technology based on face video uses imaging equipment to collect video information containing the measured part, records the pulse signal, i.e. the change in light intensity caused by the change in blood volume, in the form of video images, extracts the pulse wave signal through video image processing, and finally extracts the heart rate signal through the analysis of pulse wave characteristics [2].

[0003] Significant progress has been made in the research of heart rate measurement technology based on IPPG, enabling accurate heart rate measurement in both resting and moving subjects. Hemoglobin's light absorption varies across the entire spectrum, with the green channel exhibiting a strong pulse signal intensity. Combining color channels is a common method to address signal distortion. One type of method is based on linear channel combination using data: for example, Principal Component Analysis (PCA) defines three independent linear combinations of color channels and uses Blind Source Separation (BSS) to separate the pulse wave signal from noise. This method assumes that the components carrying the pulse signal are unknown a priori and that the pulse signal exhibits the strongest periodicity. However, since motion artifacts also exhibit strong periodicity, this method is unsuitable for motion scenarios. Another type of combination method is based on channel combination using prior knowledge, a representative example being the chrominance (CHROM) method. This method generates a chrominance signal by weighting and combining the three channel signals based on different empirical inferences. Converting the RGB color space to other forms can better separate photoplethysmography effects from distortion. In addition, there are some image processing-based methods to overcome motion artifacts, such as image registration, ROI spatial averaging, ROI tracking, anti-motion spectrum peak tracking based on time-frequency analysis, and heart rate detection based on adaptive zoom system suitable for long-distance movement [3].

[0004] However, the main limitation of existing methods is that they are only applicable to non-rigid motion scenarios such as breathing and slight head movements. They cannot accurately estimate heart rate when subjects perform spontaneous movements such as rotating their heads, speaking, and walking. From the perspective of accurately measuring heart rate in subjects' spontaneous movement states in real-world scenarios, heart rate monitoring technology based on IPPG technology urgently needs to have the ability to overcome artifacts introduced by head movements.

[0005] The following documents are technical background information related to this invention:

[0006] [1]Franco, M., Cooper, RS, Bilal, U., & Fuster, V. (2011). Challenges and opportunities for cardiovascular disease prevention. American Journal of Medicine, 124(2), 95-102.

[0007] [2]Sun, Y. ,&Thakor, N. . (2016). Photoplethysmography revisited: from contact to noncontact, from point to imaging. IEEE Transactions onBiomedical Engineering, 63(3), 463-477.

[0008] [3] Sebastian, Z. , Alexander, T. , Daniel, W. ,&Hagen, M. . (2018). Cardiovascular assessment by imaging photoplethysmography – a review. Biomedical Engineering. Summary of the Invention

[0009] To address the aforementioned problems in existing technologies—namely, the main limitation of existing methods being that they are only applicable to non-rigid motion scenarios such as breathing and slight head movements, and cannot accurately estimate heart rate when the subject performs spontaneous movements such as head rotation, speaking, and walking—this invention provides a non-contact robust heart rate detection method based on facial images and adaptive head rotation motion. This non-contact heart rate measurement method includes:

[0010] Step 1: Acquire facial videos of the subject using imaging equipment;

[0011] Step 2: Real-time detection of 68 facial feature points; generation of raw pulse wave signals frame-by-frame based on regions of interest selected from specific facial landmarks.

[0012] Step 2-1: First, the face detector is used to detect the face bounding box, and then CLNF is used to locate 68 feature points of the face in real time;

[0013] Step 2-2: Select the cheek and nose regions of a face with rich capillary distribution from the 1st, 17th, 42nd and 47th points of the 68 facial landmarks as ROIs, and calculate the mean gray value of the ROI region. Then, connect the mean values ​​of each frame according to the video time sequence to generate the original IPPG pulse wave signal.

[0014] Step 3: Process the original signal using detrending filtering and wavelet filtering algorithms, and select peak-to-peak values ​​within a specific range of the power spectrum to estimate the heart rate;

[0015] Step 4: Combine the two-dimensional and three-dimensional feature points of the face to calculate the Euler angle of the subject's head posture during movement. Based on the absolute value of the angle, a new signal quality index is obtained to reflect the signal quality. The new signal quality index is used to estimate the adaptive noise covariance.

[0016] Step 5: The estimated adaptive noise covariance modulates the Kalman gain, thereby constructing a head rotation motion adaptive filter. This filter dynamically filters out artifacts in segments with noise introduced by head rotation motion based on changes in the rotation angle and provides a single robust heart rate estimate.

[0017] Preferably, step 4 specifically includes the following three steps:

[0018] Step 4-1: First, the facial feature point coordinates obtained by the CLNF algorithm are back-projected onto the three-dimensional face model to obtain the three-dimensional face landmark point coordinates. Then, the Euler angles of the head pose are calculated by solving the perspective pose through the camera calibration parameters and the least squares method. This invention only considers pitch and yaw angles.

[0019] Step 4-2: Normalize and invert the absolute values ​​of the obtained pitch and yaw angles respectively, and then use their product as the new signal quality index. ;

[0020] Step 4-3: Utilizing a novel signal quality index Adjusting the measurement noise covariance estimate caused by motion, we construct the head rotation adaptive noise covariance.

[0021] Preferably, step 5 specifically includes the following three steps:

[0022] Step 5-1: Adjust the Kalman gain coefficient by adaptively measuring the noise covariance based on head rotation;

[0023] Step 5-2: Combining the Kalman gain coefficient, the heart rate measurement obtained in Step 3, and the prior probability at the current moment, the posterior distribution is changed according to the dynamic change of the head posture angle value to achieve adaptive modulation of the newly arrived data.

[0024] Step 5-3: Update the posterior probability of the current state to the antecedent probability of the subsequent state.

[0025] Compared with existing technologies, the beneficial effects of this invention are reflected in:

[0026] This paper uses head posture angle estimation as the primary distortion index of heart rate signals, employing it as a novel signal quality index to weight heart rate estimates from IPPG recordings, thus achieving reliable heart rate detection. Compared to traditional heart rate detection algorithms adapted to motion scenarios, the proposed method effectively suppresses non-rigid motion artifacts caused by head rotation. This method allows subjects to move spontaneously in medical testing scenarios, facilitating effective monitoring of the health status of subjects during movement in real-world settings and further advancing IPPG technology in the fields of physiological signal detection and video health monitoring. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the non-contact robust heart rate detection method based on head rotation motion adaptation of facial images, as per the present invention. Detailed Implementation

[0028] To make the objectives, advantages, and features of this invention clearer, the following detailed description, in conjunction with the accompanying drawings and specific embodiments, provides a non-contact robust heart rate detection method based on face image-adaptive head rotation motion. It should be noted that the accompanying drawings are in a very simplified form, used only to facilitate and clarify the illustration of the embodiments of this invention. The described embodiments are merely some, not all, of the embodiments of this invention. Furthermore, all other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] A non-contact robust heart rate detection method based on head rotation motion adaptation using facial images is illustrated in the flowchart below. Figure 1 As shown.

[0030] Step 1: Acquire facial videos of the subject using imaging equipment;

[0031] Step 2: Real-time facial feature points are located using CLNF. The Viola-Jones algorithm is used to initialize the position of the face, and 68 facial landmarks are detected. Key point distribution models are trained for the eyes, mouth, and eyebrows respectively. The 1st, 17th, 42nd, and 47th points of the 68 facial landmarks are selected as the cheek and nose regions of faces with rich capillary distribution as ROIs to avoid the influence of blinking and hair occlusion on the gray value of the ROI. The mean gray value of the ROI region is calculated, and then the mean values ​​of each frame are concatenated according to the video time sequence to generate the original IPPG pulse wave signal.

[0032] Step 3: Use a detrending filtering algorithm to remove the trend signal from the IPPG pulse wave signal and retain its fluctuation signal; use a wavelet filtering algorithm to eliminate high-frequency noise; select a specific range of power spectrum as the human pulse range (0.75Hz-4.5Hz), and use the frequency of the peak-to-peak value within this range to estimate the heart rate;

[0033] Step 4: Combine the two-dimensional and three-dimensional feature points of the face to calculate the Euler angle of the subject's head posture during movement. Based on the absolute value of the angle, a new signal quality index is obtained to reflect the signal quality. The new signal quality index is used to estimate the adaptive noise covariance.

[0034] Step 4-1: First, the facial feature point coordinates obtained by the CLNF algorithm are back-projected onto the three-dimensional face model to obtain the three-dimensional face landmark point coordinates. Then, the Euler angles of the head pose are calculated by solving the perspective pose through the camera calibration parameters and the least squares method. This invention only considers pitch and yaw angles.

[0035] Step 4-2: Normalize and invert the absolute values ​​of the obtained pitch and yaw angles respectively, and then use their product as the new signal quality index. ;

[0036] Step 4-3: Utilizing a novel signal quality index Adjusting the measurement noise covariance estimate caused by motion, we construct the head rotation adaptive noise covariance:

[0037]

[0038] in, Take the empirical value of 0.1.

[0039] Step 5: The estimated adaptive noise covariance modulates the Kalman gain, thereby constructing a head rotation motion adaptive filter. This filter dynamically filters out artifacts in segments with noise introduced by head rotation motion based on changes in the rotation angle, and provides a single robust heart rate estimate.

[0040] Step 5-1: Assume the first The average heart rate is Covariance is normal distribution That is, the first The covariance of the posterior estimate at time is , No. The covariance of the prior estimate at time is The Kalman gain coefficient is adjusted by adaptively measuring the noise covariance based on head rotation.

[0041]

[0042] Step 5-2: Combining the Kalman gain coefficient, the heart rate measurement obtained in Step 3, and the prior probability at the current moment, the posterior distribution is changed according to the dynamic change of the head posture angle value to achieve adaptive modulation of the newly arrived data.

[0043] Step 5-3: Update the posterior probability of the current state to the antecedent probability of the subsequent state.

[0044] An adaptive filter for head rotation motion was implemented to dynamically filter out artifacts in segments with noise introduced by head rotation motion based on changes in the rotation angle. Finally, a single, robust output of an accurate heart rate estimate was achieved.

[0045] The above embodiments are only used to illustrate and not limit the technical solutions of the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention without departing from the spirit and scope of the present invention. Any modifications or partial substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A non-contact robust heart rate detection method adapted to head rotation motion, the method comprising the following steps: S100. Acquire facial videos of the subjects using imaging equipment; S200: Real-time detection of 68 facial feature points; Based on specific facial landmarks, select the Region of Interest (ROI) to generate raw pulse wave signals frame by frame. S300 uses detrending filtering and wavelet filtering algorithms to process the original signal and selects peak-to-peak values ​​within a specific range of the power spectrum to estimate the heart rate. S400: Combine two-dimensional and three-dimensional facial feature points to calculate the Euler angle of the subject's head posture during movement. Based on the absolute value of the angle, a new signal quality index is obtained to reflect the signal quality. The new signal quality index is used to estimate the adaptive noise covariance. S500, estimated adaptive noise covariance modulates Kalman gain, thereby constructing a head rotation motion adaptive filter, which dynamically filters out artifacts in segments with noise introduced by head rotation motion based on changes in rotation angle and provides a single robust heart rate estimate. in, Step S400 specifically includes the following steps: S401. First, the facial feature point coordinates are obtained by using the Constrained Local Neural Field (CLNF) algorithm. The two-dimensional point coordinates are then projected back onto the three-dimensional face model to obtain the three-dimensional face landmark point coordinates. Then, the Euler angles of the head pose are calculated by solving the perspective pose using camera calibration parameters and the least squares method. S402. Normalize and invert the absolute values ​​of the obtained pitch and yaw angles respectively, and then use their product as a new signal quality index. ; S403, Utilizing a new signal quality index Adjusting the measurement noise covariance estimate caused by motion, we construct the head rotation adaptive noise covariance: in, Take 0.1 experience points; Specifically, step S500 includes the following steps: S501, Assuming the... The average heart rate is Covariance is normal distribution That is, the first The covariance of the posterior estimate at time is , No. The covariance of the prior estimate at time is The Kalman gain coefficient is adjusted by adaptively measuring the noise covariance based on head rotation. S502. Combining the Kalman gain coefficient, the heart rate measurement obtained from step S300, and the prior probability at the current moment, the posterior distribution is changed according to the dynamic change of the head posture angle value to achieve adaptive modulation of the newly arrived data. S503. Update the posterior probability of the current state to the antecedent probability of the subsequent state.

2. The non-contact robust heart rate detection method adapted to head rotation movement according to claim 1, characterized in that: Heart rate measurement can be achieved using imaging devices commonly used in daily life, such as webcams or mobile phone cameras.

3. The non-contact robust heart rate detection method adapted to head rotation motion according to claim 1, characterized in that: The steps in step S200 include the following steps: First, use CLNF to locate 68 feature points of the face in real time; select the cheek and nose areas of the face with rich capillary distribution as ROIs for the 1st, 17th, 42nd and 47th points of the 68 facial landmarks, so as to avoid the influence of blinking and hair on the forehead of the tested area on the gray value of the ROI. The mean grayscale value of the ROI region is calculated, and then the mean values ​​of each frame are concatenated according to the video time sequence to generate the original IPPG pulse wave signal.

4. The non-contact robust heart rate detection method adapted to head rotation movement according to claim 3, characterized in that: The CLNF algorithm model initializes the face position using the Viola-Jones algorithm and detects 68 facial landmarks. It trains key point distribution models for the eyes, mouth, and eyebrows separately, and then fuses these models with the landmarks to reduce the false detection rate of facial landmarks.

5. The non-contact robust heart rate detection method adapted to head rotation movement according to claim 1, characterized in that: The adaptive measurement noise covariance is the artifact introduced by the head rotation motion during the measurement process.

6. The non-contact robust heart rate detection method adapted to head rotation motion according to claim 1, characterized in that: The prior probability at this moment is the prediction of the true value before measuring the heart rate. After the filtering work at the previous moment is completed, the posterior probability at the previous moment is updated to the prior probability of the current state.

7. The non-contact robust heart rate detection method adapted to head rotation movement according to claim 1, characterized in that: The Kalman gain is a core step in the Kalman filtering process. When artifacts occur, the estimated measurement noise covariance increases, significantly reducing the Kalman gain. At this point, the confidence of the measurement data decreases, and the predicted value at that moment is trusted more.

8. The non-contact robust heart rate detection method adapted to head rotation movement according to claim 1, characterized in that: The detrending filtering algorithm removes the trend signal from the IPPG pulse wave signal and retains its fluctuation signal; the wavelet filtering algorithm eliminates high-frequency noise; the power spectrum is within a specific range of the human pulse range (0.75Hz-4.5Hz), and the frequency of the peak-to-peak value within this range is selected to estimate the heart rate.

9. A heart rate detection method based on head rotation motion adaptively using facial video, mainly applied in daily non-contact heart rate measurement systems, utilizing webcams or mobile phone cameras to capture video of areas containing faces, achieving accurate heart rate measurement during movement, characterized in that... The method includes a video acquisition section, a face detection section, a head pose calculation section, a time-domain signal acquisition section, a time-domain signal processing section, a heart rate calculation section, and a head rotation motion adaptive filter section. The video acquisition section uses a camera to acquire a color video image containing multiple face regions or selects a local video file. The face detection section uses the CLNF algorithm to train eye, mouth, and eyebrow models respectively, achieving accurate localization of face representation points during head rotation motion and back-projecting two-dimensional marker points to obtain three-dimensional feature point localization. The head pose calculation section uses the localization of two-dimensional and three-dimensional feature points to calculate the head pose of each frame. The head Euler angle value; the time-domain signal acquisition part is used to divide the cheek region from the ROI region of each frame, and calculate the gray-scale mean of the cheek region as the original pulse wave signal of the image of that frame, and generate a time-domain signal X(t); the time-domain signal processing part is used to suppress noise in the obtained time-domain signal X(t) to obtain the processed time-domain signal; the heart rate calculation part is used to perform spectral analysis on the time-domain signal value and generate a spectrum diagram, and extract the peak frequency in the specified frequency band in the spectrum diagram for heart rate calculation; the head rotation motion adaptive filter part first uses the Constrained Local Neural Field (CLNF) algorithm to obtain the facial feature point coordinates, projects the two-dimensional point coordinates back onto the three-dimensional face model to obtain the three-dimensional face landmark point coordinates, and then calculates the head pose Euler angle by solving the perspective pose through camera calibration parameters and least squares method. The absolute values ​​of the obtained pitch and yaw angles are normalized and inverted respectively, and then the product of the two is used as a new signal quality index. Utilizing a new signal quality index Adjusting the measurement noise covariance estimate caused by motion, we construct the head rotation adaptive noise covariance: in, Take 0.1 experience points; An adaptive head rotation motion filter is constructed by adjusting the Kalman gain using the estimated adaptive noise covariance: assuming the first... The average heart rate is Covariance is normal distribution That is, the first The covariance of the posterior estimate at time is , No. The covariance of the prior estimate at time is The Kalman gain coefficient is adjusted by adaptively measuring the noise covariance based on head rotation. By combining the Kalman gain coefficient, the obtained heart rate measurement value, and the prior probability at the current moment, the posterior distribution of the head posture angle value is changed according to the dynamic change of the head posture angle value to achieve adaptive modulation of the newly arrived data. The posterior probability of the current state is updated to the antecedent probability of the subsequent state. Based on the change of rotation angle, artifacts are dynamically filtered out in segments with noise introduced by head rotation movement. Finally, a single robust and accurate heart rate estimate is output.

Images