Non-contact heart rate detection method based on three-color channel signal fusion technology

Abstract

The invention discloses a non-contact heart rate detection method based on a three-color channel signal fusion technology, and aims to solve the problems that an existing heart rate detection method is complex in algorithm and high in calculation capability requirement, and measurement errors are caused by illumination variation and motion artifacts. According to the system, a face video is captured through a camera, a region of interest (ROI) is extracted, and non-skin regions are removed by combining skin pixel detection of a YCrCb color space. Then, rPPG signals are extracted through a three-channel signal fusion technology, signal noise reduction is carried out through signal segmentation, mean translation and discrete wavelet transform technologies, and finally the heart rate spectrum is analyzed through fast Fourier transform. Real-time, low-cost and accurate non-contact heart rate detection can be realized, the signal quality and the robustness of a measurement result are remarkably improved, and the method is suitable for various actual scenes.

Description

Technical Field

[0001] The present invention relates to the field of biomedical signal processing, and in particular to a non-contact heart rate detection method based on three-color channel signal fusion technology. Background Art

[0002] Heart rate parameters are important features that need to be considered in the healthcare field and various clinical environments, so it is of great significance to study the detection methods of heart rate. Traditional heart rate detection is achieved through the following technologies: based on photoplethysmography (PPG) and electrocardiogram pulse oximetry (ECG). Although the heart rate results obtained by these two methods are accurate, they also have limitations: they require direct contact with the human body. Therefore, this method may cause discomfort to some special groups, such as the elderly, infants, and patients with skin burns. In recent years, with the popularization of low-cost networks and mobile phone cameras, researchers have been committed to studying remote non-contact heart rate detection based on the face to overcome the shortcomings of traditional heart rate monitoring.

[0003] The principle of remote contactless heart rate detection is remote photoplethysmography (rPPG). Traditional PPG technology measures blood flow changes caused by heartbeats by placing photoelectric sensors on the skin to detect changes in the absorption or reflection of light by the blood. The rPPG technology uses a camera to capture the color changes on the human face and extract heart rate information from it. Specifically, every time the heart beats, the amount of blood changes, causing the intensity of light reflected from the skin surface to change. Although these changes are very small, they can be captured by high-speed cameras. Therefore, the heart rate information can be extracted by using a camera to capture subtle color changes in the facial area. Compared with traditional PPG technology, rPPG technology is more patient-friendly and convenient, reduces the burden on medical staff, and can also be measured at home.

[0004] In recent years, more and more complex and advanced signal processing technologies have been developed to obtain high-quality rPPG signals. Patent CN116327152B discloses a remote non-contact heart rate detection system and method, which uses the heat map method in deep learning to obtain facial features. Patent CN116458861A discloses a non-contact heart rate detection method for facial occlusion scenes, which assigns different weight values ​​to different facial areas to overcome the problem of reduced accuracy under facial occlusion. However, the above patents all use neural network models, which increases the complexity of the algorithm and places high demands on the processing power of the computer. Patent CN117678998A discloses a non-contact heart rate detection method based on adaptive projection plane and feature screening, which reduces the dimensionality of the three-dimensional RGB signal to a one-dimensional pulse wave signal, but it needs to determine the final signal from two candidate signals, the algorithm is complex, and the detection time is increased. In addition, most of the previous studies were based on stationary experimental subjects, but in daily life, the light intensity will change or the human body will have slight movements, which will reduce the robustness of the algorithm and increase the error of the measurement results.

[0005] To this end, in response to the above problems, we have developed a non-contact heart rate detection method based on three-color channel signal fusion technology, aiming to reduce the complexity of the heart rate detection algorithm and improve the robustness of the algorithm. Summary of the invention

[0006] The purpose of the present invention is to design a detection system and method for extracting heart rate signals from face videos, so as to overcome the defects of existing detection methods: complex algorithms, high requirements on computer processing power, poor algorithm robustness when there are changes in illumination and slight movements of the human body, resulting in large errors in the measurement results. The present invention adopts a method combining facial feature point recognition with skin pixel detection in the Ycrcb color space to effectively filter non-skin pixels such as hair and eyebrows in the ROI. The rPPG signal is obtained by mixing the red, green and blue three-color channels, and then the mixed signal is post-processed by signal segmentation, mean shift and discrete wavelet transform, which effectively reduces the motion artifacts caused by motion or illumination changes, reduces the complexity of the algorithm, and improves the robustness of the algorithm.

[0007] The specific implementation steps of the present invention are:

[0008] Step S1: Aim a camera at a human face to capture a video sequence of the facial area;

[0009] Step S2: using a face feature detector to identify and determine the face area of ​​the corresponding feature points from the facial video sequence as a region of interest (ROI), then converting the RGB color space of the region of interest into the YCrCb color space, identifying the skin pixels of the ROI, and removing non-skin pixels;

[0010] Step S3: extracting red, green and blue channel signals from the acquired region of interest, and then mixing the three color channel signals to obtain a pulse wave signal;

[0011] Step S4: Post-process the mixed pulse wave signal using signal segmentation and mean shift technology and discrete wavelet transform to remove obvious noise signals and obtain the final rPPG signal;

[0012] Step S5: Perform fast Fourier transform on the rPPG signal to convert the time domain signal into a frequency domain signal, and then determine the frequency value of the peak position by analyzing the power spectrum diagram to obtain the heart rate value.

[0013] Furthermore, in S2, the face feature detector is first used to divide the region of interest according to the custom feature points to narrow the range, and then the divided region of interest is converted from the RGB color space to the YCrCb color space, and the Cr value range and Cb value range of the pixel are specified. Pixels within the range are identified as skin pixels, and non-skin pixels outside the range are eliminated.

[0014] Furthermore, the three-color channel mixed signal formula in S3 is:

[0015]

[0016] Where R(t) represents the red channel signal, G(t) represents the green channel signal, B(t) represents the blue channel signal, RGB(t) represents the mixed signal of the three channels, t represents time, i represents pixel, and n represents the pixel value in ROI.

[0017] Furthermore, the signal post-processing technology in S4 includes the following:

[0018] S41. First, perform signal segmentation processing on the three-color channel mixed signal, divide a continuous time series rPPG signal into many small segments, and set the length of each small segment, requiring that each small segment must include a complete heartbeat cycle or at least a part of the heartbeat cycle, to ensure that the main components of the heart signal will not be lost due to too short segmentation;

[0019] S42. Calculate the mean of each signal segment and shift the signal segment to zero mean. The calculation formula is as follows:

[0020]

[0021] In the formula represents the mean value of the segmented signal, and n represents the number of segments;

[0022] S43. After calculating the mean value of each segment, the signal of each segment is moved to the average value level. The calculation formula is as follows:

[0023]

[0024] Where sRGB(t) is the three-channel mixed signal after mean shift;

[0025] S44. Perform discrete wavelet transform on sRGB(t), extract low-frequency components of the signal using a low-pass filter to obtain approximation coefficients; extract high-frequency components of the signal using a high-pass filter to obtain detail coefficients; and further decompose the approximation coefficients repeatedly until a predetermined scale is reached;

[0026] S45. Using the wavelet function Symlet8 as the mother function, the signal is decomposed through 5 layers, and the absolute value of the coefficient is gradually reduced using the soft threshold method to smooth the noise;

[0027] S46. After completing the threshold processing, the signal is reconstructed from the wavelet domain back to the time domain through inverse wavelet transform. The inverse wavelet transform recombines the processed approximation coefficients and detail coefficients to generate a new time domain signal.

[0028] The beneficial effects of the present invention are:

[0029] 1. The present invention uses skin segmentation technology based on YCrCb color space to detect the skin in the area of ​​interest, automatically removes non-skin pixels such as hair and eyebrows, significantly improves the signal-to-noise ratio of the skin area, and provides high-quality pixel information for subsequent steps;

[0030] 2. The present invention uses three-color channel fusion technology to extract pulse wave signals, which is different from traditional green channel signals. Signal segmentation and mean shift technology and discrete wavelet transform are used to post-process the mixed pulse wave signals, which fully considers the noise influence caused by illumination changes and motion artifacts, and significantly improves the accuracy of heart rate measurement results.

[0031] 3. The present invention provides a complete heart rate detection process, which does not require contact with the person being detected, and thus will not cause physical discomfort to the person being detected, and can provide the person being detected with a good detection experience. In addition, the present invention has low cost and fast detection speed, and can realize online real-time heart rate measurement, and can be used for measurement in daily life. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a processing flow chart of the system of the present invention;

[0033] Figure 2 This is a schematic diagram of the ROI area division effect of the present invention;

[0034] Figure 3 This is a signal diagram after the three-color channel fusion technology of the present invention;

[0035] Figure 4 It is a signal diagram after the signal post-processing technology of the present invention;

[0036] Figure 5 This is a power spectrum diagram of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0038] like Figure 1 A non-contact heart rate detection method based on three-color channel signal fusion technology in this embodiment mainly includes the following six steps:

[0039] Step S1: Use a camera with a frame rate of 30 frames per second to capture a face video. In order to obtain a face video sequence, it is necessary to ensure that the camera is aimed at the face area of ​​the face, and the sampling frequency should be at least twice the highest frequency of the original signal to ensure that the sampled signal can fully contain the information of the original signal;

[0040] Step S2: Use the 81 feature point model shape_predictor_81_face_landmarks.dat in the Dlib library to detect facial feature points, obtain facial features, and divide the region of interest. To ensure pixel continuity, face correction and motion tracking are used to minimize the relative motion between the subject and the camera. Dlib-81 face feature point detection is an improvement on Dlib-68 face feature point detection. 13 additional feature points are added to the original 68 feature points to cover the forehead area, which can accurately detect the head. Face correction can be achieved through affine transformation, using the feature point coordinates of both eyes to calculate the tilt of the face and straighten the face. Motion tracking can be achieved using the sparse optical flow Lucas-Kanade algorithm, which uses the changes in pixels in the time domain in the image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames. Figure 2 The detailed ROI area division is shown. The specific sub-steps are as follows:

[0041] Step S2.1: Use the lines between the feature points to divide the ROI area. The ROI is divided into three areas, namely the forehead area, the left cheek area, and the right cheek area. The feature points of the forehead area include: 69, 70, 71, 80, 72, 25, 24, 23, 22, 21, 20, 19, 18; the feature points of the left cheek area include: 1, 2, 3, 4, 48, 31, 28, 39; the feature points of the right cheek area include: 15, 14, 14, 12, 54, 35, 28, 42.

[0042] Step S2.2: Calculate the tilt of the face using the coordinates of the feature points of both eyes and straighten the face. The calculation formula for the tilt angle α is as follows:

[0043]

[0044] Where L(x,y) represents the coordinates of the feature points of the left eye, R(x,y) represents the coordinates of the feature points of the right eye, and s i (x,y) represents the coordinates of the i-th feature point.

[0045] Step S2.3: The sparse optical flow Lucas-Kanade algorithm is implemented by the cv2.calcOpticalFlowPyrLK() function of OpenCV. The optical flow is calculated through the highest level of the image pyramid, and the motion estimation result is used as the starting point of the next level of the pyramid. The whole process is repeated until the bottom level of the pyramid is reached. In this way, large movements can be simplified to small movements to the maximum extent, and small movements can be eliminated, thereby achieving tracking of larger and faster movements.

[0046] Step S3: After obtaining the three ROI regions through step S2, the ROI regions are converted from the RGB color space to the YCrCb color space, the skin within the ROI is detected, and non-skin pixels such as hair and eyebrows are automatically removed. The Cr value range of the specified pixel is [77, 125], and the Cb value range is [137, 170].

[0047] Step S4: Calculate the pixel mean of the red, green and blue color channels in the ROI. The pixel mean of each channel is a time-dependent signal. The red channel is set to R(t), the green channel is set to G(t), and the blue channel is set to B(t). Assuming that light intensity fluctuations have the same effect on all color channels, the green channel is most sensitive to changes in blood volume. The addition of the red channel helps to balance the errors caused by changes in illumination. The blue channel provides supplementary information and can capture more changes in skin-reflected illumination. Therefore, the stability of the rPPG signal can be enhanced by combining the ratio of the green channel to the red and blue channels. Specifically, the rPPG signal is formed by calculating the ratio of the green channel to the red and blue channels in turn, and adding these two ratios. The calculation formula is as follows:

[0048]

[0049] Where RGB(t) represents the signal after the three channels are mixed, t represents time, i represents pixel, and n represents the pixel value in ROI.

[0050] Step S5: Since the rPPG signal is a continuous time series, the effects of facial movement or environmental interference are usually short-term local artifacts. If the entire signal sequence is processed directly, the effects of the artifacts will spread to the entire signal, resulting in a decrease in signal quality. Therefore, it is necessary to use signal segmentation and mean shift technology and discrete wavelet transform to post-process the mixed pulse wave signal, remove obvious noise signals, and obtain the final rPPG signal. The specific sub-steps are as follows:

[0051] Step S5.1: Divide the signal into many small segments, each of which is 0.5 seconds long. Thus, a 30-second video clip will be divided into 60 signal segments. The reason for selecting 0.5 seconds as the segmentation unit is that the normal heart rate of an adult ranges from about 60 to 100 beats per minute, i.e., 1 to 2 heartbeats per second. Within the 0.5-second time period, a complete heartbeat cycle or at least a portion of a heartbeat cycle is included, thus ensuring that the main components of the heart signal will not be lost due to too short a segment.

[0052] Step S5.2: Calculate the mean of each signal segment and shift the signal segment to zero mean. The calculation formula is as follows:

[0053]

[0054] In the formula It represents the mean value of the segmented signal, and n represents the number of segments.

[0055] Step S5.3: After calculating the mean of each segment, the signal of each segment is moved to the average level. The calculation formula is as follows:

[0056]

[0057] Where sRGB(t) is the three-channel mixed signal after mean shift.

[0058] Step S5.4: Perform discrete wavelet transform on sRGB(t), use a low-pass filter to extract the low-frequency components of the signal to obtain the approximation coefficient; use a high-pass filter to extract the high-frequency components of the signal to obtain the detail coefficient; and then repeatedly decompose the approximation coefficient until it reaches a predetermined scale. After the αth layer of decomposition, the expressions of the approximation coefficient and detail coefficient of the input signal are:

[0059]

[0060] Where sRGB α,L [t] represents the approximation coefficient of the three-channel mixed signal after the α layer decomposition and mean shift, which contains the heart rate information we need, sRGB α,H [t] represents the detail coefficient of the mean-shifted three-channel mixed signal after α-layer decomposition, which contains noise information.

[0061] Step S5.5: Use the wavelet function Symlet8 as the mother function, decompose the signal through 5 layers, and use the soft threshold method to gradually reduce the absolute value of the coefficient to smooth the noise. The soft threshold processing calculation formula is as follows:

[0062]

[0063] Where sRGB′ H is the updated detail coefficient.

[0064] Step S5.6: After completing the threshold processing, the signal is reconstructed from the wavelet domain back to the time domain through the inverse wavelet transform. The inverse wavelet transform recombines the processed approximation coefficients and detail coefficients to generate a new time domain signal, which has removed most of the high-frequency noise and retained the main heart rate components. The reconstructed signal is smoother in the time domain, and the heart rate fluctuation in the signal is more significant. Figure 3 and Figure 4 The rPPG signal after three-channel fusion and the final signal after signal post-processing are shown respectively.

[0065] Step S6: Use Fourier transform to convert the time domain signal into a frequency domain signal, find the frequency value corresponding to the maximum peak in the power spectrum, and multiply the frequency value by 60 to obtain the corresponding heart rate value. Figure 5 As shown, the corresponding heart rate value is approximately 1.2 times 60, that is, the heart rate value is 72 beats / minute.

[0066] The present invention does not elaborate on Fourier transform in detail, which is common knowledge in the art.

[0067] The examples of the present invention are described above in conjunction with the accompanying drawings. Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Anyone familiar with this technology can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be based on the definition of the claims.

Claims

1. A non-contact heart rate detection method based on three-color channel signal fusion technology, characterized in that: The specific implementation includes the following steps: S1: Aim the camera at the face to capture a video sequence of the facial area; S2: using a face feature detector to identify and determine the face area of ​​the corresponding feature points from the face video sequence as the region of interest (ROI), then converting the RGB color space of the region of interest into the YCrCb color space, identifying the skin pixels of the ROI, and removing non-skin pixels; S3: extracting red, green and blue channel signals from the region of interest, and then mixing the three color channel signals to obtain a pulse wave signal; S4: Post-process the mixed pulse wave signal using signal segmentation and mean shift technology and discrete wavelet transform to remove obvious noise signals and obtain the final rPPG signal; S5: Perform fast Fourier transform on the rPPG signal to convert the time domain signal into a frequency domain signal, and then determine the frequency value of the peak position by analyzing the power spectrum diagram to obtain the heart rate value.

2. According to the non-contact heart rate detection method based on three-color channel signal fusion technology of claim 1, the step of obtaining the region of interest S2 is characterized by: It is necessary to first use the face feature detector to divide the region of interest according to the custom feature points to narrow the range, and then convert the divided region of interest from the RGB color space to the YCrCb color space, specify the Cr value range and Cb value range of the pixel, identify the pixels within the range as skin pixels, and eliminate non-skin pixels outside the range.

3. According to the non-contact heart rate detection method based on three-color channel signal fusion technology of claim 1, for the step of three-color channel signal mixing S3, it is characterized in that The calculation method for mixing the three color channel signals is as follows: Where RGB(t) represents the signal after three-channel mixing, R(t) represents the red channel signal, G(t) represents the green channel signal, and B(t) represents the blue channel signal.

4. According to the non-contact heart rate detection method based on three-color channel signal fusion technology of claim 1, the mixed signal post-processing step S4 is characterized in that: The specific steps include: For signal segmentation, the signal needs to be divided into many small segments of equal length. Each small segment must contain a complete heartbeat cycle or at least a part of the heartbeat cycle to ensure that the main components of the heart signal will not be lost due to too short segments. The signal segmentation formula is: In the formula represents the mean value of the segmented signal, and n represents the number of segments; For mean shift, the signal of each small segment needs to be moved to the average level after the mean calculation. The mean shift formula is: Where sRGB(t) represents the three-channel mixed signal after mean shift; For discrete wavelet transform, a low-pass filter should be used to extract the low-frequency components of the signal to obtain the approximation coefficients; a high-pass filter should be used to extract the high-frequency components of the signal to obtain the detail coefficients; the approximation coefficients are further decomposed repeatedly until the predetermined scale is reached; the signal is then decomposed through 5 layers, using the wavelet function Symlet8 as the mother function, and the soft threshold method is used to gradually reduce the absolute value of the coefficient to smooth the noise, and finally the signal is reconstructed from the wavelet domain back to the time domain through the inverse wavelet transform.

Images