Human side face non-contact heart rate detection method, system and device based on rPPG and medium

Abstract

The invention discloses a human side face non-contact heart rate detection method, system and device based on rPPG and a medium, and the method comprises the steps: capturing the side face of a person through a camera, carrying out the key point detection of the side face of the person through Mediape, and obtaining all key points of the side face of the person; the method comprises the following steps: firstly, determining an ROI (region of interest) for heart rate detection, extracting signals of a GREEN channel and an RED channel in an RGB image from image data in the ROI through an rPPG algorithm, weighting and combining the signals as bvp (blood volume pulse) signals, processing the bvp signals to obtain heart rate information of a person, and displaying the heart rate information on an interface. The non-contact measurement of the heart rate of a person is realized by detecting the side face of the person; the system, the device and the medium realize human side face non-contact heart rate detection based on the rPPG human side face non-contact heart rate detection method. Under the condition that the front face of the human face is possibly shielded or the angle is not beneficial to detection, the heart rate of the person can still be detected in real time, the method has the advantages of being convenient, fast, high in adaptability and high in accuracy, and meanwhile inconvenience and infection risks possibly brought by traditional contact type detection are avoided.

Description

Technical Field

[0001] The present invention relates to the field of biomedical detection technology, and in particular to a method, system, device and medium for contactless heart rate detection of a person's side face based on rPPG. Background Art

[0002] Heart rate is one of the key indicators that reflect the physiological state of the human body. It plays a vital role in many fields such as medical diagnosis, health monitoring, sports training, etc. Traditional heart rate detection methods mainly include electrocardiogram (ECG), stethoscope measurement, etc. Although ECG detection is accurate, it requires electrodes to be directly attached to human skin and connected to the detection instrument through a complex lead system. This contact measurement method may cause inconvenience to the person being tested in some cases. For example, for burn patients, people with skin allergies, or individuals who need to monitor their heart rate continuously for a long time (such as patients in intensive care units), the sticking of electrodes may cause skin discomfort or even damage. Stethoscope measurement requires professional medical staff to operate, and is greatly affected by environmental noise. At the same time, it cannot achieve long-term continuous and automatic heart rate monitoring.

[0003] With the development of science and technology, non-contact heart rate detection technology has gradually become a research hotspot. Among them, the technology based on the principle of photoplethysmography (PPG) has received widespread attention. Traditional PPG measures changes in light absorption by contacting light-emitting diodes (LEDs) and photodetectors with the surface of the skin to obtain heart rate information. However, this method still requires contact with the skin and has certain limitations. On this basis, remote photoplethysmography (rPPG) came into being. It uses a camera to collect image information of the human skin area and extracts heart rate signals by analyzing changes in skin color without direct contact with the human body. This method has greatly improved convenience and comfort, and can be widely used in scenarios such as home health monitoring and telemedicine.

[0004] At present, non-contact heart rate detection based on rPPG is mostly concentrated on the front area of ​​the face. This is because the front of the face is relatively flat, which is convenient for image acquisition and analysis, and the distribution of blood vessels under the skin of the face is obviously related to heart rate changes. However, when detecting the front of the face, it is easily affected by factors such as changes in facial expressions, head movements, and interference from ambient light. For example, when a person speaks, smiles, or frowns, the movement of facial muscles may change the optical properties of the skin, thereby generating noise interference to the heart rate detection signal. The side face of a person provides new possibilities for non-contact heart rate detection. It has the advantages of relatively stable blood vessel distribution and being less affected by expressions, creating conditions for more accurate heart rate detection.

[0005] Patent application CN202410600014.0 uses the CardiA2Net method to combine a self-attention convolutional hybrid network (ACmix) network with an attention-based long short-term memory (ALSTM) network for heart rate detection. The defect is that this method is only for the front face, and the number of facial feature points is too small, the positioning ROI is not accurate, and heart rate detection of the side face cannot be realized. Summary of the invention

[0006] In order to overcome the defects of the above-mentioned prior art, the purpose of the present invention is to provide a method, system, device and medium for contactless heart rate detection of a person's side face based on rPPG, which captures the side face of a person through a camera, detects key points of the side face of a person through Mediapipe, and obtains all key points of the side face of a person; then determines the ROI (region of interest) for heart rate detection, extracts the signals of the GREEN channel and the RED channel in the RGB image from the image data in the ROI through the rPPG algorithm, and weightedly combines them as a bvp (blood volume pulse) signal, processes the bvp signal to obtain the person's heart rate information, and displays it on the interface, thereby realizing contactless measurement of the person's heart rate by detecting the side face of a person; the present invention can still detect the person's heart rate in real time when the front of the person's face may be blocked or the angle is not conducive to detection, and has the advantages of convenience, fastness, strong adaptability and high accuracy.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A method for detecting a person's side face contactless heart rate based on rPPG, specifically comprising the following steps:

[0009] Step 1, collecting a side face image of a person through a camera, wherein the collected side face image is clear and complete;

[0010] Step 2, performing regional positioning on the side face image acquired in step 1 to determine a region of interest (ROI) for heart rate detection;

[0011] Step 3, using the rPPG algorithm to calculate the instantaneous heart rate, confidence level, and weighted sum of the final heart rate for each ROI;

[0012] Step 4: Display the image captured by the camera in step 1 and the instantaneous heart rate, confidence level, and weighted sum of the final heart rate of each ROI calculated in step 3 on the interface.

[0013] When collecting the side face image of a person in step 1, natural light is used for illumination, and the lighting condition is stable and can penetrate the surface of the skin to enhance the ability to capture blood flow signals.

[0014] The specific method of step 2 is:

[0015] All key points of the side face of a person are obtained through the visualization tool Mediapipe, that is, the coordinates of each part of the side face are accurately located, and then the ROI for heart rate detection is selected.

[0016] The specific method of step 3 is:

[0017] Step 3.1, read the image data in the ROI determined in step 2; obtain the RGB channel histogram of each ROI; subtract the signals of the green (GREEN) channel and the red (RED) channel in the RGB channel histogram, and obtain the absolute value of the difference between the two as the BVP signal; filter out the deviation data in the BVP signal that is greater than the preset deviation, and then use a bandpass filter to filter to obtain a filtered signal; perform a fast Fourier transform FFT on the obtained filtered signal to obtain the filtered signal spectrum, and then perform peak detection to obtain the instantaneous heart rate of each ROI;

[0018] Step 3.2, use the formula BPM(n)=α·BPM(n-1)+1-αBPMSig(n) to weighted update the instantaneous heart rate obtained in step 3.1, and set α=0.95 to ensure that the heart rate results of each ROI will not jump;

[0019] Step 3.3, compare the peak value of the bvp signal of each ROI obtained in step 3.1 with the filtered signal, and use the function to calculate the mean square error (MSE) of the spectrum to represent the degree of spectrum leakage, recorded as leakage, and use confidence i =1 / 1+leakage calculates the BPM confidence of each ROI; based on the confidence and the instantaneous heart rate that will not jump calculated in step 3.2, the final heart rate result is calculated by weighted summation using the following formula:

[0020]

[0021] A human side face contactless heart rate detection system based on rPPG, comprising:

[0022] The image acquisition module is used in step 1 to capture the side face images with a camera;

[0023] The ROI confirmation module is used in step 2 to perform profile detection on the collected profiles and identify all key points of the profiles to determine the ROI area;

[0024] The signal processing module is used in step 3 to extract and process the signal of the ROI determined in step 2 to calculate the heart rate.

[0025] A human side face contactless heart rate detection device based on rPPG, comprising:

[0026] Memory for storing computer programs;

[0027] A processor is used to implement the rPPG-based non-contact heart rate detection method for the side face of a person described in steps 1 to 4 when executing the computer program.

[0028] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it can perform rPPG-based side face contactless heart rate detection based on the rPPG-based side face contactless heart rate detection method described in steps 1 to 4.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. The present invention realizes contactless heart rate detection through the side of the face, avoiding the inconvenience and discomfort of the traditional contact detection method. The user does not need to wear any equipment and can be tested in a natural state. It is especially suitable for special groups such as the elderly, infants, and people with sensitive skin.

[0031] 2. Compared with the existing non-contact detection methods that mainly detect the front of the face, the side face detection function of the present invention has advantages in more scenarios. For example, when the person being tested is resting on his side or turning his head, the heart rate can still be accurately detected, which expands the scope of application of the detection.

[0032] 3. The present invention can effectively extract the BVP signal of ROI through advanced rPPG algorithm, and obtain the instantaneous heart rate of each ROI by peak detection after filtering and Fourier transform; weighted update of instantaneous heart rate increases the detection accuracy; based on the confidence and the instantaneous heart rate that will not jump, the final heart rate result is calculated by weighted summation, with efficient processing and high accuracy; at the same time, the system has anti-interference ability, and can work stably under natural lighting conditions and a certain degree of environmental interference, providing reliable technical support for long-term and continuous heart rate monitoring.

[0033] 4. The present invention only needs one camera to realize heart rate monitoring, which has low requirements on equipment, reduces the application threshold and expands the application scope. It is suitable for a variety of scenarios, such as family health management, remote medical monitoring, online education platform, etc.

[0034] In summary, the user of the present invention does not need to wear any equipment and can be tested in a natural state, which has advantages in more scenarios. At the same time, the present invention uses an advanced rPPG algorithm, which has efficient processing, strong anti-interference ability and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1It is a scatter plot of the simulation experiment results of the present invention.

[0036] Figure 2 Flow chart of calculating human heart rate according to the present invention.

[0037] Figure 3 The figure is a flow chart of the rPPG algorithm for calculating the human heart rate according to the present invention. DETAILED DESCRIPTION

[0038] See also Figure 2 , Figure 3 , a non-contact heart rate detection method for a person's side face based on rPPG, specifically comprising the following steps:

[0039] Step 1: Capture a side face image of a person; use a high-resolution, high-frame-rate optical imaging device, such as a camera, to capture a side face image of the person. The camera frame rate must be greater than 30fps to ensure that the side face is presented clearly and completely in the image. The acquisition process must be carried out under stable lighting conditions, using natural light to reduce ambient light interference.

[0040] Step 2: Perform regional positioning on the side face image collected in step 1 to determine the region of interest (ROI) for heart rate detection; perform key point detection on the side face of a person through Mediapipe to obtain 478 key points of the side face of a person, that is, accurately locate the coordinates of various parts of the side face, including the forehead, nose, chin, cheeks and other key parts. Then, according to the preset proportion and position relationship, the region of interest (ROI) for heart rate detection is determined according to the coordinates of the key points, including the temple, side cheeks, and side of the chin.

[0041] Step 3: Use the rPPG algorithm to analyze the image data within the ROI determined in step 2, extract physiological signals related to heart rate, filter to remove noise and other interference factors, extract the pulse wave signal reflecting the heart rate change, and calculate the heart rate by detecting the peak point or period of the pulse wave signal.

[0042] Step 3.1, read the image data of the three ROIs of the temple, the side cheek and the side of the chin; use the function of the opencv library to calculate the RGB channel histogram of each ROI; subtract the signals of the green (GREEN) channel and the red (RED) channel in the extracted RGB channel histogram, and obtain the absolute value of the difference as the bvp signal; use the detrend tool to filter out the data with large deviations in the bvp signal, and then use a 5th-order Butterworth bandpass filter to filter and obtain the filtered signal; obtain the filtered signal spectrum after FFT of the filtered signal, and then use the argmax function to perform peak detection to obtain the instantaneous heart rate of each ROI;

[0043] Step 3.2, use the formula BPM(n)=α·BPM(n-1)+1-αBPMSig(n) to weighted update the instantaneous heart rate, and set α=0.95 to ensure that the heart rate results of each ROI will not jump;

[0044] Step 3.3, based on the peak value of the bvp signal of each ROI extracted in step 3.1, the signal after filtering is compared, and the mean square error (MSE) of the spectrum is calculated using the function to represent the degree of spectrum leakage, which is recorded as leakage, and the confidence is used. i =1 / 1+leakage Calculate the BPM confidence of each ROI, and calculate the final heart rate result by weighted summation based on the confidence and the instantaneous heart rate that will not jump calculated in step 3.2 using the following formula:

[0045]

[0046] Simulation experiment

[0047] Simulation conditions and equipment

[0048] 1. Environmental conditions

[0049] The simulation environment is a general indoor environment with no ambient noise, the ambient temperature is maintained at 20-25℃, and the humidity is maintained at 40%-60%RH. The subjects who are healthy and have no obvious cardiovascular disease, respiratory disease or skin disease are selected. The subjects should keep their emotions stable before the experiment, avoid tension, anxiety or excitement, etc.; the skin on the side of the face is clean, without cosmetic residue, obvious scars or rashes.

[0050] During the experiment, the subjects kept sitting with their heads naturally upright, their side faces perpendicular to the camera lens, and their eyes looking straight ahead. During the experiment, the subjects were required to minimize facial muscle movements (such as blinking, frowning, smiling, etc.) and large movements of other parts of the body.

[0051] 2. Simulation equipment

[0052] 1) HD camera, resolution 1080p, frame rate 50fps, bracket mountable.

[0053] 2) High-performance computers with professional data processing software, Python, etc.

[0054] Simulation content

[0055] Aim the camera at the side of the person's face to collect data, and each collection lasts for 3-5 minutes. Next, confirm the ROI and calculate the heart rate using the rPPG-based side face contactless heart rate detection method. Compare and analyze the calculated heart rate result with the actual heart rate, which is the result of the medical oximeter.

[0056] The simulation results are as follows:

[0057]

[0058]

[0059] To draw a scatter plot, see Figure 1 , the horizontal axis is the heart rate detected by the present invention, and the vertical axis is the heart rate measured by the medical pulse oximeter. The calculated R-square (used to measure the degree of fit of the predicted value to the true value, a value of 1 is called a perfect fit, and it is generally believed that the fitting effect is very good when its value is greater than or equal to 0.9.) value is 0.898, indicating that the present invention can not only realize the face detection of the side face, but also the accuracy of the heart rate prediction of the present invention is very high, and the fitting effect is good.

Claims

1. A method for detecting human side face heart rate without contact based on rPPG, characterized in that: The specific steps include: Step 1, collecting a side face image of a person through a camera, wherein the collected side face image is clear and complete; Step 2, performing regional positioning on the side face image collected in step 1 to determine a region of interest (ROI) for heart rate detection; Step 3, using the rPPG algorithm to calculate the instantaneous heart rate, confidence level, and weighted sum of the final heart rate for each ROI; Step 4: Display the image captured by the camera in step 1 and the instantaneous heart rate, confidence level, and weighted sum of the final heart rate of each ROI calculated in step 3 on the interface.

2. The method for detecting a person's side face contactless heart rate based on rPPG according to claim 1, characterized in that: When collecting the side face image of a person in step 1, natural light is used for illumination, and the lighting condition is stable and can penetrate the surface of the skin to enhance the ability to capture blood flow signals.

3. The method for detecting human side face heart rate without contact based on rPPG according to claim 1, characterized in that: The specific method of step 2 is: All key points of the side face of a person are obtained through the visualization tool Mediapipe, that is, the coordinates of each part of the side face are accurately located, and then the ROI for heart rate detection is selected.

4. The method for detecting human side face heart rate without contact based on rPPG according to claim 1, characterized in that: The specific method of step 3 is: Step 3.1, read the image data in the ROI determined in step 2; obtain the RGB channel histogram of each ROI; subtract the signals of the green (GREEN) channel and the red (RED) channel in the RGB channel histogram, and obtain the absolute value of the difference between the two as the BVP signal; filter out the deviation data in the BVP signal that is greater than the preset deviation, and then use a bandpass filter to filter to obtain a filtered signal; perform a fast Fourier transform FFT on the obtained filtered signal to obtain the filtered signal spectrum, and then perform peak detection to obtain the instantaneous heart rate of each ROI; Step 3.2, use the formula BPM(n)=α·BPM(n-1)+1-αBPMSig(n) to weighted update the instantaneous heart rate obtained in step 3.1, and set α=0.95 to ensure that the heart rate results of each ROI will not jump; Step 3.3, compare the peak value of the bvp signal of each ROI obtained in step 3.1 with the filtered signal, and use the function to calculate the mean square error (MSE) of the spectrum to represent the degree of spectrum leakage, recorded as leakage, and use confidence i =1 / 1+leakage calculates the BPM confidence of each ROI; based on the confidence and the instantaneous heart rate that will not jump calculated in step 3.2, the final heart rate result is calculated by weighted summation using the following formula:

5. A non-contact heart rate detection system for human side face based on rPPG, characterized in that: include: The image acquisition module is used in step 1 to capture the side face images with a camera; The ROI confirmation module is used in step 2 to perform profile detection on the collected profiles and identify all key points of the profiles to determine the ROI area; The signal processing module is used in step 3 to extract and process the signal of the ROI determined in step 2 to calculate the heart rate.

6. A non-contact heart rate detection device for human side face based on rPPG, characterized in that: include: Memory for storing computer programs; A processor is used to implement the rPPG-based non-contact heart rate detection method for the side face of a person described in steps 1 to 3 when executing the computer program.

7. A computer-readable storage medium storing a computer program, characterized in that When the computer program is executed by the processor, it can perform rPPG-based contactless heart rate detection on the side of the human face based on the rPPG-based contactless heart rate detection method on the side of the human face described in steps 1 to 3.

Images