A method, system, device and medium for contactless heart rate detection of human side face based on rPPG

Abstract

A method, system, device and medium for contactless heart rate detection of a person's side face based on rPPG. The method captures a person's side face with a camera, detects key points of the person's side face with MediaPipe, and obtains all key points of the person's side face; then determines an ROI (region of interest) for heart rate detection, extracts signals of the GREEN channel and RED channel in the RGB image from the image data within the ROI using an rPPG algorithm, and weightedly combines them as a BVP (blood volume pulse) signal; processes the BVP signal to obtain the person's heart rate information, and displays it on an interface, thereby realizing contactless measurement of the person's heart rate by detecting the person's side face; the system, device and medium realize contactless heart rate detection of the person's side face based on the rPPG method; the present invention can still detect the person's heart rate in real time even when the front of the person's face may be blocked or the angle is not conducive to detection, and has the advantages of convenience, high adaptability and high accuracy, while avoiding the inconvenience and infection risk that may be caused by traditional contact detection.

Description

Technical Field

[0001] The present invention relates to the field of biomedical detection technology, and in particular to a method, system, device and medium for contactless heart rate detection of a person's side face based on rPPG. Background Art

[0002] Heart rate is one of the key indicators reflecting the human physiological state and plays a vital role in many fields, including medical diagnosis, health monitoring, and sports training. Traditional heart rate detection methods mainly include electrocardiogram (ECG) and stethoscope measurement. Although ECG detection is accurate, it requires electrodes to be directly attached to the human skin and connected to the detection instrument through a complex lead system. This contact measurement method can cause inconvenience to the person being tested in some cases. For example, for burn patients, people with skin allergies, or individuals who need to monitor their heart rate continuously for a long time (such as patients in intensive care units), the attachment of electrodes may cause skin discomfort or even damage. Stethoscope measurement requires professional medical personnel to operate and is greatly affected by environmental noise. It also cannot achieve long-term continuous and automatic heart rate monitoring.

[0003] With the advancement of science and technology, contactless heart rate detection technology has gradually become a research hotspot. Among them, technologies based on the principle of photoplethysmography (PPG) have received widespread attention. Traditional PPG measures changes in light absorption by placing a light-emitting diode (LED) and a photodetector in contact with the skin surface, thereby obtaining heart rate information. However, this method still requires contact with the skin, which has certain limitations. Consequently, remote photoplethysmography (rPPG) has emerged. It uses a camera to capture image information of the human skin area and extracts heart rate signals by analyzing changes in skin color, without the need for direct contact with the human body. This method has greatly improved convenience and comfort, and can be widely used in scenarios such as home health monitoring and telemedicine.

[0004] Currently, rPPG-based contactless heart rate monitoring is mostly focused on the frontal area of ​​the face. This is because the front of the face is relatively flat, making image acquisition and analysis easier, and the distribution of blood vessels beneath the skin clearly correlates with heart rate fluctuations. However, frontal facial detection is susceptible to factors such as changes in facial expression, head movement, and ambient light interference. For example, when a person speaks, smiles, or frowns, the movement of facial muscles can alter the optical properties of the skin, thereby generating noise interference in the heart rate signal. The side of the face offers new possibilities for contactless heart rate monitoring, with its relatively stable blood vessel distribution and reduced sensitivity to facial expressions, creating the conditions for more accurate heart rate detection.

[0005] Patent application CN202410600014.0 uses the CardiA2Net method to combine a self-attention convolutional hybrid network (ACmix) network with an attention-based long short-term memory (ALSTM) network for heart rate detection. The disadvantage is that this method only targets the front face, and the number of facial feature points is too small, the positioning ROI is not accurate, and heart rate detection of the side face cannot be achieved. Summary of the Invention

[0006] In order to overcome the defects of the above-mentioned prior art, the purpose of the present invention is to provide a method, system, device and medium for contactless heart rate detection of a person's side face based on rPPG, which captures the side face of a person through a camera, detects key points of the side face of the person through Mediapipe, and obtains all key points of the side face of the person; then determines the ROI (region of interest) for heart rate detection, extracts the signals of the GREEN channel and the RED channel in the RGB image of the image data within the ROI through the rPPG algorithm, and weightedly combines them as a BVP (blood volume pulse) signal, processes the BVP signal to obtain the person's heart rate information, and displays it on the interface, thereby realizing contactless measurement of the person's heart rate by detecting the side face of the person; the present invention can still detect the person's heart rate in real time even when the front of the person's face may be blocked or the angle is not conducive to detection, and has the advantages of convenience, strong adaptability and high accuracy.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A method for detecting a person's side face heart rate without contact based on rPPG, comprising the following steps:

[0009] Step 1: Capture a side face image of a person through a camera, and the captured side face image is clear and complete;

[0010] Step 2: performing regional positioning on the profile face image acquired in step 1 to determine a region of interest (ROI) for heart rate detection;

[0011] Step 3: Calculate the instantaneous heart rate, confidence level, and weighted sum of the final heart rate for each ROI using the rPPG algorithm;

[0012] Step 4: Display the image captured by the camera in step 1 and the instantaneous heart rate, confidence level, and weighted sum of the final heart rate of each ROI calculated in step 3 on the interface.

[0013] When collecting the side face image of a person in step 1, natural light is used for illumination, and the lighting conditions are stable and can penetrate the surface of the skin to enhance the ability to capture blood flow signals.

[0014] The specific method of step 2 is:

[0015] All key points of the side face of a person are obtained through the visualization tool Mediapipe, that is, the coordinates of each part of the side face are accurately located, and then the ROI for heart rate detection is selected.

[0016] The specific method of step 3 is:

[0017] Step 3.1: Read the image data within the ROI determined in step 2; obtain the RGB channel histogram of each ROI; subtract the signals of the green (GREEN) channel and the red (RED) channel in the RGB channel histogram, and obtain the absolute value of the difference between the two as the BVP signal; filter out the deviation data in the BVP signal that is greater than the preset deviation, and then filter it using a bandpass filter to obtain a filtered signal; perform a fast Fourier transform (FFT) on the obtained filtered signal to obtain the filtered signal spectrum, and then perform peak detection to obtain the instantaneous heart rate of each ROI;

[0018] Step 3.2: Use the formula BPM(n) = α·BPM(n-1) + 1-αBPMSig(n) to weight the instantaneous heart rate obtained in step 3.1. Set α = 0.95 to ensure that the heart rate results of each ROI do not fluctuate.

[0019] Step 3.3, compare the peak value of the bvp signal of each ROI obtained in step 3.1 with the filtered signal, and use the function to calculate the mean square error (MSE) of the spectrum to represent the degree of spectrum leakage, recorded as leakage, and use confidence i =1 / 1 + leakage is used to calculate the BPM confidence of each ROI. Based on the confidence and the instantaneous heart rate calculated in step 3.2, the final heart rate result is calculated by weighted summation using the following formula:

[0020]

[0021] A non-contact heart rate detection system for a person's side face based on rPPG, comprising:

[0022] The image acquisition module is used in step 1 to capture the side face image with a camera;

[0023] The ROI confirmation module is used in step 2 to perform profile detection on the collected profiles and identify all key points of the profiles to determine the ROI area;

[0024] The signal processing module is used in step 3 to extract and process the signal of the ROI determined in step 2 to calculate the heart rate.

[0025] A non-contact heart rate detection device for a person's side face based on rPPG, comprising:

[0026] memory for storing computer programs;

[0027] The processor is used to implement the rPPG-based non-contact heart rate detection method for the side face of a person described in steps 1 to 4 when executing the computer program.

[0028] A computer-readable storage medium stores a computer program. When executed by a processor, the computer program can perform rPPG-based contactless heart rate detection of a person's side face based on the rPPG-based contactless heart rate detection method described in steps 1 to 4.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. The present invention realizes contactless heart rate detection through the side of the face, avoiding the inconvenience and discomfort of traditional contact detection methods. Users do not need to wear any equipment and can be tested in a natural state. It is especially suitable for special groups such as the elderly, infants, and people with sensitive skin.

[0031] 2. Compared to existing non-contact facial detection methods that primarily focus on the front of the face, the profile detection function of this invention offers advantages in a wider range of scenarios. For example, it can accurately detect the heart rate of the person being tested while they are resting on their side or turning their head, thus expanding the scope of detection.

[0032] 3. Through the advanced rPPG algorithm, the present invention can effectively extract the BVP signal of the ROI, perform filtering and Fourier transform, and then perform peak detection to obtain the instantaneous heart rate of each ROI; weighted update of the instantaneous heart rate increases the detection accuracy; based on the confidence level and the obtained instantaneous heart rate that does not jump, the weighted summation is used to calculate the final heart rate result, which is efficient and accurate; at the same time, the system has anti-interference ability and can operate stably under natural lighting conditions and a certain degree of environmental interference, providing reliable technical support for long-term, continuous heart rate monitoring.

[0033] 4. This invention only requires a camera to monitor heart rate, which reduces the equipment requirements, lowers the application threshold, and expands the scope of application. It is suitable for various scenarios, such as family health management, remote medical monitoring, and online education platforms.

[0034] In summary, the present invention does not require users to wear any equipment and can be tested in a natural state, which has advantages in more scenarios. At the same time, the present invention uses advanced rPPG algorithms, which have efficient processing, strong anti-interference ability and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1It is a scatter plot of the simulation experiment results of the present invention.

[0036] Figure 2 This is a flow chart of calculating a person's heart rate according to the present invention.

[0037] Figure 3 Flowchart of the rPPG algorithm for calculating human heart rate in the present invention. DETAILED DESCRIPTION

[0038] See also Figure 2 、 Figure 3 A method for detecting a person's side face heart rate without contact based on rPPG specifically comprises the following steps:

[0039] Step 1: Capture a profile image of the person's face. Use a high-resolution, high-frame-rate optical imaging device, such as a camera, to capture the profile image. The camera frame rate must be greater than 30 fps to ensure a clear and complete image of the face. The acquisition process must be performed under stable lighting conditions, using natural light to minimize ambient light interference.

[0040] Step 2: Localize the profile image captured in Step 1 to determine the region of interest (ROI) for heart rate detection. Mediapipe detects key points on the profile face, obtaining 478 key points. This accurately locates the coordinates of various facial features, including the forehead, nose, chin, and cheeks. Then, based on the preset proportions and positional relationships, the key point coordinates are used to determine the region of interest (ROI) for heart rate detection, including the temple, cheek, and chin.

[0041] In step 3, the image data within the ROI determined in step 2 is analyzed using the rPPG algorithm to extract physiological signals related to heart rate, filter out noise and other interference factors, extract the pulse wave signal reflecting the heart rate changes, and calculate the heart rate by detecting the peak point or period of the pulse wave signal.

[0042] Step 3.1: Read image data from the temple, cheek, and chin lateral regions of interest (ROIs). Use the OpenCV library's functions to calculate the RGB channel histogram for each ROI. Subtract the green and red channel signals from the extracted RGB channel histograms, and obtain the absolute value of the difference as the BVP signal. Use the detrend tool to remove data with large deviations from the BVP signal, then filter it using a 5th-order Butterworth bandpass filter to obtain a filtered signal. Perform the FFT on the filtered signal to obtain the filtered signal spectrum, and then perform peak detection using the argmax function to obtain the instantaneous heart rate for each ROI.

[0043] Step 3.2: Use the formula BPM(n) = α·BPM(n-1) + 1-αBPMSig(n) to weight the instantaneous heart rate, and set α = 0.95 to ensure that the heart rate results of each ROI do not fluctuate.

[0044] In step 3.3, the peak value of the BVP signal of each ROI extracted in step 3.1 is compared with the filtered signal, and the mean square error (MSE) of the spectrum is calculated using the function to represent the degree of spectrum leakage, which is recorded as leakage, and confidence is used. i = 1 / 1 + leakage. Calculate the BPM confidence of each ROI. Based on the confidence and the instantaneous heart rate calculated in step 3.2, the final heart rate result is calculated by weighted summation using the following formula:

[0045]

[0046] Simulation experiment

[0047] Simulation conditions and equipment

[0048] 1. Environmental conditions

[0049] The simulation environment was a typical indoor setting, free of ambient noise, maintained at a temperature of 20-25°C and a humidity of 40%-60% RH. Participants were selected to be in good health, with no apparent cardiovascular, respiratory, or skin diseases. Before the experiment, participants were required to maintain a stable mood and avoid any stress, anxiety, or excitement. Their facial skin was clean, free of makeup residue, and without any visible scars or rashes.

[0050] During the experiment, the subjects kept sitting with their heads naturally upright, their side faces perpendicular to the camera lens, and their eyes looking straight ahead. During the experiment, the subjects were required to minimize facial muscle movements (such as blinking, frowning, smiling, etc.) and large movements of other parts of the body.

[0051] 2. Simulation equipment

[0052] 1) High-definition camera, resolution 1080p, frame rate 50fps, bracket mountable.

[0053] 2) High-performance computers with professional data processing software, Python, etc.

[0054] Simulation content

[0055] The camera is pointed at the side of a person's face to collect data, with each acquisition lasting 3-5 minutes. Next, the ROI is determined, and the heart rate is calculated using a non-contact heart rate measurement method based on rPPG. The calculated heart rate result is compared and analyzed with the actual heart rate, which is the result of a medical oximeter.

[0056] The simulation results are as follows:

[0057]

[0058]

[0059] To draw a scatter plot, see Figure 1 The horizontal axis represents the heart rate detected by the present invention, and the vertical axis represents the heart rate measured by a medical pulse oximeter. The calculated R-squared value (which measures how well the predicted value fits the true value, with a value of 1 considered a perfect fit and values ​​greater than or equal to 0.9 generally considered to be very good) is 0.898, demonstrating that the present invention not only detects profile faces but also achieves highly accurate heart rate prediction, resulting in a good fit.

Claims

1. A method for detecting a person's side face contactless heart rate based on rPPG, characterized in that: The specific steps include: Step 1: Capture a side face image of a person through a camera, and the captured side face image is clear and complete; Step 2: performing regional positioning on the profile face image acquired in step 1 to determine a region of interest (ROI) for heart rate detection; Step 3: Calculate the instantaneous heart rate, confidence level, and weighted sum of the final heart rate for each ROI using the rPPG algorithm; Step 3.1, read the image data within the ROI determined in step 2; obtain the RGB channel histogram of each ROI; The signals of the green (GREEN) channel and the red (RED) channel in the RGB channel histogram are subtracted to obtain the absolute value of the difference between the two as the BVP signal; the deviation data in the BVP signal that is greater than the preset deviation is filtered out, and then filtered using a bandpass filter to obtain a filtered signal; the obtained filtered signal is subjected to a fast Fourier transform (FFT) to obtain the filtered signal spectrum, and then peak detection is performed to obtain the instantaneous heart rate of each ROI; Step 3.2: Use the formula BMP(n) = α·BMP(n-1) + (1-α)BMP(Sig(n)) to weight the instantaneous heart rate obtained in step 3.

1. Set α = 0.95 to ensure that the heart rate results of each ROI do not jump. n is the number of BMPs. Step 3.3, compare the peak value of the bvp signal of each ROI obtained in step 3.1 with the filtered signal, and use the function to calculate the mean square error (MSE) of the spectrum to represent the degree of spectrum leakage, recorded as leakage, and use confidence i = 1 / (1 + leakage) to calculate the BPM confidence of each ROI, where i is the ROI region identifier; based on the confidence and the instantaneous heart rate that does not jump calculated in step 3.2, the final heart rate result is calculated using the following formula: Step 4: Display the image captured by the camera in step 1 and the instantaneous heart rate, confidence level, and weighted sum of the final heart rate of each ROI calculated in step 3 on the interface.

2. The method for detecting a person's side face contactless heart rate based on rPPG according to claim 1, characterized in that: When collecting the side face image of a person in step 1, natural light is used for illumination, and the lighting conditions are stable and can penetrate the surface of the skin to enhance the ability to capture blood flow signals.

3. The method for detecting a person's side face contactless heart rate based on rPPG according to claim 1, characterized in that: The specific method of step 2 is: All key points of the side face of a person are obtained through the visualization tool Mediapipe, that is, the coordinates of each part of the side face are accurately located, and then the ROI for heart rate detection is selected.

4. The detection system of the rPPG-based non-contact heart rate detection method for the side face of a person according to claim 1, characterized in that: include: The image acquisition module is used in step 1 to capture the side face image with a camera; The ROI confirmation module is used in step 2 to perform profile detection on the collected profiles and identify all key points of the profiles to determine the ROI area; The signal processing module is used in step 3 to extract and process the signal of the ROI determined in step 2 to calculate the heart rate.

5. The detection device of the method for detecting the human side face contactless heart rate based on rPPG according to claim 1, characterized in that: include: memory for storing computer programs; A processor is used to implement the rPPG-based non-contact heart rate detection method for the side face of a person described in steps 1 to 3 when executing the computer program.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it can perform rPPG-based contactless heart rate detection of a person's side face based on steps 1 to 3 of the rPPG-based contactless heart rate detection method of a person's side face according to claim 1.

Images