A multi-region of interest tracking and selection method for video heart rate detection

Abstract

The application discloses a multi-interest region tracking and selecting method for video heart rate detection, which comprises the following steps: reading the first frame image of a face video, extracting face feature points, dividing the face skin region into 18 symmetrical sub-regions, and taking each two symmetrical sub-regions as one interest region; calculating the distance from the nose tip to the eyes to determine whether the head is rotated greatly, and discarding the sub-region which is shielded in the interest region if the head is rotated greatly; averaging the pixels in the green channel of each interest region to obtain 9 signal values; processing the subsequent video frames by repeating the above steps to obtain 9 signal streams; calculating the spectrum of each signal stream, selecting the peak with the highest occurrence frequency, and taking the frequency of the peak as a reference frequency; and calculating the heart rate values of the 9 signal streams respectively, and selecting the heart rate value closest to the reference frequency as the final heart rate value. The application can reduce motion interference, adaptively select the best signal, and improve the accuracy of heart rate estimation.

Description

Technical Field

[0001] The present invention relates to the field of digital image processing technology and digital signal processing technology, and in particular to a method for tracking and selecting multiple regions of interest for video heart rate detection. Background Art

[0002] Heart rate refers to the number of times the heart beats per minute. As one of the important physiological indicators for measuring health, regular heart rate monitoring can prevent and assist in the treatment of cardiovascular diseases.

[0003] Traditional heart rate measurement methods generally require direct contact between the sensor and the skin, making long-term monitoring inconvenient and unsuitable for patients with sensitive skin or surface trauma. With the widespread adoption of consumer cameras, imaging photoplethysmography (IPPG), a non-contact method, has become a popular development direction for future physiological parameter monitoring devices. Its principle is similar to that of IPPG: as the heart continuously contracts and relaxes, the blood filling level in the human blood vessels also changes accordingly. When ambient light shines on the human skin, the skin's absorption of light exhibits pulsatile variations consistent with the heart's contraction and relaxation, as blood volume changes. Consequently, the intensity of light reflected from the skin surface also undergoes corresponding periodic variations. IPPG technology captures subtle changes in light reflected from the skin to extract physiological signals and obtain physiological characteristics. Compared with traditional contact methods, IPPG offers many advantages, such as simplicity, low cost, and non-invasiveness. It has practical value and significant prospects for daily monitoring and telemedicine.

[0004] However, IPPG measurement is susceptible to various interference factors such as motion. Extracting high-quality raw heart rate signals and accurately calculating heart rate remain major challenges for this method. Current applications often directly select one or more fixed areas on the face as regions of interest. However, due to irregular head movements, some areas of the face are sometimes not captured by the camera at all, resulting in poor extracted signal quality or even no signal at all. Many studies have used signal processing methods to filter the extracted heart rate signals to remove interference such as motion, but the effectiveness of these signal processing methods is limited by the quality of the source signal. Furthermore, motion conditions are ever-changing, and fixed signal processing methods cannot account for all motion scenarios.

[0005] To address this issue, the present invention proposes a multi-ROI tracking and selection method for video heart rate detection. This method, designed to improve source signal quality, selects multiple ROIs and determines in real time whether each ROI is occluded by motion. If so, the ROI size is adaptively adjusted to remove the influence of non-skin areas. Furthermore, the method automatically selects the best-quality signal from the signals extracted from these ROIs to calculate heart rate. This method dynamically selects the best-quality heart rate signal for the entire face, helping to improve the accuracy of heart rate estimation. Summary of the Invention

[0006] In order to solve the problems existing in the background technology, the purpose of the present invention is to provide a method for tracking and selecting multiple regions of interest for video heart rate detection. This method can effectively improve the quality of the original heart rate signal, reduce the influence of motion interference, and improve the accuracy of heart rate estimation. Starting from the perspective of improving the quality of the source signal, the present invention selects multiple regions of interest and determines in real time whether each region of interest is occluded by motion. If occluded, the size of the region of interest is adaptively adjusted to remove the influence of non-skin areas. At the same time, the best quality signal is automatically selected from the signals extracted from these regions of interest to calculate the heart rate. The present invention can dynamically select the best quality heart rate signal for the entire face, which helps to improve the accuracy of heart rate estimation.

[0007] The specific technical solution for achieving the purpose of the present invention is:

[0008] A method for tracking and selecting multiple regions of interest for video heart rate detection, the method comprising the following steps:

[0009] (1) Read the first frame of the face video, use the Mediapipe Face Detection method to detect the face position and extract the facial feature points, divide the facial skin area into 18 symmetrical circular sub-areas, and use each of the symmetrical circular sub-areas as a region of interest, resulting in a total of 9 regions of interest;

[0010] (2) Select the feature points of the nose tip and both eyes from the facial feature points, calculate the distance from the nose tip to both eyes, set a threshold, and judge whether the head rotates significantly by the threshold value. If the head rotates significantly, discard the occluded sub-region in the region of interest and update the size of the region of interest;

[0011] (3) averaging the pixel values ​​of the green channels of the nine regions of interest in the first frame image to obtain nine signal values;

[0012] (4) Repeat steps (1) to (3) to process subsequent video frames, and sort the 9 signal values ​​obtained from each frame of image by time to obtain 9 signal streams;

[0013] (5) performing Fourier transform on the nine signal streams to obtain nine frequency spectra, selecting the peak with the most occurrences in the nine frequency spectra, and using the frequency of the peak as the reference frequency;

[0014] (6) Calculate the nine candidate heart rate values ​​of the nine signal streams respectively using a peak detection method, and select the candidate heart rate value with a frequency closest to the reference frequency from the nine candidate heart rate values ​​as the final heart rate value.

[0015] The steps of reading the first frame of the face video, detecting the face position and extracting facial feature points using the Mediapipe Face Detection method, dividing the facial skin area into 18 symmetrical circular sub-areas, and using each of the symmetrical circular sub-areas as a region of interest, to obtain a total of 9 regions of interest, are specifically as follows:

[0016] Read the first frame of the face video;

[0017] Use the Mediapipe Face Detection method to detect the face position and accurately extract 468 feature points of the face, marked as F1, F2, F3, ..., F 468 ;

[0018] For the left side of the face, take the feature points F 104 (Left side of forehead), F 108 (Left side of forehead), F 111 (left cheek), F 206 (left cheek), F 147 (left side of left cheek), F 101 (right side of left cheek), F 214 (left corner of mouth), F 204 (Left chin), F 201 (left chin) as the center of 9 circles, take F among the feature points 103 、F 109 、F 226 、F 207 、F 123 、F 142 、F 135 、F 194 、F 200 The distances from the nine circle centers are used as radii to divide the left face skin area into nine circular sub-areas l1 to l9;

[0019] For the right face, take F among the feature points 337 (right side of forehead), F 333 (right side of forehead), F 340 (right cheek), F 426 (right cheek), F 330(left cheek left point), F 376 (right cheek right point), F 434 (right mouth corner), F 424 (right chin), F 421 (right chin) as the center of 9 circles, take F 338 , F 332 , F 446 , F 427 , F 371 , F 352 , F 367 , F 422 , F 418 the distance between each of the 9 circles and F

[0020] respectively as the radius, divide the right face skin area into 9 circular sub-regions r1-r9 which are symmetrical to the circular sub-regions l1-l9 of the left face;

[0021] The two symmetrical sub-regions l1 and r1, l2 and r2, …, l9 and r9 are respectively grouped into a complete region of interest, that is, 9 regions of interest ROI1-ROI9 are obtained.

[0022] The steps of selecting the feature points of the nose tip and the eyes, calculating the distance from the nose tip to the eyes, and setting a threshold value to determine whether the head has undergone a large rotation, and discarding the occluded sub-regions in the region of interest and updating the size of the region of interest if a large rotation has occurred, are as follows:

[0023] Select the nose tip feature point F1 and the eye feature points F 133 (left inner corner point), F 362 (right inner corner point), calculate the distance between F1 and F 133 and the distance between F1 and F 362 , and record them as D1 and D2 respectively.

[0024] Set threshold values T1=D2-5xD1 and T2=D1-5xD2.

[0025] When the condition T1>0 is met, it is determined that the face has undergone a large rotation to the left side, and the sub-regions located on the left side in each region of interest are automatically deleted.

[0026] When the condition T2>0 is met, it is determined that the face has undergone a large rotation to the right side, and the sub-regions located on the right side in each region of interest are automatically deleted.

[0027] Update the region of interest, that is, ROI1 is composed only of r1, …, and ROI9 is composed only of r9.

[0028] When none of the above conditions are met, the region of interest remains unchanged.

[0029] The steps of performing Fourier transform on the nine signal streams to obtain nine frequency spectra, selecting the peak with the most occurrences in the nine frequency spectra, and using the frequency of the peak as the reference frequency are specifically as follows:

[0030] Performing Fourier transform on the nine signal streams to obtain nine frequency spectra F1 to F9;

[0031] Find the frequencies corresponding to the three highest peaks in the 0.7-3 Hz range in each spectrum, a total of 27 frequencies, denoted as f1-f 27 ;

[0032] From the 27 frequencies f1 to f 27 Find the frequency with the largest number of occurrences as the reference frequency, recorded as f'.

[0033] The steps of respectively calculating the nine candidate heart rate values ​​of the nine signal streams using the peak detection method, and selecting the candidate heart rate value with the frequency closest to the reference frequency from the nine candidate heart rate values ​​as the final heart rate value are specifically as follows:

[0034] Use the peak detection method to detect all the peak points of the 9 signal streams respectively, and record the time corresponding to the peak point. The time corresponding to the n peak points of signal stream 1 is (p11, p12, p13, ..., p1 n ), ..., the time corresponding to the m peak points of signal stream 9 is (p91, p92, p93, ..., p9 m );

[0035] The heart rate is calculated using the time corresponding to the peak point to obtain candidate heart rate values ​​for the nine signal streams, which are recorded as hr1 to hr9. The formula for calculating the heart rate of signal streams 1 to 9 using the peak point is as follows:

[0036]

[0037] Find the candidate heart rate value closest to the reference frequency f'×60 in hr1~hr9, and record it as hr final , which is the final heart rate value.

[0038] The technical solution provided by the present invention has the following beneficial effects: The proposed method for tracking and selecting multiple regions of interest (ROIs) for video heart rate detection sets multiple ROIs, each consisting of two bilaterally symmetrical subregions. By determining the motion state and discarding invisible subregions in real time, the problem of ROI occlusion caused by head movement is effectively resolved. Simultaneously, by analyzing the spectrum of the heart rate signals extracted from the multiple ROIs, the best-quality heart rate signal is adaptively selected and used to calculate the heart rate value. This method significantly reduces motion interference and improves the accuracy of heart rate estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the present invention;

[0040] Figure 2 Schematic diagram of nine regions of interest determined using facial feature points in the present invention;

[0041] Figure 3 A flow chart of updating a region of interest according to a motion state of the present invention;

[0042] Figure 4 Flowchart for adaptively selecting the best heart rate signal of the present invention. DETAILED DESCRIPTION

[0043] In order to more clearly illustrate the technical means, technical improvements and beneficial effects of the present invention, the present invention is described in detail below with reference to the accompanying drawings.

[0044] See also Figure 1-4 The present invention provides a method for tracking and selecting multiple regions of interest for video heart rate detection, comprising the following specific steps:

[0045] S101: Read the first frame image of the face video, use the Mediapipe Face Detection method to detect the face position and extract the facial feature points, select the feature point as the center of the circle to draw a circle, divide the facial skin area into 18 symmetrical circular sub-areas, and use any two of the symmetrical circular sub-areas as one region of interest, resulting in a total of 9 regions of interest.

[0046] The specific steps are:

[0047] Read the first frame of the face video;

[0048] Use the Mediapipe Face Detection method to detect the face position and accurately extract 468 feature points of the face, marked as F1, F2, F3, ..., F 468 ;

[0049] For the left side of the face, take the feature points F 104 (Left side of forehead), F 108 (Left side of forehead), F 111 (left cheek), F 206 (left cheek), F 147 (left side of left cheek), F 101 (right side of left cheek), F 214 (left corner of mouth), F 204 (Left chin), F 201 (Left chin) is the center of 9 circles, and F is taken from the feature points 103 、F 109 、F 226 、F 207 、F 123 、F 142 、F 135 、F 194 、F 200 The distances from the centers of the nine circles are used as radii to divide the skin area of ​​the left face into nine circular sub-areas l1 to l9;

[0050] For the right side of the face, take the feature points F 337 (right side of forehead), F 333 (right side of forehead), F 340 (right cheek), F 426 (right cheek), F 330 (Point on the left side of the right cheek), F 376 (right side of right cheek), F 434 (right corner of mouth), F 424 (right chin), F 421 (right chin) as the center of the 9 circles, take F among the feature points 338 、F 332 、F 446 、F 427 、F 371 、F 352 、F 367 、F 422 、F 418 The distances from the centers of the nine circles are used as radii to divide the skin area on the right face into nine circular sub-areas r1 to r9 that are symmetrical to the circular sub-areas l1 to l9 on the left face;

[0051] The symmetrical sub-regions l1 and r1, l2 and r2, ..., l9 and r9 are respectively combined into a complete region of interest, that is, 9 regions of interest ROI1 to ROI9 are obtained;

[0052] Figure 2 FIG. 1 is a schematic diagram of nine regions of interest determined by using facial feature points provided in the present invention.

[0053] The Mediapipe Face Detection and Mediapipe Face Tracker methods are well-known algorithms [Lugaresi C, Tang J, Nash H, et al. Mediapipe: A framework for building perception pipelines [J]. arXiv preprint arXiv:1906.08172, 2019.] and will not be introduced in detail here.

[0054] S102: Select the feature points of the nose tip and both eyes from the facial feature points, calculate the distance from the nose tip to the eyes respectively, and set thresholds T1 and T2. The threshold values ​​are used to determine whether the head has rotated significantly. If so, the occluded sub-region in the region of interest is discarded and the size of the region of interest is updated.

[0055] See also Figure 3 As shown in the flowchart, the specific process of this step is:

[0056] Select the nose tip feature point F1 and the eye feature point F 133 (left inner corner of the eye), F 362 (right inner corner point), calculate F1 and F 133 The distance between F1 and F 362 The distances are recorded as D1 and D2 respectively;

[0057] Set threshold T1 = D2 - 5 × D1, threshold T2 = D1 - 5 × D2;

[0058] Determine whether the thresholds T1 and T2 are greater than 0. If T1 > 0, it is determined that the face has rotated significantly to the left, and the sub-region on the left side of each region of interest is automatically deleted.

[0059] Update the region of interest, i.e. ROI1 is composed of only r1, ..., ROI9 is ​​composed of only r9;

[0060] When the condition T2>0 is met, it is determined that the face has rotated significantly to the right, and the sub-region on the right side of each region of interest is automatically deleted;

[0061] Update the region of interest, i.e. ROI1 is composed of only l1, ..., ROI9 is ​​composed of only l9;

[0062] When none of the above conditions are met, the region of interest remains unchanged.

[0063] S103: Average the pixel values of the green channel of the 9 regions of interest in the first frame image to obtain 9 signal values in the first frame image.

[0064] This step is specifically:

[0065] The average values avg11 to avg19 of each region of interest are obtained by averaging the pixels of the green channel of the 9 regions of interest.

[0066] S104: Repeat steps S101-S103 to process subsequent video frames, and sort the 9 signal values obtained from each frame image by time to finally obtain 9 signal streams.

[0067] This step is specifically:

[0068] For each frame in the subsequent video, repeat the processing steps S101-S103 to form 9 signal streams S1-S9 by time order, S1=(avg11, avg21, avg31, …), …, S9=(avg19, avg29, avg39, …).

[0069] S105: Fourier transform the 9 signal streams respectively to obtain 9 spectra, find the top 3 peaks in the 0.7-3 Hz interval in each of the spectra, select the peak with the highest frequency from the 27 peaks, and take the frequency of the peak as the reference frequency.

[0070] Referring to Figure 4 the flowchart, this step is specifically:

[0071] Fourier transform the 9 signal streams respectively to obtain 9 spectra F1-F9;

[0072] Find the frequencies corresponding to the top 3 peaks in the 0.7-3 Hz interval in each of the spectra, a total of 27 frequencies, denoted as f1-f 27 ;

[0073] Find the frequency with the highest frequency from the 27 frequencies f1-f 27 , as the reference frequency, denoted as f'.

[0074] S106: Calculate 9 candidate heart rate values for the 9 signal streams using the peak detection method, and select the candidate heart rate value closest to the reference frequency from the 9 candidate heart rate values as the final heart rate value.

[0075] Referring to Figure 4 the flowchart, this step is specifically:

[0076] Use the peak detection method to detect all the peak points of the 9 signal streams respectively, and record the time corresponding to the peak point. The time corresponding to the n peak points of signal stream 1 is (p11, p12, p13, ..., p1 n ), ..., the time corresponding to the m peak points of signal stream 9 is (p91, p92, p93, ..., p9 m );

[0077] The heart rate is calculated using the time corresponding to the peak point to obtain candidate heart rate values ​​for the nine signal streams, which are recorded as hr1 to hr9. The formula for calculating the heart rate of signal streams 1 to 9 using the peak point is as follows:

[0078]

[0079] Find the candidate heart rate value closest to the reference frequency f'×60 in hr1~hr9, and record it as hr final , which is the final heart rate value.

[0080] Example

[0081] This embodiment takes a 10-second video of an adult male's face with a frame rate of 30 frames per second as an example.

[0082] 1) Read the first frame of the face video, use the Mediapipe Face Detection method to detect the face position, and accurately extract 468 feature points of the face, marked as F1, F2, F3, ..., F 468 ;

[0083] For the left side of the face, take the feature points F 104 (Left side of forehead), F 108 (Left side of forehead), F 111 (left cheek), F 206 (left cheek), F 147 (left side of left cheek), F 101 (right side of left cheek), F 214 (left corner of mouth), F 204 (Left chin), F 201 (left chin) as the center of 9 circles, take F among the feature points 103 、F 109 、F 226 、F 207 、F 123 、F 142 、F 135 、F 194 、F 200 The distances from the centers of the nine circles are used as radii to divide the skin area of ​​the left face into nine circular sub-areas l1 to l9;

[0084] For the right side of the face, take the feature points F 337 (right side of forehead), F 333 (right side of forehead), F 340 (right cheek), F 426 (right cheek), F 330 (Point on the left side of the right cheek), F 376 (right side of right cheek), F 434 (right corner of mouth), F 424 (right chin), F 421 (right chin) as the center of the 9 circles, take F among the feature points 338 、F 332 、F 446 、F 427 、F 371 、F 352 、F 367 、F 422 、F 418 The distances from the centers of the nine circles are used as radii to divide the skin area on the right face into nine circular sub-areas r1 to r9 that are symmetrical to the circular sub-areas l1 to l9 on the left face;

[0085] The mutually symmetrical sub-regions l1 and r1, l2 and r2, ..., l9 and r9 are respectively combined into a complete region of interest, that is, 9 regions of interest ROI1 to ROI9 are obtained.

[0086] 2) Select the nose tip feature point F1 and the eye feature point F 133 (left inner corner of the eye), F 362 (right inner corner point), calculate F1 and F 133 The distance between F1 and F 362 The distances are recorded as D1 and D2 respectively;

[0087] Set threshold T1 = D2 - 5 × D1, threshold T2 = D1 - 5 × D2;

[0088] Determine whether the thresholds T1 and T2 are greater than 0. If T1 > 0, it is determined that the face has rotated significantly to the left, and the sub-region on the left side of each region of interest is automatically deleted.

[0089] Update the region of interest, i.e. ROI1 is composed of only r1, ..., ROI9 is ​​composed of only r9;

[0090] When the condition T2>0 is met, it is determined that the face has rotated significantly to the right, and the sub-region on the right side of each region of interest is automatically deleted;

[0091] Update the region of interest, i.e. ROI1 is composed of only l1, ..., ROI9 is ​​composed of only l9;

[0092] When none of the above conditions are met, the region of interest remains unchanged.

[0093] 3) Averaging the green channel pixel values ​​of the nine regions of interest obtains the mean value avg11 to avg19 of each region of interest.

[0094] 4) Repeat steps 1) to 3) for each frame in the subsequent video;

[0095] In this example, at 2 to 5 seconds, step 3) satisfies the condition T2>0, and it is determined that the face has deflected to the right. The sub-regions r1 to r9 are automatically deleted, and ROI1 to ROI9 are only composed of l1 to l9 respectively, and the region of interest is updated;

[0096] During the period from 0 to 2 seconds and from 5 to 10 seconds, no face deflection is detected, and ROI1 to ROI9 are still composed of l1+r1 to l9+r9 respectively, and the regions of interest remain unchanged.

[0097] The 9 values ​​of each frame signal are composed of 9 signal streams S1 to S9 according to time, S1 is (avg11, avg21, avg31, ..., avg3001), ..., S9 is (avg19, avg29, avg39, ..., avg3009).

[0098] 5) performing fast Fourier transform on the nine signal streams to obtain nine frequency spectra F1 to F9;

[0099] Find the frequencies corresponding to the top three highest peaks in the 0.7-3 Hz range in each spectrum, a total of 27 frequencies, denoted as f1-f 27 In this example, f1~f 27 They are 1.20, 1.17, 1.36, 1.37, 1.30, 1.60, 1.24, 1.17, 1.16, 1.30, 1.17, 1.20, 1.19, 1.10, 1.14, 1.10, 1.17, 1.04, 1.51, 1.23, 1.17, 1.19, 1.11, 1.17, 1.16, 1.17, 1.67. From the 27 frequencies, find the frequency that occurs the most and record it as f'. In this example, 1.17 is the frequency that occurs the most, and the reference frequency is finally determined to be 1.17.

[0100] 6) Detect all peak points of the 9 signal streams respectively by using peak detection method, and record the time corresponding to the peak points. In this example, the time corresponding to the 12 peak points of S1 is 0.30, 1.13, 1.77, 2.63, 3.73, 4.60, 5.13, 6.30, 6.83, 7.77, 8.57, 9.73, the time corresponding to the 12 peak points of S2 is 0.20, 1.13, 1.93, 2.53, 3.40, 3.90, 5.03, 5.83, 6.40, 7.73, 8.53, 9.80, the time corresponding to the 13 peak points of S3 is 0.17, 1.13, 1.80, 2.60, 3.36, 4.30, 5.16, 6.33, 6.83, 7.40, 8.46, 9.00, 9.80, the time corresponding to the 11 peak points of S4 is 0.20, 1.13, 1.96, 3.36, 4.16, 5.13, 5.86, 6.43, 7.70, 8.56, 9.76, the time corresponding to the 12 peak points of S5 is 0.16, 1.03, 2.10, 3.20, 4.10, 5.10, 5.60, 6.43, 7.23, 7.73, 8.56, 9.46, the time corresponding to the 14 peak points of S6 is 0.16, 0.66, 1.16, 2.06, 2.60, 3.33, 4.53, 5.10, 5.86, 6.63, 7.73, 8.60, 9.23, 9.73, the time corresponding to the 12 peak points of S7 is 0.70, 1.53, 2.33, 3.06, 4.03, 4.53, 5.73, 6.56, 7.26, 8.03, 9.33, 9.90, the time corresponding to the 12 peak points of S8 is 0.43, 1.53, 2.16, 3.13, 3.96, 5.03, 6.16, 7.06, 7.83, 8.76, 9.30, 9.83, the time corresponding to the 13 peak points of S9 is 0.13, 0.93, 1.76, 2.66, 3.40, 4.03, 4.63, 5.13, 6.43, 7.26, 8.06, 8.86, 9.63;

[0101] Calculate the heart rate by using the peak points to obtain a group of candidate heart rate values, denoted as hr1~hr9. In this example, hr1~hr9 are 71, 69, 75, 63, 71, 82, 72, 70, 76 respectively;

[0102] Find the candidate heart rate value closest to f'x60 in hr1~hr9, denoted as hr final , which is the final obtained heart rate value:

[0103] In this example, the value closest to 1.17x60=70.2 in hr1~hr9 is hr8, so hr final=hr8=70, the final calculated heart rate value is 70, and the 8th signal stream is the signal stream with the best quality selected by adaptive selection.

[0104] In summary, the present invention proposes a multi-ROI tracking and selection method for video heart rate detection. Based on face recognition and facial feature point extraction, the method automatically divides the face into multiple symmetrical sub-regions, with each two symmetrical sub-regions forming a region of interest. Each region of interest is independently tracked, and the region of interest is automatically updated once motion exceeding a threshold is detected, effectively resolving the issue of partially invisible facial skin caused by motion. Furthermore, from the perspective of improving the quality of the original signal, the method compares the spectrum of the original heart rate signal extracted from each region of interest in real time, finding the peak with the most occurrences to determine the reference range of the actual heart rate. Statistical characteristics are then used to automatically and dynamically select the best-quality signal from multiple original heart rate signals extracted from different regions of interest, thereby improving the accuracy of heart rate estimation. The present method provides effective data for non-contact heart rate measurement. Depending on the application context, the present invention, with appropriate modifications, is also applicable to the extraction of other relevant non-contact physiological features.

[0105] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for tracking and selecting multiple regions of interest for video heart rate detection, characterized in that: The method comprises the following specific steps: Step 1: Read the first frame of the face video, use the Mediapipe Face Detection method to detect the face position and extract facial feature points, divide the facial skin area into 18 symmetrical circular sub-regions, and use each of the symmetrical circular sub-regions as a region of interest, resulting in a total of 9 regions of interest; Step 2: Select the feature points of the nose tip and eyes from the facial feature points, calculate the distance from the nose tip to the eyes respectively, and set a threshold. The threshold value is used to determine whether the head has rotated significantly. If it has rotated significantly, the occluded sub-region in the region of interest is discarded and the size of the region of interest is updated; Step 3: averaging the pixel values ​​of the green channels of the nine regions of interest in the first frame image to obtain nine signal values; Step 4: Repeat steps 1 to 3 to process subsequent video frames, sort the 9 signal values ​​obtained from each frame of image by time, and obtain 9 signal streams; Step 5: Perform Fourier transform on the nine signal streams to obtain nine frequency spectra, select the peak that appears the most times in the nine frequency spectra, and use the frequency of the peak as the reference frequency; Step 6: Calculate the nine candidate heart rate values ​​of the nine signal streams respectively using a peak detection method, and select the candidate heart rate value with a frequency closest to the reference frequency from the nine candidate heart rate values ​​as the final heart rate value.

2. The method for tracking and selecting multiple regions of interest for video heart rate detection according to claim 1, characterized in that: The specific process of step 1 is: Read the first frame of the face video; Use the Mediapipe Face Detection method to detect the face position and accurately extract 468 feature points of the face, marked as F1, F2, F3, ..., F 468 ; For the left side of the face, take the feature points F 104 That is, the left side of the forehead, F 108 That is, the left side of the forehead, F 111 That is, the left cheek, F 206 That is, the left cheek, F 147 That is, the left side of the left cheek, F 101 That is, the right side of the left cheek, F 214 That is, the left corner of the mouth, F 204 That is, the left chin, F 201 That is, the left chin is taken as the center of the 9 circles, and F is taken from the feature points 103 、F 109 、F 226 、F 207 、F 123 、F 142 、F 135 、F 194 、F 200 The distances from the nine circle centers are used as radii to divide the left face skin area into nine circular sub-areas l1 to l9; For the right side of the face, take the feature points F 337 That is, the right side of the forehead, F 333 That is, the right side of the forehead, F 340 That is, right cheek, F 426 That is, right cheek, F 330 That is, the left side of the right cheek, F 376 That is, the right side of the right cheek, F 434 That is, the right corner of the mouth, F 424 That is, right chin, F 421 That is, the right chin is taken as the center of the 9 circles, and F is taken from the feature points 338 、F 332 、F 446 、F 427 、F 371 、F 352 、F 367 、F 422 、F 418 The distances from the centers of the nine circles are used as radii to divide the skin area on the right face into nine circular sub-areas r1 to r9 that are symmetrical to the nine circular sub-areas l1 to l9 on the left face; The mutually symmetrical sub-regions l1 and r1, l2 and r2, ..., l9 and r9 are respectively combined into a complete region of interest, that is, 9 regions of interest ROI1 to ROI9 are obtained.

3. The method for tracking and selecting multiple regions of interest for video heart rate detection according to claim 1, characterized in that: The specific process of step 2 is: Select the nose tip feature point F1 and the eye feature point F 133 That is, the left inner corner point, F 362 That is the right inner corner point, calculate F1 and F 133 The distance between F1 and F 362 The distances are recorded as D1 and D2 respectively; Set threshold T1 = D2 - 5 × D1, threshold T2 = D1 - 5 × D2; Determine whether the thresholds T1 and T2 are greater than 0. If T1>0 is met, it is determined that the face has rotated significantly to the left, and the sub-region on the left side of each region of interest is automatically deleted; Update the region of interest, i.e. ROI1 is composed of only r1, ..., ROI9 is ​​composed of only r9; When the condition T2>0 is met, it is determined that the face has rotated significantly to the right, and the sub-region on the right side of each region of interest is automatically deleted; Update the region of interest, i.e. ROI1 is composed of only l1, ..., ROI9 is ​​composed of only l9; When none of the above conditions are met, the region of interest remains unchanged.

4. The method for tracking and selecting multiple regions of interest for video heart rate detection according to claim 1, characterized in that: The specific process of step 5 is as follows: Performing Fourier transform on the nine signal streams to obtain nine frequency spectra F1 to F9; Find the frequencies corresponding to the three highest peaks in the 0.7-3 Hz range in each spectrum, a total of 27 frequencies, denoted as f1-f 27 ; From the 27 frequencies f1 to f 27 Find the frequency with the largest number of occurrences as the reference frequency, recorded as f'.

5. The method for tracking and selecting multiple regions of interest for video heart rate detection according to claim 1, characterized in that: The specific process of step 6 is as follows: Use the peak detection method to detect all the peak points of the 9 signal streams respectively, and record the time corresponding to the peak point. The time corresponding to the n peak points of signal stream 1 is (p11, p12, p13, ..., p1 n ), ..., the time corresponding to the m peak points of signal stream 9 is (p91, p92, p93, ..., p9 m ); The heart rate is calculated using the time corresponding to the peak point to obtain candidate heart rate values ​​for the nine signal streams, which are recorded as hr1 to hr9. The formula for calculating the heart rate of signal streams 1 to 9 using the peak point is as follows: Find the candidate heart rate value closest to the reference frequency f'×60 in hr1~hr9, and record it as hr final , which is the final heart rate value.

Images