Non-contact heart rate measurement method based on face video

Abstract

The invention discloses a non-contact heart rate measurement method based on a face video. The method comprises the following steps: S1, acquiring a static video containing the face of a person to be detected; s2, performing face detection on the static video, positioning a face position, and determining face feature points; s3, selecting a plurality of measurement target points according to the face feature points and the face surface blood vessel distribution condition; and S4, performing signal frequency measurement on the measurement target point and reconstructing the video after signal amplification. The invention provides a non-contact heart rate measurement method based on a face video, which comprises the following steps of: processing a face static video acquired by a camera by using an image processing technology, determining a face position and acquiring face feature points, acquiring a plurality of target points on the basis, and calculating the heart rate of a human body by measuring the signal frequency of the target points. The method is used for solving the problems of defects and the like of a traditional contact measurement method.

Description

Technical Field

[0001] The invention relates to a non-contact heart rate measurement method based on face video, and belongs to the field of machine vision and signal processing. Background Art

[0002] Resting heart rate, also known as resting heart rate, refers to the number of heartbeats per minute when awake, inactive, and at rest. It reflects a person's health and varies depending on age, gender, and other physiological conditions. Studies have shown that a high resting heart rate can accelerate the progression of cardiovascular disease several times over, so measures should be taken to control excessive heart rate. As a crucial physiological parameter, real-time heart rate measurement plays a crucial role in the prevention and control of these diseases.

[0003] Currently, most professional heart rate monitoring devices on the market require contact with the skin to measure, and their application scenarios have certain limitations. This direct contact measurement method restricts the user's movement. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the existing technology and provide a non-contact heart rate measurement method based on face video. The method uses image processing technology to process the static face video captured by the camera, determine the face position and obtain the face feature points, and obtain several target points based on this. The human heart rate is calculated by measuring the signal frequency of the target points to solve the problems of the traditional contact measurement method.

[0005] In order to solve the above technical problems, the technical solution of the present invention is:

[0006] A non-contact heart rate measurement method based on face video, comprising the following steps:

[0007] Step S1: Obtain a static video containing the face of the person to be detected;

[0008] Step S2: perform face detection on the static video, locate the face position, and determine the facial feature points;

[0009] Step S3: selecting a number of measurement target points based on facial feature points and the distribution of blood vessels on the facial surface;

[0010] Step S4: measuring the signal frequency of the measurement target point and reconstructing the video after signal amplification;

[0011] Step S5: performing weighted average calculation on the signal frequencies of each measurement target point to obtain a final signal frequency, wherein the weight is determined according to the blood flow volume of the blood vessel corresponding to the measurement target point;

[0012] Step S6: Calculate the heart rate according to the signal frequency of the measurement target point.

[0013] Furthermore, in step S1, a static video containing the face of the person to be detected is obtained, which specifically includes the following steps:

[0014] A video of a certain length containing the face of the person to be detected is obtained through the camera, and the obtained video is loaded for subsequent processing by using Python and the image library included in Python.

[0015] Furthermore, in step S2, face detection is performed on the static video to locate the face position and determine the facial feature points, which specifically includes the following steps:

[0016] Step S21: First, scale the original image to different sizes, then arrange the images of different sizes into a pyramid shape, and then use the MTCNN model to calculate the image of each size. The MTCNN model includes a P-Net network, an R-Net network, and an O-Net network.

[0017] Step S22: A 3-channel RGB image with a width and height of 12 pixels is used as input to the P-Net network. The P-Net network determines whether the 12×12 RGB image contains a face. The P-Net network outputs the probability of containing a face and the probability of not containing a face, and provides the location of the face border and the location of the facial feature points.

[0018] Step S23: Each candidate face region output by the P-Net network is enlarged to a size of 24×24, and then input into the R-Net network for fine-tuning; the R-Net network determines whether the 24×24×3 image contains a face, and gives the location of the face frame and the location of the facial feature points;

[0019] In step S24, the R-Net network further enlarges the obtained face area to a size of 48×48, inputs it into the O-Net network, and outputs the final face position prediction frame and facial feature point positions.

[0020] Furthermore, the facial feature point positions include the position of the left eye, the position of the right eye, the position of the nose, the position of the left corner of the mouth, and the position of the right corner of the mouth.

[0021] Furthermore, in step S3, a number of measurement target points are selected according to the facial feature points and the distribution of blood vessels on the facial surface, which specifically includes the following steps:

[0022] Points with blood vessels are selected as measurement target points at the positions of the left eye, right eye, nose, left corner of the mouth and right corner of the mouth on the face, respectively, to obtain the measurement target points of the left eye position, the right eye position, the nose position, the left corner of the mouth and the right corner of the mouth.

[0023] Furthermore, in step S4, the signal frequency of the measurement target point is measured and the video after the signal amplification is reconstructed, which specifically includes the following steps:

[0024] Step S41: Convert the color space of the image from RGB to YIQ;

[0025] Step S42: Decompose each frame using a Laplacian pyramid to obtain basebands of different spatial frequencies;

[0026] Step S43: performing time-domain filtering on the constructed Laplacian pyramid through a filter;

[0027] Step S44: Amplify the signal frequency of the measurement target point

[0028] Step S45: Calculate the frequency of the measurement target point and reconstruct the video.

[0029] Furthermore, in step S5, the signal frequencies of each measurement target point are weighted averaged to obtain a final signal frequency, where the weight is determined according to the blood flow volume of the blood vessel corresponding to the measurement target point, and specifically includes the following steps:

[0030] After the selected measurement target points at the left eye position, the right eye position, the nose position, the left corner of the mouth position, and the right corner of the mouth position are processed in step S4, the signal frequency at each measurement target point can be obtained based on the respective heart rate main frequency graphs;

[0031] The signal frequencies of the five measurement target points are calculated using a weighted average method to obtain the final signal frequency. The calculation formula for the final signal frequency v is:

[0032]

[0033] Among them, the weight of the measurement target point at the left eye position is f1, the weight of the measurement target point at the right eye position is f2, the weight of the measurement target point at the nose position is f3, the weight of the measurement target point at the left corner of the mouth is f4, and the weight of the measurement target point at the right corner of the mouth is f5. The signal frequency of the measurement target point at the left eye position is v1, the signal frequency of the measurement target point at the right eye position is v2, the signal frequency of the measurement target point at the nose position is v3, the signal frequency of the measurement target point at the left corner of the mouth is v4, and the signal frequency of the measurement target point at the right corner of the mouth is v5.

[0034] Furthermore, in step S6, the heart rate is calculated according to the signal frequency of the measurement target point, which specifically includes the following steps:

[0035] When the heart beats once, the signal at the target point changes every time it completes one cycle. The heart rate is obtained by multiplying the signal frequency at the target point by 60.

[0036] By employing the above-mentioned technical solution, the present invention's non-contact heart rate measurement method based on facial video requires only the capture of facial video as input via a camera. Combined with corresponding video image processing methods and signal analysis and processing methods, it can quickly and accurately calculate the subject's real-time heart rate. This overcomes the shortcomings of traditional contact heart rate measurement methods by not requiring direct contact between human skin and any device. Therefore, it can achieve remote heart rate measurement, eliminating the need to wait in line at a hospital. This is highly convenient and suitable for individuals who are unsuitable for traditional contact measurement methods, such as newborns and those with extensive skin infections. The present invention offers the advantages of high accuracy, convenience, non-contact, and long-distance measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of the non-contact heart rate measurement method based on face video of the present invention;

[0038] Figure 2 This is a flowchart of the specific execution steps of the non-contact heart rate measurement method based on face video of the present invention;

[0039] Figure 3 The distribution map of blood vessels on the surface of the human face of the present invention;

[0040] Figure 4 This is a distribution diagram of the measurement target points of the present invention. DETAILED DESCRIPTION

[0041] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0042] like Figure 1 、 2As shown, this embodiment provides a non-contact heart rate measurement method based on face video, which includes the following steps:

[0043] Step S1: Obtain a static video containing the face of the person to be detected;

[0044] Step S2: perform face detection on the static video, locate the face position, and determine the facial feature points;

[0045] Step S3: selecting a number of measurement target points based on facial feature points and the distribution of blood vessels on the facial surface;

[0046] Step S4: measuring the signal frequency of the measurement target point and reconstructing the video after signal amplification;

[0047] Step S5: performing weighted average calculation on the signal frequencies of each measurement target point to obtain a final signal frequency, wherein the weight is determined according to the blood flow volume of the blood vessel corresponding to the measurement target point;

[0048] Step S6: Calculate the heart rate according to the signal frequency of the measurement target point.

[0049] In step S1 of this embodiment, obtaining a static video containing the face of the person to be detected specifically includes the following steps:

[0050] A video of a certain length containing the face of the person to be detected is obtained through a camera, and the obtained video is loaded for subsequent processing by using Python and the image library included in Python. Other languages ​​that can load and process videos can also be used. This implementation is preferably but not limited to Python.

[0051] Thirty frames of image data are read every second. To speed up image processing and reduce memory consumption, the region of interest (ROI) can be determined in the first frame of the video. Because the acquired video is static, only the content within the same region can be processed in subsequent frames. However, it is important to note that the ROI should be slightly larger than the image that fully exposes the subject's face to avoid incomplete facial images. Subsequent processing will accurately locate the face.

[0052] In step S2 of this embodiment, face detection is performed on a static video to locate the face position and determine the facial feature points, which specifically includes the following steps:

[0053] Step S21: First, scale the original image to different sizes, then arrange the images of different sizes into a pyramid shape, and then use the MTCNN model to calculate the images of each size to achieve the purpose of detecting faces at a unified scale. The MTCNN model includes the P-Net network, the R-Net network, and the O-Net network.

[0054] Step S22: A 3-channel RGB image with a width and height of 12 pixels is used as input to the P-Net network. The P-Net network determines whether this 12×12 RGB image contains a face. The P-Net network outputs the probability of containing a face and the probability of not containing a face. The output vector has a dimension of 1×1×2, that is, it contains two values, namely the probability of containing a face and the probability of not containing a face in the image, and provides the position of the face border and the position of the facial feature points. The facial feature point positions in this embodiment include the position of the left eye, the position of the right eye, the position of the nose, the position of the left corner of the mouth, and the position of the right corner of the mouth.

[0055] Step S23: Each candidate face region output by the P-Net network is enlarged to a size of 24×24 and then input into the R-Net network for fine-tuning; the R-Net network determines whether the 24×24×3 image contains a face and gives the location of the face border and the location of the facial feature points. The R-Net network eliminates many misjudgments in the P-Net network.

[0056] In step S24, the R-Net network further enlarges the obtained face area to a size of 48×48 and inputs it into the O-Net network. Compared with the R-Net network, the O-Net network performs more supervision on the face area and outputs the final face position prediction frame and facial feature point positions.

[0057] The face detection technology used in this embodiment is MTCNN. First, the image is scaled and transformed to construct an image pyramid to adapt to the detection of faces of different sizes. Then, the image pyramid constructed in the previous step is subjected to preliminary feature extraction and border calibration through the P-net network, and the window is adjusted by Bounding-Box Regression. Most of the windows are filtered through NMS to obtain a large number of candidates. The candidate images screened by the P-net network are fine-tuned by the R-net network. This network will filter out a large number of candidate frames with relatively poor results. Finally, the selected candidate frames are subjected to Bounding-Box Regression and NMS to further optimize the prediction results. The image after many candidates are filtered out by the R-net network is input into the O-net, which outputs accurate Bounding-Box coordinates and the coordinates of the five facial feature points.

[0058] In step S3 of this embodiment, a number of measurement target points are selected based on the facial feature points and the distribution of blood vessels on the facial surface, which specifically includes the following steps:

[0059] The blood vessels are distributed at the positions of the left eye, right eye, nose, left corner of mouth and right corner of mouth respectively. The points with blood vessels are selected as the measurement target points, and the measurement target points of the left eye position, right eye position, nose position, left corner of mouth and right corner of mouth are obtained. The blood vessels distribution on the face surface is as follows Figure 3 As shown in , according to experience, the greater the blood flow in the human face, the closer the heart rate calculated from the signal frequency at a certain point on the blood vessel is to the true value. Therefore, we try to find the measurement target point where the blood vessel is located. We can add or subtract a certain pixel value from the five known measurement feature points to find Figure 4 The five measurement target points are shown in the middle circles.

[0060] In step S4 of this embodiment, the signal frequency of the measurement target point is measured and the video after the signal amplification is reconstructed, which specifically includes the following steps:

[0061] Step S41: Color space conversion:

[0062] Convert the image's color space from RGB to YIQ. The RGB color model is an industry color standard that creates a variety of colors by varying and superimposing the three color channels: red (R), green (G), and blue (B). YIQ is the NTSC (National Television Standards Committee) television system standard. Y provides the brightness signal, or luminance, for both black-and-white and color televisions. I stands for in-phase, with colors ranging from orange to cyan. Q stands for quadrature-phase, with colors ranging from purple to yellow-green. The YIQ color space also has excellent clustering properties, which aids in cluster analysis in image processing. The YIQ color space also has excellent noise suppression capabilities, enabling better noise suppression and improved image quality during image processing. The YIQ color space is suitable for identifying and processing moving targets in complex backgrounds captured under natural conditions and can be effectively used in color image processing. The formula for converting between RGB and YIQ color spaces is as follows:

[0063]

[0064] Step S42: spatial filtering:

[0065] Use the Laplacian pyramid to decompose each frame and obtain basebands of different spatial frequencies; the Laplacian pyramid is used to store the difference between the downsampled image and the original image. i (Assuming G0 is the initial high-resolution image) first downsample to obtain the image Down(G i), and then up-sample to get the graph Up(Down(Gi)), and get Up(Down(G i )) and G i There is a difference, because the information lost in the downsampling process cannot be fully recovered by upsampling. The Laplace pyramid records the difference between each level of image downsampling and upsampling and before downsampling. i , the purpose is to be able to completely restore the image before downsampling at each level, H i The calculation formula is as follows:

[0066] H i =G i -Up(Down(G i ));

[0067] The number of layers of the pyramid can be adjusted according to the actual input video and actual application needs. In this embodiment, 4 layers are preferred.

[0068] Step S43: time domain filtering:

[0069] The constructed Laplacian pyramid is filtered in the time domain using a filter. The choice of filter generally depends on the application. For blood flow color amplification, a narrow-bandpass filter produces less noise. The present invention selects a bandpass filter between 0.83 Hz and 1.1 Hz. The pre-filtered, chaotic signal is filtered through a low-pass narrowband filter into a sine or cosine signal.

[0070] Step S44: amplify the signal frequency of the measurement target point:

[0071] The result of bandpass filtering is the approximation of the change of interest. Assume that the signal I(x, t) of the measurement target point at any time is:

[0072]

[0073] Among them, δ(t) is the change signal, and what we hope to obtain is the result after amplifying this change by α times, that is:

[0074]

[0075] In order to separate the changing part, the first-order Taylor series expansion is used to approximate the signal I(x, t):

[0076]

[0077] Ideally, the frequency range of all changing signals δ(t) is exactly the frequency band selected during bandpass filtering. The results of bandpass filtering are as follows:

[0078]

[0079] Multiply the changed part by an amplification factor α and add it to the original signal to amplify the signal using Taylor series approximation, that is:

[0080]

[0081] Ideally, is approximately equal to I(x,t), but the frequency range of the changing signal may exceed the selected frequency band. In this case, the bandpass filter only retains part of the changing signal, while other signals outside the selected frequency band will be weakened. Therefore, use γ k (t) represents the multiple of the k-th change signal attenuation at time t (0≤γ k ≤1), then:

[0082]

[0083] Set the magnification factor α k =γ k α, then:

[0084]

[0085] The amplified signal can be calculated from the above formula.

[0086] Step S45: Calculate the frequency of the measurement target point and reconstruct the video:

[0087] The amplified signal at the target point doesn't resemble the standard sine or cosine signals obtained after bandpass filtering, but it does have distinct regularity. The frequency of the target point can be determined by observing the heart rate main frequency graph obtained from that point. By superimposing the image of the changing signal with the original image, the amplified signal image can be reconstructed.

[0088] In step S5 of this embodiment, the signal frequencies of each measurement target point are weighted averaged to obtain a final signal frequency, where the weight is determined according to the blood flow volume of the blood vessel corresponding to the measurement target point. Specifically, the steps include:

[0089] After the selected measurement target points at the left eye position, the right eye position, the nose position, the left corner of the mouth position, and the right corner of the mouth position are processed in step S4, the signal frequency at each measurement target point can be obtained based on the respective heart rate main frequency graphs;

[0090] The greater the blood flow of the blood vessels at the facial position corresponding to the measurement target point, the greater the possibility that the frequency of the signal at the measurement target point is consistent with the heart rate. To reduce the error, this embodiment adopts a weighted average method for the signal frequencies of the five measurement target points, and considers the calculated result as the final signal frequency. The five measurement target points are as follows: Figure 4 As shown in the five origins, the final signal frequency v is calculated as follows:

[0091]

[0092] Among them, the weight of the measurement target point at the left eye position is f1, the weight of the measurement target point at the right eye position is f2, the weight of the measurement target point at the nose position is f3, the weight of the measurement target point at the left corner of the mouth is f4, and the weight of the measurement target point at the right corner of the mouth is f5. The signal frequency of the measurement target point at the left eye position is v1, the signal frequency of the measurement target point at the right eye position is v2, the signal frequency of the measurement target point at the nose position is v3, the signal frequency of the measurement target point at the left corner of the mouth is v4, and the signal frequency of the measurement target point at the right corner of the mouth is v5.

[0093] This embodiment uses the Euler image magnification algorithm as the signal frequency measurement technology. First, the video is read and the color space of each frame is converted from RGB to YIQ. Then, spatial filtering is performed, using a Laplacian pyramid to decompose each frame into basebands of different spatial frequencies. Once the basebands are obtained, each baseband is bandpass filtered in the time domain to extract the signal changes of interest. Each filtered layer of the signal is multiplied by a specific magnification factor and added to the original signal before frequency domain filtering to form a new pyramid. Finally, each layer of the new pyramid is synthesized to produce the final video of the amplified blood color changes.

[0094] In step S6 of this embodiment, the heart rate is calculated based on the signal frequency of the measurement target point, which specifically includes the following steps:

[0095] When the heart beats once, the signal at the target point changes every time it completes one cycle. The heart rate can be obtained by multiplying the signal frequency at the target point by 60.

[0096] The specific embodiments described above further illustrate the technical problems, technical solutions and beneficial effects solved by the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A non-contact heart rate measurement method based on face video, characterized in that: It includes the following steps: Step S1: Obtain a static video containing the face of the person to be detected; Step S2: perform face detection on the static video, locate the face position, and determine the facial feature points; Step S3: selecting a number of measurement target points based on facial feature points and the distribution of blood vessels on the facial surface; Step S4: measuring the signal frequency of the measurement target point and reconstructing the video after signal amplification; Step S5: performing weighted average calculation on the signal frequencies of each measurement target point to obtain a final signal frequency, wherein the weight is determined according to the blood flow volume of the blood vessel corresponding to the measurement target point; Step S6: Calculate the heart rate according to the signal frequency of the measurement target point.

2. The non-contact heart rate measurement method based on face video according to claim 1, characterized in that: In step S1, a static video containing the face of the person to be detected is obtained, which specifically includes the following steps: A video of a certain length containing the face of the person to be detected is obtained through the camera, and the obtained video is loaded for subsequent processing by using Python and the image library included in Python.

3. The non-contact heart rate measurement method based on face video according to claim 1, characterized in that: In step S2, face detection is performed on the static video to locate the face position and determine the facial feature points, which specifically includes the following steps: Step S21: First, scale the original image to different sizes, then arrange the images of different sizes into a pyramid shape, and then use the MTCNN model to calculate the image of each size. The MTCNN model includes a P-Net network, an R-Net network, and an O-Net network. Step S22: A 3-channel RGB image with a width and height of 12 pixels is used as input to the P-Net network. The P-Net network determines whether the 12×12 RGB image contains a face. The P-Net network outputs the probability of containing a face and the probability of not containing a face, and provides the location of the face border and the location of the facial feature points. Step S23: Each candidate face region output by the P-Net network is enlarged to a size of 24×24, and then input into the R-Net network for fine-tuning; the R-Net network determines whether the 24×24×3 image contains a face, and gives the location of the face frame and the location of the facial feature points; In step S24, the R-Net network further enlarges the obtained face area to a size of 48×48, inputs it into the O-Net network, and outputs the final face position prediction frame and facial feature point positions.

4. The non-contact heart rate measurement method based on face video according to claim 3, characterized in that: The facial feature point positions include the position of the left eye, the position of the right eye, the position of the nose, the position of the left corner of the mouth, and the position of the right corner of the mouth.

5. The non-contact heart rate measurement method based on face video according to claim 1, characterized in that: In step S3, a number of measurement target points are selected according to the facial feature points and the distribution of blood vessels on the facial surface, which specifically includes the following steps: Points with blood vessels are selected as measurement target points at the positions of the left eye, right eye, nose, left corner of the mouth and right corner of the mouth on the face, respectively, to obtain the measurement target points of the left eye position, the right eye position, the nose position, the left corner of the mouth and the right corner of the mouth.

6. The non-contact heart rate measurement method based on face video according to claim 1, characterized in that: In step S4, the signal frequency of the measurement target point is measured and the video after the signal amplification is reconstructed, which specifically includes the following steps: Step S41: Convert the color space of the image from RGB to YIQ; Step S42: Decompose each frame using a Laplacian pyramid to obtain basebands of different spatial frequencies; Step S43: performing time-domain filtering on the constructed Laplacian pyramid through a filter; Step S44: Amplify the signal frequency of the measurement target point Step S45: Calculate the frequency of the measurement target point and reconstruct the video.

7. The non-contact heart rate measurement method based on face video according to claim 1, characterized in that: In step S5, the signal frequencies of each measurement target point are weighted averaged to obtain the final signal frequency, where the weight is determined according to the blood flow volume of the blood vessel corresponding to the measurement target point. Specifically, the steps include: After the selected measurement target points at the left eye position, the right eye position, the nose position, the left corner of the mouth position, and the right corner of the mouth position are processed in step S4, the signal frequency at each measurement target point can be obtained based on the respective heart rate main frequency graphs; The signal frequencies of the five measurement target points are calculated using a weighted average method to obtain the final signal frequency. The calculation formula for the final signal frequency v is: Among them, the weight of the measurement target point at the left eye position is f1, the weight of the measurement target point at the right eye position is f2, the weight of the measurement target point at the nose position is f3, the weight of the measurement target point at the left corner of the mouth is f4, and the weight of the measurement target point at the right corner of the mouth is f5. The signal frequency of the measurement target point at the left eye position is v1, the signal frequency of the measurement target point at the right eye position is v2, the signal frequency of the measurement target point at the nose position is v3, the signal frequency of the measurement target point at the left corner of the mouth is v4, and the signal frequency of the measurement target point at the right corner of the mouth is v5.

8. The non-contact heart rate measurement method based on face video according to claim 1, characterized in that: In step S6, the heart rate is calculated according to the signal frequency of the measurement target point, which specifically includes the following steps: When the heart beats once, the signal at the target point changes every time it completes one cycle. The heart rate is obtained by multiplying the signal frequency at the target point by 60.

Images