Near-infrared video heart rate detection method based on region selection and robust principal component analysis

Abstract

The invention discloses a near-infrared video heart rate detection method based on region selection and robust principal component analysis, and the method comprises the steps: capturing a face video through a camera, obtaining the feature points of a cheek region, and selecting a region of interest; pixels in a plurality of regions of interest of each frame of image are averaged and connected frame by frame, and a plurality of rPPG signals containing pulse wave information can be obtained. The rPPG signal is preprocessed, and a pre-reference signal is constructed to carry out preferential selection on original signals of different regions of interest, so that a high-quality rPPG signal is obtained. The selected signals are subjected to noise reduction processing through a robust principal component analysis algorithm, and contained pulse wave information is highlighted. And the detected heart rate is determined through frequency domain analysis of the signals. According to the method, heart rate detection is successfully completed through a common consumption-level near-infrared camera in a non-contact scene.

Description

Technical Field

[0001] The invention belongs to the field of non-contact biomedical signal processing technology, specifically a method for non-contact heart rate detection in facial videos captured by a near-infrared camera. Background Technology

[0002] In recent years, cardiovascular and cerebrovascular diseases have posed an increasingly serious threat to human health. Heart rate, as one of the body's vital signs, directly reflects the robustness of the cardiovascular system. Monitoring heart rate helps in the early detection and prevention of cardiovascular diseases and plays a crucial role in subsequent diagnosis and long-term health monitoring. With the continuous progress of the times and the development of modern technology, heart rate detection methods and equipment are also constantly being updated and evolved.

[0003] Generally speaking, heart rate detection methods are mainly divided into two categories: contact and non-contact. However, although contact heart rate detection devices exist as vital sign measurement tools in daily life, they cannot achieve real-time heart rate monitoring due to limitations in application scenarios.

[0004] To address this issue, a non-contact method for monitoring blood volume and pulse using traditional cameras has emerged, known as Remote Photoplethysmography (rPPG). Specifically, this technology uses natural light and a regular camera to acquire video of a face, recording the periodic changes in the subtle brightness of the skin tone. This periodic signal is imperceptible to the human eye, but by analyzing the minute color changes contained in the video signal, non-contact heart rate extraction can be achieved. Currently, rPPG technology can reliably extract heart rate under stable lighting conditions.

[0005] It's worth noting that these heart rate extraction techniques are performed under ambient light, with RGB video providing three channels. However, near-infrared video only has one grayscale channel. To overcome this challenge, multiple infrared filters or multiple regions of interest (ROIs) are typically used to expand the single-channel signal into a multi-channel signal, thereby extracting heart rate information more accurately. The development of this non-contact heart rate monitoring technology provides a more convenient and effective tool for future medical and health monitoring. Summary of the Invention

[0006] The purpose of this invention is to address the limitations of using the method in scenarios such as changing lighting, darkness, and insufficient lighting by designing a near-infrared video heart rate detection method based on region selection and robust principal component analysis, thereby enabling video heart rate detection in scenarios such as nighttime.

[0007] This invention proposes an efficient method for extracting pulse wave signals from near-infrared video rPPG signals. The main steps include: video capture, face recognition and region of interest tracking, obtaining the raw signal by averaging pixel values, rPPG signal preprocessing, robust principal component analysis, and heart rate calculation. The specific processing steps are as follows:

[0008] Step 1: Use a near-infrared camera to capture a video of the face, and use the entire video as input for the heart rate detection method at a frame rate of F;

[0009] Step 2: Divide the video into consecutive time frames, and determine M regions of interest (ROIs) in each frame using face recognition and feature point localization. The regions should be selected from areas with larger skin areas.

[0010] Step 3: In N frames of images, calculate the average pixel value of each region of interest in each frame, and connect them frame by frame to obtain M single-channel grayscale raw temporal sequences of length N. Therefore, M raw signals from different regions can be obtained from the video.

[0011] Step 4: Filter and normalize the mean and variance of the original signal. The filter is a 6th-order Butterworth bandpass filter with a filtering range of [0.85, 4.0] Hz, which can filter out most of the noise that differs significantly from the heart rate frequency; the mean and variance normalization can standardize the original signals in different regions, bringing them to the same order of magnitude.

[0012] Step 5: Construct a pre-reference signal, select the best K signals from multiple regions of interest to ensure the quality of the signals in subsequent processing and to achieve the highest signal-to-noise ratio;

[0013] Step 6: Robust principal component analysis algorithm can extract common signal components from signals in different regions. Since different regions are simultaneously affected by the blood volume brought by the heart, they all contain pulse wave information.

[0014] Step 7: Convert the signal from the time domain to the frequency domain using Fast Fourier Transform, and calculate the signal-to-noise ratio and the frequency corresponding to the peak frequency of the K spectra. Select the peak frequency of the spectrum corresponding to the signal with the highest signal-to-noise ratio as the heartbeat frequency.

[0015] Innovations and advantages of this invention:

[0016] (1) Under non-contact conditions, the heart rate value is extracted by a near-infrared camera, and the pulse wave signal can be detected relatively accurately in dark and low-light environments.

[0017] (2) Unlike other methods that use a fixed region of interest to acquire signals, this method can make optimal region selection that adapts to different testers and different environments.

[0018] (3) The same pulse wave signal and the noise signal with large differences contained in different regions are separated by robust principal component analysis to further improve the accuracy of heart rate detection. Attached Figure Description

[0019] Figure 1 This invention presents the main process for extracting heart rate from video.

[0020] Figure 2 A schematic diagram of the Region of Interest (ROI) for a face;

[0021] Figure 3 This is a single-channel pixel mean image obtained from multiple facial ROI regions;

[0022] Figure 4 This is a time series diagram of the signal obtained after preprocessing;

[0023] Figure 5 Pre-reference heart rate calculation flowchart;

[0024] Figure 6 Selection process based on merit;

[0025] Figure 7 The result of robust principal component analysis decomposition is shown in the figure.

[0026] Figure 8 This is the frequency domain power spectrum of the rPPG signal. Detailed Implementation

[0027] To more clearly illustrate the purpose, technical solution, and advantages of the embodiments of the present invention, the technical solution of the embodiments will be described in detail and completely below. The present invention can effectively detect the pulse wave signal contained in the rPPG signal. In the processing flow, combined with... Figure 1The process comprises four key steps: First, by detecting face regions in each frame of the near-infrared video and tracking regions of interest (ROIs) within each frame, we obtain the average pixel values ​​of M different facial ROIs. The average values ​​of these different regions are then concatenated frame by frame to obtain M raw single-channel signals. Second, these raw signals undergo preprocessing, and the best single-channel signals are selected to ensure that the chosen K information channels have the highest signal-to-noise ratio. Third, for the selected best signals, robust principal component analysis (RPCA) is used to separate the identical heart rate signals from irrelevant noise in the different single-channel signals, thus obtaining the heart rate signal components with the strongest correlation among the different signals. Finally, a Fourier transform is performed on the signals to convert them from the time domain to the frequency domain, and the heart rate value is calculated at the frequency, thereby selecting the final heart rate value.

[0028] Step 1: Use a near-infrared camera to capture a face video. The camera used is RealSense. Use the entire video as input for the heart rate detection method. The frame rate is F = 30 frames / s. The entire video has N frames.

[0029] Step 2: Divide the video into temporally continuous frames and determine the Region of Interest (ROI) in each frame using face recognition and feature point localization. These ROIs are located near the forehead, left and right cheeks, chin, and nose, areas with large facial skin areas. Generally, larger skin areas have more pixels and contain more pulse wave information, resulting in a higher signal-to-noise ratio. Figure 2 As shown;

[0030] Step 3: Calculate the average pixel value of each region of interest in each frame, and concatenate them frame by frame to obtain the single-channel grayscale raw signal of the facial region. The video consists of N frames, therefore M raw signal sequences from different regions can be obtained from the video. Each signal sequence consists of N values, such as... Figure 3 As shown;

[0031] Step 4: Perform filtering and mean-variance normalization on the original signal. The filter is a 6th-order Butterworth bandpass filter with a filtering range of [0.85, 4.0] Hz, corresponding to a heart rate range of [49, 240] bpm, which is consistent with the normal range of human heart rate. This filtering range can filter out most of the noise that differs significantly from the heart rate frequency. Mean-variance normalization scales the signal amplitude to a mean of zero and a unit variance of zero. The mathematical formula is as follows:

[0032]

[0033] Calculate the mean and standard deviation of the signal amplitude separately. Subtract the mean from each signal amplitude and then divide by the standard deviation. The mean and variance can standardize the original signals from different regions, bringing them to a similar order of magnitude. The processed signal sequence is as follows: Figure 4 As shown;

[0034] Step 5: Add the signals from the M regions to construct a pre-reference signal, such as... Figure 5 As shown, since signals from different regions possess the same pulse wave signal and heart rate information, but have different noise components, summing them can enhance the heart rate signal and reduce noise components to a certain extent. A pre-reference signal is shifted from the time domain to the frequency domain, and the frequency corresponding to the peak value in the frequency domain is used as the reference heart rate. The signal-to-noise ratio (SNR) of the signals from M regions is calculated using the reference heart rate as the standard. The K regions with the highest SNR are selected to form a signal matrix, which serves as the subsequent input. The filtered signals are shown below. Figure 6 As shown.

[0035] Step 6: The robust principal component analysis algorithm decomposes the observation matrix D into the sum of a low-rank matrix A and a sparse matrix E, as shown in the following formula:

[0036]

[0037] subject A+E=D

[0038] Since signals from different Regions of Interest (ROIs) contain the same pulse wave signal, the low-rank matrix A is our target matrix for obtaining heart rate information. The sparse matrix E contains a matrix of discrete sparse noise points separated from signals from different ROIs that exhibit significant differences, such as... Figure 7 As shown;

[0039] Step 7: Convert the signal from the time domain to the frequency domain using Fast Fourier Transform, and calculate the signal-to-noise ratio (SNR) and the frequency corresponding to the peak value of each of the K spectra. Compare the K SNR values, and finally select the peak frequency of the spectrum corresponding to the signal with the highest SNR as the heart rate f. The common expression for heart rate is HR = f × 60, as shown below. Figure 8 As shown.

[0040] To further verify the accuracy of the near-infrared video heart rate detection method of this invention, the video detection results were compared with a reference heart rate, and the accuracy of the detection method was reflected by several indicators, including accuracy (the probability that the absolute value of the difference between the measured heart rate and the true heart rate is within 6 bpm), root mean square error, mean absolute error, and mean error rate. Results for 20 near-infrared video clips are presented below.

[0041] Table 1. Heart rate detection results using this method

[0042]

[0043] As can be seen from Table 1, the method of the invention can detect the heart rate value of the subject from the near-infrared video relatively accurately, and the error index values ​​are all small.

[0044] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A near-infrared video heart rate detection method based on region selection and robust principal component analysis, characterized in that... Includes the following steps: S1. Use a near-infrared camera to capture facial video. S2. Determine the region of interest (ROI) on the face through face recognition and feature point localization. S3. The entire video has N frames. Calculate the average pixel value of M regions of interest in each frame of the video, and connect them frame by frame to obtain M single-channel grayscale raw temporal signals of length N for the facial region. S4. Perform two preprocessing operations on the above M original signals of length N: filtering and mean-variance normalization. S5. Sum the M signals in the time domain to construct a pre-reference signal. Based on this pre-reference signal, select the best signals from multiple regions of interest and choose the K optimal signals. S6. Use robust principal component analysis algorithm to denoise the signal, and decompose the K*N signal matrix D into a low-rank matrix A and a sparse matrix E. S7. Convert the signal to the frequency domain for spectral peak analysis and calculate the heart rate.

2. The method according to claim 1, wherein step S2 is characterized in that, The process of determining the region of interest on the face includes: While ensuring that the selected facial area is as large as possible, select M regions of interest for subsequent processing. The larger the area of ​​the region, the more pixel values ​​it contains, and the more pulse wave information it contains.

3. The method according to claim 1, wherein step S3 is characterized in that, Obtain M single-channel grayscale raw temporal signals of length N for the facial region, including: Each image in a near-infrared video contains only grayscale values, and each region can only provide a single signal. Therefore, according to power requirement 2, M single-channel grayscale raw time-domain signals of length N can be obtained from M different regions.

4. The method according to claim 1, wherein step S4 is characterized in that, Two preprocessing operations are included: The filtering process uses a 6th-order Butterworth bandpass filter with a filtering range of [missing information]. [0.85Hz, 4.0Hz]. Mean-variance normalization involves calculating the mean and standard deviation of the signal amplitude separately, subtracting the mean from each signal amplitude, and then dividing by the standard deviation. The formula for mean-variance normalization is: Where, x mean σ represents the mean of the signal amplitude, and σ represents the standard deviation of the signal amplitude.

5. The method according to claim 1, wherein step S5 is characterized in that, Constructing a pre-reference signal includes: The M signals of length N are summed in the time domain to obtain a 1×N pre-reference signal. The specific summation formula is as follows: Among them, sig pre [n] represents the magnitude of the nth amplitude of the pre-reference signal; sig i [n] represents the magnitude of the nth amplitude in the i-th region. sig pre Perform a Fourier transform and take the frequency corresponding to the peak value of the power spectral density graph as the pre-reference heart rate value.

6. The method according to claim 1, wherein step S5 is characterized in that, Selecting the best signal from multiple regions of interest, including: Using the pre-referenced heart rate value in claim 5 as a reference, calculate the signal-to-noise ratio of M signals, and select the K (1 < K < M) signals with the highest signal-to-noise ratio to form a signal matrix D.

7. The method according to claim 1, wherein step S6 is characterized in that, Robust principal component analysis algorithms include: The multiple signals in claim 6 are combined into a signal matrix D, which is then decomposed into a low-rank matrix A and a sparse noise matrix E. The decomposition objective formula is constructed as follows: subject A+E=D The decomposed low-rank matrix A contains pulse wave information with a high signal-to-noise ratio; while matrix E is the separated noise.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the signal processing method as described in claim 1.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the processing method as described in claim 1.

Images