A method for overcoming rppg signal motion artifact to improve heart rate detection accuracy

Abstract

The application discloses a method for overcoming rPPG signal motion artifact and improving heart rate detection accuracy. The method flow comprises: inputting a video containing a face; detecting a face area by using a face detection algorithm, and segmenting the face skin; extracting original RGB signals in the face area and acquiring three-axis motion information by using feature point tracking; performing dimension reduction on the RGB signals, and extracting rPPG signals; classifying the rPPG signals by using the motion information, and adaptively selecting three-dimensional cascade adaptive filtering and trajectory energy attenuation spectrum subtraction for noise reduction to overcome motion artifacts; and solving a heart rate trajectory in a frequency domain of the rPPG signals by using a hidden Markov model. The application effectively improves the accuracy of heart rate estimation in a motion scene from two aspects of improving signal signal-to-noise ratio and robust heart rate estimation.

Description

Technical Field

[0001] This invention belongs to the field of computer biomedical signal processing, specifically relating to a method for improving the accuracy of heart rate detection by overcoming motion artifacts in rPPG signals during non-contact heart rate estimation using an RGB camera. Technical Background

[0002] When the heart beats, the blood volume in the blood vessels changes with the heartbeat, generating a physiological signal (blood volume pulse wave, BVP). BVP is highly synchronized with the heartbeat. The principle of contact PPG is that when light shines into the skin tissue, it is absorbed and reflected by the blood. The absorption of light by the blood causes the intensity of the reflected light to change with the blood volume. Since changes in blood volume are related to the heartbeat, information related to the heartbeat can be obtained simply by detecting changes in the intensity of the reflected light.

[0003] In contact-based heart rate monitoring, most devices are based on the PPG principle, such as finger clip pulse wave detectors, finger clip pulse oximeters, smartwatches, and heart rate belts. These devices provide relatively accurate heart rate readings; however, the testing process either requires the measurement site to remain stable or is susceptible to accuracy reduction due to sweat. Furthermore, the presence of wiring can also affect the subject's condition, especially for infants and the elderly. Additionally, contact-based monitoring is difficult to sustain for extended periods.

[0004] Non-contact devices based on rPPG (remote PPG) only require a single RGB camera, such as for closed-circuit monitoring, eliminating the need for additional contact equipment. During the detection process, the subject remains free and unrestrained, offering significant advantages for individuals with sensitive skin, such as infants and young children, or those with damaged skin. Furthermore, it allows for long-term health monitoring. The devices are simple to operate and inexpensive, hence the widespread use of rPPG-based heart rate detection methods.

[0005] Since camera-based rPPG methods are highly susceptible to ambient lighting and motion, noise reduction processing is necessary for the extracted raw rPPG signals. Especially in scenes of intense motion, motion artifacts and their harmonics can overwhelm the rate component in the frequency spectrum. Therefore, we need to optimize the signal denoising algorithm to improve the signal-to-noise ratio of the rPPG signal and overcome the impact of motion artifacts on heart rate estimation. Summary of the Invention

[0006] To address the issue of motion artifacts in non-contact heart rate detection using RGB cameras, a method is proposed to overcome motion artifacts in rPPG signals and improve the accuracy of heart rate estimation.

[0007] This invention proposes a method to overcome motion artifacts in rPPG signals and improve the accuracy of heart rate detection. The specific steps include:

[0008] 1. Input a video containing a human face, use a neural network to detect the face region and segment the face skin, extract the three-channel RGB time-series signal by spatial pixel averaging, and use relevant algorithms to reduce the dimensionality to obtain a one-dimensional rPPG signal;

[0009] 2. Motion information in the horizontal X and vertical Y directions is extracted by feature point tracking, and motion information in the Z direction perpendicular to the face plane is indirectly extracted by calculating the area of ​​the facial skin region;

[0010] 3. Perform a short-time Fourier transform (STFT) on the motion signal to obtain the signal time-frequency graph, and then normalize and superimpose it. Train a CNN network to divide the motion scene into periodic motion and non-periodic motion according to the motion signal time-frequency graph, so as to select different noise reduction methods.

[0011] 4. Adaptively select trajectory energy attenuation spectrum subtraction for periodic motion to reduce noise, perform short-time Fourier transform on the rPPG signal to obtain amplitude spectrum information, use the obtained motion signal trajectory to perform energy attenuation at the corresponding position of the signal amplitude spectrum to suppress motion artifact components, and then use the amplitude spectrum with suppressed motion artifacts and the original phase spectrum to perform short-time Fourier inverse transform to obtain the denoised rPPG signal, thereby overcoming periodic motion artifacts and improving the quality of rPPG signal;

[0012] 5. Adaptively select a three-dimensional cascaded adaptive filter based on affine projection (AP) for noise reduction of non-periodic motion. In a cascaded manner, the noise reduction output of the previous stage filter is used as one of the inputs of the next stage filter, and the XYZ motion signals are input one by one as the other input of the filter. Motion artifact filtering in the corresponding direction is completed one by one, thereby overcoming non-periodic motion artifacts and improving the quality of rPPG signal.

[0013] 6. Construct a hidden Markov model using the time-frequency information of the filtered rPPG signal, and use the Viterbi algorithm to solve for the heart rate trajectory, thereby estimating and outputting the heart rate.

[0014] Based on the above, the innovations and advantages of this invention are as follows:

[0015] (1) Without adding an additional sensor, the camera imaging principle is used to indirectly obtain the Z-axis motion signal perpendicular to the face plane based on the area of ​​the face skin region, providing a third-direction reference signal for suppressing motion artifacts.

[0016] (2) Taking into full account the problem of inconsistent scale between the reference motion signal and the rPPG signal and the motion artifacts therein, the spectral subtraction method is improved by using only the spectral distribution information of the reference motion signal. The effect of "spectral subtraction" is achieved by suppressing the trajectory energy, thereby suppressing the artifacts of violent motion such as rowing machines and improving the signal-to-noise ratio of the signal.

[0017] (3) Taking into full account the spectral distribution characteristics of periodic motion artifacts and non-periodic motion artifacts, the CNN network is trained to divide the motion scene into periodic and non-periodic and adaptively select different noise reduction methods, which improves the robustness of motion artifact suppression.

[0018] (4) Taking full account of the three-dimensional spatial characteristics of motion, the motion is divided into three directions XYZ and the motion artifacts are denoised in a cascade manner to improve the quality of denoising. Attached Figure Description

[0019] Figure 1 The flowchart illustrates the method for overcoming motion artifacts in rPPG signals and improving the accuracy of heart rate detection provided by this invention.

[0020] Figure 2 This is a diagram of the skin area of ​​the face.

[0021] Figure 3 This is a schematic diagram of the XYZ three-axis motion.

[0022] Figure 4 Comparison of time-frequency diagrams for periodic and non-periodic motion signals.

[0023] Figure 5 This is the CNN classification network structure used in this paper.

[0024] Figure 6 A comparison of the frequency domain distribution of rPPG signals under different types of motion artifacts.

[0025] Figure 7 This is an improved spectral subtraction process based on trajectory energy decay.

[0026] Figure 8 This is a three-dimensional cascaded adaptive filtering process based on AP. Detailed Implementation

[0027] This invention can effectively overcome motion artifacts in rPPG signals and improve the accuracy of heart rate detection. It will be described in detail with reference to the accompanying drawings.

[0028] Combination Figure 1 The method presented in this paper can be divided into signal extraction, signal denoising, and heart rate estimation:

[0029] 1. Signal extraction stage:

[0030] First, a neural network is used to detect the face region, and skin segmentation is performed for signal extraction (the segmented facial skin is shown in the image). Figure 2 (As shown). Face detection is performed simultaneously with motion tracking to acquire motion signals along the X and Y axes, based on the camera's imaging principle:

[0031]

[0032] By indirectly obtaining Z-axis motion information through calculating the area of ​​facial skin, the final XYZ three-axis motion form is as follows: Figure 3 As shown.

[0033] Frame by frame, the facial skin region is averaged to obtain a three-channel temporal RGB signal, i.e.

[0034]

[0035] Where ROI(t) represents the facial skin region at time t, [i,j] represents a pixel within the facial skin region, and R, G, and B are the three pixel values ​​corresponding to that pixel.

[0036] Then, a colorimetric-based rPPG extraction algorithm is used to denoise the three-channel RGB time-series signal, extracting a one-dimensional rPPG signal, i.e.

[0037]

[0038]

[0039] Where N represents the total number of frames in the video, σ(·) represents the standard deviation operator, and P 1,2 This represents the projection axis within the RGB space.

[0040] 2. Signal noise reduction stage:

[0041] The extracted XYZ three-axis motion signals are subjected to first-order difference to obtain the corresponding velocity signals, i.e.

[0042]

[0043] Where f s This indicates the sampling rate, which is the video frame rate.

[0044] Perform short-time Fourier transforms on the XYZ three-axis velocity signals respectively, normalize the amplitude spectra, and plot time-frequency graphs. Then, superimpose the normalized time-frequency graphs into a single time-frequency graph. Figure 4 As shown. From Figure 4As can be seen, there is a clear difference between the time-frequency diagrams of periodic and non-periodic motions. Therefore, a CNN network can be trained to classify them, and then the motion can be determined as periodic or non-periodic based on the network classification results. The CNN network structure designed in this invention is as follows: Figure 5 As shown, the input is a time-frequency graph of the signal, and after convolutional pooling, the output is a classification result.

[0045] a) Regarding periodic motion artifacts:

[0046] Periodic motion is characterized by high intensity, strong periodicity, a relatively regular spectral distribution, and significant harmonics, exhibiting slow changes in the frequency domain, such as... Figure 4 As shown in .a. When modulated onto an rPPG signal, the heart rate component is easily submerged in the spectrum of motion artifacts, such as... Figure 6 As shown in .a.

[0047] Given the additive noise, spectral subtraction can be used to reduce it. To overcome the inconsistency between the scale of the extracted reference motion signal and the scale of the motion artifacts modulated on the rPPG signal, we only utilize the spectral distribution information of the reference motion signal. Since the motion artifacts in the rPPG signal are additive noise, their spectral distribution is consistent with that of the reference motion signal.

[0048] The specific processing flow of spectral subtraction based on trajectory energy decay is as follows: Figure 7 As shown.

[0049] Performing a short-time Fourier transform on the velocity signal yields the following results: Figure 4 The motion signal time-frequency graph shown in .a is used to obtain the frequency trajectory of the motion signal by performing frequency trajectory tracking on it. Given the frequency bandwidth σ, this frequency trajectory is then substituted into the time-frequency graph of the noisy rPPG signal to perform energy suppression on the frequency trajectory and its surrounding bandwidth.

[0050]

[0051] Where Z[m,n] represents the spectral information of the m-th row and n-th column of the signal time-frequency plot, F represents the frequency axis of length M corresponding to Z in the signal time-frequency plot, and σ n This represents the frequency bandwidth of the motion signal in the nth column of the time-frequency plot. After attenuation, the phase information Ψ and the attenuated amplitude information are obtained using the short-time Fourier transform of the rPPG signal. Perform an inverse short-time Fourier transform (ISTFT) to recover the signal from the frequency domain and obtain the denoised rPPG signal.

[0052] b) Targeting non-periodic motion artifacts

[0053] Non-periodic motion exhibits strong randomness and chaotic frequency domain distribution, such as Figure 4As shown in .b, and with low energy dispersion, the modulation has a weak impact on the rPPG signal, and the heartbeat component remains the main component of the rPPG signal. Figure 6 As shown in .b.

[0054] Assuming the clean rPPG signal is s(t), then the rPPG signal containing additive noise x′(t) is d(t) = s(t) + x′(t). The extracted velocity signal is used as the reference noise signal x(t), and the filter parameters for L taps are w. L If x(t) is a variable, then the output of x(t) after passing through the filter is:

[0055] y(t)=w L (txt)

[0056] The error signal between the noisy rPPG signal and the filtered output is:

[0057] e(t)=d(t)-y(t)=s(t)+x'(t)-w L (txt)

[0058] Since motion artifacts in the rPPG signal are additive noise, meaning x′(t) is highly correlated with x(t), motion artifacts can be suppressed by minimizing the error signal e(t). The e(t) with suppressed motion artifacts is the denoised rPPG signal. The minimization of the error signal e(t) is obtained through an iterative process based on affine projection transformation (AP), as follows:

[0059] y(t)=w L (txt)

[0060] e(t) = d(t) - y(t)

[0061]

[0062] Where L is the number of filter taps, P is the projection order in the AP algorithm, i.e. the number of historical data used, iterating from t=1 until t=N-L+1, and N is the signal length.

[0063] The above describes the specific implementation process of adaptive filtering. In this invention, we applied, as follows: Figure 8The diagram illustrates a 3D cascaded adaptive filter for noise reduction. First, the noisy rPPG signal and the X-axis velocity signal are input into the AP adaptive filter to reduce X-axis motion artifacts. Then, the rPPG signal with X-axis motion artifacts removed and the Y-axis velocity signal are input into the AP adaptive filter to reduce Y-axis motion artifacts. Finally, the rPPG signal with X and Y-axis motion artifacts removed and the Z-axis velocity signal are input into the AP adaptive filter to reduce Z-axis motion artifacts. This cascaded approach achieves noise reduction for XYZ-axis motion artifacts.

[0064] 3. Heart rate estimation stage

[0065] The signal-to-noise ratio of the rPPG signal is improved by suppressing motion artifacts. A short-time Fourier transform is performed on the denoised rPPG signal to obtain its time-frequency plot information. A hidden Markov model is then constructed using the time-frequency plot of the denoised rPPG signal, and the heart rate trajectory can be obtained by solving it using the Viterbi algorithm. Finally, the frequency is converted into heart rate, thus estimating and outputting the heart rate of the subject in the video.

[0066] The above description is only the preferred combination used in this invention. It should be noted that those skilled in the art can make several improvements and adjustments without departing from the principle of this invention, and these improvements and adjustments should also be considered within the scope of protection of this invention.

Claims

1. A method for overcoming motion artifacts in rPPG signals and improving the accuracy of heart rate detection, characterized in that, include: S1. Use facial feature point tracking to obtain motion signals in the horizontal X direction and vertical Y direction, and at the same time measure the size of the facial skin area to indirectly obtain motion signals in the Z direction perpendicular to the facial plane. S2. Perform a short-time Fourier transform on the motion signal to obtain time-frequency information, and use a CNN network to classify the motion artifacts in the rPPG signal into periodic motion artifacts and non-periodic motion artifacts based on the time-frequency graph. S3. Adaptively select different noise reduction methods based on the classification results of the motion artifacts: If the artifact is a periodic motion artifact, then spectral subtraction based on trajectory energy attenuation is used to denoise the rPPG signal. Specifically, the motion frequency trajectory is detected in the time-frequency plot of the motion signal. Where N is the number of data points of the frequency trajectory and the bandwidth of the trajectory in the frequency domain. The power spectrum and phase spectrum of the rPPG signal are obtained by performing a short-time Fourier transform on the rPPG signal. The power of the motion frequency trajectory l and the surrounding bandwidth are attenuated on the power spectrum. The attenuated power spectrum and the phase spectrum of the rPPG signal are then used to perform an inverse short-time Fourier transform to obtain the rPPG noise-reduced signal. If the motion artifacts are non-periodic, a three-dimensional cascaded adaptive filter based on affine projection transformation (AP) is used to denoise the rPPG signal. Specifically, an adaptive filter based on affine projection transformation (AP) is selected, and the motion artifacts modulated on the rPPG signal are gradually eliminated in a three-dimensional cascade manner using the extracted XYZ three-axis motion signals. First, the original rPPG signal and the X-axis motion signal are input for AP-based adaptive filtering. The denoised signal with the X-axis motion artifacts eliminated and the Y-axis motion signal are input into the filter for adaptive filtering again. Then, the denoised signal with the X and Y-axis motion artifacts eliminated and the Z-axis motion signal are input into the filter for a third adaptive filtering. Finally, the denoised rPPG signal with the X, Y, and Z-axis motion artifacts eliminated is output.

2. The method for overcoming motion artifacts in rPPG signals and improving the accuracy of heart rate detection according to claim 1, specifically for indirectly obtaining Z-direction motion signals using the size of facial skin area in S1, is characterized in that... Based on the camera imaging principle, the property of similar triangles is used to measure the size of the face to reflect the change in distance between the face and the camera, thereby indirectly obtaining the motion signal in the Z direction.

3. The method for overcoming motion artifacts in rPPG signals and improving the accuracy of heart rate detection according to claim 1, specifically for the classification process based on the time-frequency graph of the motion signal in S2, is characterized in that... Short-time Fourier transform is performed on the three-axis motion signal to obtain time-frequency information. The information is then normalized and finally superimposed into a time-frequency graph. A CNN network is trained and classified based on the spectral distribution characteristics of periodic and non-periodic motion signals.

4. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a method as described in any one of claims 1-3 for overcoming motion artifacts in rPPG signals and improving the accuracy of heart rate detection.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 3.

Images