Heart rate estimation method and device

Abstract

The invention provides a heart rate estimation method and device, and relates to the technical field of image processing, and the method comprises the steps: for each frame of a first face image in a face video, determining a weighted average pixel value corresponding to the first face image; based on the weighted average pixel values corresponding to all the first face images, determining a weighted average pixel value time sequence tensor; processing the weighted average pixel value time sequence tensor by adopting a sliding window mode to obtain a remote photoplethysmography (rPPG) pulse signal; a heart rate is estimated based on the rPPG pulse signal. According to the method, the weighted average pixel value time sequence tensor is determined according to the weighted average pixel values corresponding to all the first face images, accurate estimation of the heart rate is achieved on the basis of the rPPG pulse signals obtained in a sliding window mode, and the accuracy and efficiency of heart rate estimation are improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a heart rate estimation method and apparatus. Background Technology

[0002] Heart rate (HR) is one of the core vital signs reflecting the functional status of the cardiovascular system. Resting heart rate is not only used for routine health assessments but also plays a crucial role in the early screening, risk prediction, and rehabilitation management of cardiovascular diseases. Studies have shown that abnormally elevated resting heart rate is closely related to all-cause mortality and the incidence of cardiovascular events, thus it is widely used for chronic disease early warning and health risk assessment. Furthermore, beyond the medical field, heart rate is also used in various scenarios such as exercise training, psychological state assessment, driving safety, and human-computer interaction, demonstrating significant application value.

[0003] Traditional heart rate monitoring methods are mostly contact-based, such as electrocardiograms (ECG) or finger-clip photoplethysmography (PPG) devices. While these are highly accurate, they are inconvenient to wear and unsuitable for long-term, non-invasive, or remote monitoring. Therefore, remote photoplethysmography (rPPG) has been proposed in recent years. This technology is a non-contact physiological parameter monitoring method based on video signals.

[0004] The current mainstream rPPG implementation methods mainly include the Plane-Orthogonal-to-Skin (POS) method. This method, due to its high estimation accuracy and low computational cost, performs stably on standard datasets and is often used as a baseline for comparison with traditional methods. Traditional POS methods rely on relatively ideal shooting environments, such as stable lighting, a stable camera, and minimal head and facial muscle movement. Therefore, this method performs well on publicly available datasets. However, in real-world applications, video data is often affected by factors such as changes in lighting, camera shake, and occlusion of the Region of Interest (ROI), leading to decreased algorithm performance and lower accuracy in heart rate estimation. Summary of the Invention

[0005] This application provides a heart rate estimation method and apparatus to solve the problem of low accuracy in heart rate estimation.

[0006] This application provides a heart rate estimation method, including: For each frame of the face video, the weighted average pixel value corresponding to the first face image is determined. Based on the weighted average pixel values ​​corresponding to each of the first face images, determine the weighted average pixel value time series tensor; The weighted average pixel value time series tensor is processed using a sliding window method to obtain the remote photoplethysmography (rPPG) pulse signal. Heart rate is estimated based on the rPPG pulse signal.

[0007] According to a heart rate estimation method provided in this application, determining the weighted average pixel value corresponding to the first face image includes: Determine multiple regions of interest corresponding to the first face image; The weighted average pixel value is calculated based on each region of interest and the preset weight corresponding to each region of interest.

[0008] According to a heart rate estimation method provided in this application, determining multiple regions of interest corresponding to the first face image includes: Facial landmark detection is performed on the first face image to obtain facial landmarks; Based on the aforementioned facial key points, target key points are determined; Based on target key points, facial key points and preset target points, an affine matrix is ​​generated by least squares fitting. Based on the affine matrix, the first face image is aligned to obtain the second face image; Spatial filtering is applied to the second face image to obtain the filtered face image; The filtered face image is divided into regions of interest to obtain multiple regions of interest corresponding to the first face image.

[0009] According to a heart rate estimation method provided in this application, the step of processing the weighted average pixel value time series tensor using a sliding window approach to obtain an rPPG pulse signal includes: The weighted average pixel values ​​corresponding to the time series tensor of the weighted average pixel value in each sliding window are subjected to mean normalization to obtain the processed signal. The processed signal is projected onto two orthogonal components of the skin color space; The rPPG pulse signal is determined based on the two orthogonal components.

[0010] According to a heart rate estimation method provided in this application, determining the rPPG pulse signal based on the two orthogonal components includes: The fused signal is determined based on the two orthogonal components and the standard deviation corresponding to each orthogonal component; The rPPG pulse signal is determined based on the fused signals within all sliding windows.

[0011] According to a heart rate estimation method provided in this application, estimating the heart rate based on the rPPG pulse signal includes: Determine the optimal shift offset between the rPPG pulse signal and the real pulse signal; Based on the optimal shift offset, the rPPG pulse signal is cropped to obtain the cropped rPPG pulse signal; The cropped rPPG pulse signal is filtered to obtain the filtered rPPG pulse signal; Heart rate is estimated based on the filtered rPPG pulse signal.

[0012] According to a heart rate estimation method provided in this application, estimating the heart rate based on the filtered rPPG pulse signal includes: Peak detection is performed on the filtered rPPG pulse signal to obtain the peak value of the filtered rPPG pulse signal; Heart rate is estimated based on the intervals between all adjacent peaks.

[0013] According to a heart rate estimation method provided in this application, estimating the heart rate based on the filtered rPPG pulse signal includes: The heart rate frequency of the filtered rPPG pulse signal was extracted using Fast Fourier Transform (FFT). The heart rate is estimated based on the stated heart rate frequency.

[0014] According to the heart rate estimation method provided in this application, the method further includes: Detect whether the face video is shaking; If the shaking level of the face video exceeds a preset value, a target model is used to perform global motion compensation and stabilization processing on the face video.

[0015] This application also provides a heart rate estimation device, comprising: The first determining module is used to determine the weighted average pixel value corresponding to the first face image for each frame of the face video. The second determining module is used to determine the weighted average pixel value time series tensor based on the weighted average pixel value corresponding to each of the first face images; The processing module is used to process the weighted average pixel value time series tensor using a sliding window method to obtain the rPPG pulse signal; An estimation module is used to estimate the heart rate based on the rPPG pulse signal.

[0016] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the heart rate estimation method as described above.

[0017] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the heart rate estimation method as described above.

[0018] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the heart rate estimation method as described above.

[0019] The heart rate estimation method and apparatus provided in this application determine the weighted average pixel value corresponding to each frame of a first face image in a face video; based on the weighted average pixel values ​​corresponding to each of the first face images, a weighted average pixel value time series tensor is determined; the weighted average pixel value time series tensor is processed using a sliding window method to obtain a remote photoplethysmography (rPPG) pulse signal; and the heart rate is estimated based on the rPPG pulse signal. By determining the weighted average pixel value time series tensor corresponding to each of the first face images and using the rPPG pulse signal obtained by the sliding window method, accurate heart rate estimation is achieved, improving the accuracy and efficiency of heart rate estimation. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is one of the flowcharts illustrating the heart rate estimation method provided in this application.

[0022] Figure 2 This is a schematic diagram comparing the rPPG pulse signal obtained using the standard POS method with the real pulse signal, provided in an embodiment of this application.

[0023] Figure 3 This is a schematic diagram comparing the rPPG pulse signal and the real pulse signal provided in the embodiments of this application.

[0024] Figure 4This is a schematic diagram showing the peak values ​​of the rPPG pulse signal obtained using the standard POS method and the real pulse signal, provided in an embodiment of this application.

[0025] Figure 5 This is a schematic diagram comparing the peak values ​​of the rPPG pulse signal and the real pulse signal provided in the embodiments of this application.

[0026] Figure 6 This is a schematic diagram of the heart rate frequency of a real pulse signal provided in the embodiments of this application.

[0027] Figure 7 This is a schematic diagram of the heart rate frequency of the rPPG pulse signal provided in the embodiments of this application.

[0028] Figure 8 This is the second flowchart illustrating the heart rate estimation method provided in the embodiments of this application.

[0029] Figure 9 This is a schematic diagram of the heart rate estimation device provided in the embodiments of this application.

[0030] Figure 10 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] To facilitate understanding of the various embodiments of this application, the relevant background knowledge of this application will be introduced first.

[0033] In recent years, remote photoplethysmography (rPPG) has been proposed. This technology is a non-contact physiological parameter monitoring method based on video signals. Its basic principle is to use a camera to capture the changes in blood volume in exposed skin areas (such as the face and hands) under visible light illumination, caused by the contraction and relaxation of the heart, resulting in subtle periodic changes in the intensity of skin reflection. These changes mainly originate from the periodic fluctuations in subcutaneous blood volume in the face with the heartbeat, leading to subtle changes in the intensity of skin reflection, which is most noticeable in the green channel.

[0034] Current mainstream rPPG implementations include traditional signal processing methods such as Positive Positioning (POS), the green channel-based rPPG algorithm (GREEN), and chroma-based methods (CHROM), as well as deep learning-based estimation models (such as DeepPhys and PhysNet). These methods are implemented and integrated in various open-source toolboxes (such as rPPG-toolbox), covering the complete process from classical algorithms to modern deep neural networks. Among them, the POS method, due to its high estimation accuracy and low computational cost, performs stably on standard datasets (such as UBFC-RPPG) and is often used as a baseline for comparison with traditional methods. This method mainly separates the pulse signal from noise by projecting the RGB video signal onto the skin orthogonal plane. However, the traditional POS method relies on relatively ideal shooting environments, such as stable lighting, a stable camera, and minimal head and facial muscle movement. Therefore, while this method performs well on publicly available datasets, in real-world applications, video data is often affected by factors such as changes in lighting, camera shake, and ROI occlusion, leading to a decrease in algorithm performance.

[0035] Existing rPPG techniques (such as the POS method implemented in the rPPG Toolbox) perform well under ideal conditions, but have the following drawbacks: (1) ROI extraction usually uses simple rectangular cropping, which does not distinguish between areas with high and low signal quality, and is prone to introducing interference (such as eye and mouth movements), resulting in unstable signals; (2) Peak detection algorithms (such as the basic find_peaks algorithm without optimized parameters) are sensitive to noise, especially in complex environments, and are prone to generating false peaks, which leads to increased heart rate estimation error; (3) It has poor robustness to complex datasets (jitter, occlusion) and lacks technologies such as video stabilization and face alignment.

[0036] Therefore, this application proposes a heart rate estimation method based on the traditional POS method to improve adaptability in complex environments and the accuracy of heart rate estimation, thereby improving the precision of heart rate estimation.

[0037] The following is combined with Figures 1-8 Describe the heart rate estimation method of this application.

[0038] Figure 1 This is one of the flowcharts illustrating the heart rate estimation method provided in this application, such as... Figure 1 As shown, the method includes the following steps 101-104.

[0039] Step 101: For each frame of the face video, determine the weighted average pixel value corresponding to the first face image.

[0040] Specifically, the weighted average pixel value is the weighted average RGB signal corresponding to multiple regions of interest in the first face image, including the forehead, left and right cheeks, eyes and eyebrows, mouth and other areas.

[0041] For the first face image in each frame of a face video, the weighted average pixel value corresponding to the first face image can be determined.

[0042] Step 102: Determine the weighted average pixel value time series tensor based on the weighted average pixel value corresponding to each of the first face images.

[0043] Specifically, the weighted average pixel value time series tensor is the weighted average RGB signal time series tensor (rgb_tensor).

[0044] By combining the weighted average pixel values ​​corresponding to each of the first face images, a weighted average pixel value time series tensor can be determined. For example, the size of the weighted average pixel value time series tensor is (N, 3), where N represents the total number of frames of the first face image and 3 represents the number of RGB channels.

[0045] Step 103: Process the weighted average pixel value time series tensor using a sliding window method to obtain the remote photoplethysmography (rPPG) pulse signal.

[0046] Specifically, the length of the sliding window is determined by the frame rate (e.g., 30fps by default) and the window duration (e.g., 1.6 seconds by default), with a step size of 1 frame, corresponding to processing 48 frames of data each time. A new sliding window is started for calculation in each frame, thereby achieving fine-grained signal coverage and high temporal resolution rPPG pulse signal extraction.

[0047] By processing the time series tensor of the weighted average pixel value using a sliding window method, the rPPG pulse signal can be obtained.

[0048] Step 104: Estimate heart rate based on the rPPG pulse signal.

[0049] Specifically, heart rate can be estimated based on rPPG pulse signals.

[0050] The heart rate estimation method provided in this application determines the weighted average pixel value corresponding to the first face image in each frame of a face video; based on the weighted average pixel values ​​corresponding to each of the first face images, a weighted average pixel value time series tensor is determined; the weighted average pixel value time series tensor is processed using a sliding window method to obtain a remote photoplethysmography (rPPG) pulse signal; and the heart rate is estimated based on the rPPG pulse signal. By determining the weighted average pixel value time series tensor corresponding to each of the first face images and using the rPPG pulse signal obtained by the sliding window method, accurate heart rate estimation is achieved, improving the accuracy and efficiency of heart rate estimation.

[0051] Optionally, the method further includes: Detect whether the face video is shaking; if the shaking degree of the face video exceeds a preset value, use a target model to perform global motion compensation and stabilization processing on the face video.

[0052] Specifically, the target model can be the GlobalFlowNet-master model, which is a teacher-student network architecture. Through this architecture, combined with knowledge distillation training using a robust loss function, "accurate extraction of global motion and effective filtering of local motion" are achieved. After training, GlobalFlowNet-master can directly output a smooth global motion optical flow field, providing a reliable basis for motion compensation for video stabilization, while preserving local facial physiological motion, thus meeting the core requirements of rPPG video preprocessing.

[0053] After inputting a face video, the system first detects whether the video is shaky, i.e., assesses the degree of shakiness. If the shakiness exceeds a preset value, i.e., for severely shaky face videos, the GlobalFlowNet-master model is used for global motion compensation and stabilization. Specifically, inputting a severely shaky face video into the GlobalFlowNet-master model yields a stable and optimized face video output by the model. This GlobalFlowNet-master model achieves global motion compensation and stabilization by calculating dense optical flow between adjacent frames, thereby reducing global artifact interference caused by camera movement or head swaying.

[0054] Optionally, the specific implementation of step 101 above includes: Multiple regions of interest (ROIs) corresponding to the first face image are determined; the weighted average pixel value is calculated based on each ROI and its corresponding preset weight.

[0055] Specifically, based on medical priors and signal interference factors, different preset weights can be assigned to each region of interest (ROI) on the face. For example, the preset weight for the forehead is 0.4, for the cheeks it is 0.3, for the eyes and mouth it is 0.1, and for other regions it is 0.2. During the ROI signal fusion stage, regions with high signal-to-noise ratios and low motion interference (such as the forehead and cheeks) are strengthened, while regions with greater interference (such as the eyes and mouth) are suppressed, thereby improving the overall quality of the rPPG pulse signal. In this way, noise interference caused by facial expression changes and local occlusion is effectively reduced, providing a more stable and reliable signal basis for subsequent heart rate estimation.

[0056] Multiple regions of interest (ROIs) are identified for the first face image. The mean of all pixels in each ROI is multiplied by the preset weights corresponding to each ROI to obtain the corresponding products. The corresponding products are then summed to obtain the sum. The mean of all pixels can be the mean of the RGB three channels. The sum is then divided by the number of ROIs to obtain the weighted average pixel value corresponding to the first face image.

[0057] Optionally, determining the multiple regions of interest corresponding to the first face image includes: Facial landmark detection is performed on the first face image to obtain facial landmarks; target landmarks are determined based on the facial landmarks; an affine matrix is ​​generated by least-squares fitting based on the target landmarks, facial landmarks, and preset target landmarks; the first face image is aligned based on the affine matrix to obtain a second face image; spatial filtering is performed on the second face image to obtain a filtered face image; and regions of interest are divided into regions of interest in the filtered face image to obtain multiple regions of interest corresponding to the first face image.

[0058] Specifically, facial landmark detection is performed on the first face image using MediaPipe to obtain facial landmarks. Based on these landmarks, target landmarks are determined, which can be the left pupil, right pupil, and tip of the nose. Then, based on the target landmarks, the facial landmarks, and the preset target points, a least-squares fitting is used to generate an affine matrix M, where the preset target points are the left pupil, right pupil, and tip of the nose. The coordinates of the left pupil are... , (Image width 30%, height 40%); Coordinates of the right pupil are: , (Image width 70%, height 40%); the coordinates of the nose tip are... , (Image center, height 60%).

[0059] The first face image is aligned by solving for the affine matrix M to obtain the coordinates of the aligned pixels. Based on the coordinates of the aligned pixels, the second face image is determined to ensure the consistency of subsequent ROI region division and temporal sequence.

[0060] Spatial filtering of the second face image yields a filtered face image. By default, spatial filtering uses Gaussian filtering (e.g., a Gaussian filter with a kernel size of 5). Spatial filtering smooths the image and suppresses high-frequency noise, further improving signal quality. Spatial filtering can also be seen as preprocessing the Region of Interest (ROI), helping to reduce interference from uneven lighting and local noise on the rPPG signal.

[0061] By dividing the filtered face image into regions of interest, multiple regions of interest corresponding to the first face image can be obtained.

[0062] Optionally, the specific implementation of step 103 above includes: The weighted average pixel values ​​corresponding to the time series tensor of the weighted average pixel value in each sliding window are normalized to obtain the processed signal; the processed signal is projected onto two orthogonal components of the skin color space; and the rPPG pulse signal is determined based on the two orthogonal components.

[0063] Specifically, the weighted average pixel value time series tensor within each sliding window corresponds to multiple weighted average pixel values. For example, the weighted average pixel value time series tensor within each sliding window corresponds to the weighted average pixel values ​​of the first face image in 48 frames.

[0064] By normalizing the weighted average pixel values ​​corresponding to the time series tensor of the weighted average pixel values ​​within each sliding window, the processed signal can be obtained. The processed signal is then projected onto two orthogonal components of the skin color space, where the orthogonal matrix is ​​defined as follows: ,use Calculate two orthogonal components, where one orthogonal component... Another orthogonal component , 、 and This represents the processed signals from the R channel, G channel, and B channel. Based on these two orthogonal components, the rPPG pulse signal can be determined.

[0065] Optionally, determining the rPPG pulse signal based on the two orthogonal components includes: The fused signal is determined based on the two orthogonal components and the standard deviation corresponding to each orthogonal component; the rPPG pulse signal is determined based on the fused signal within all sliding windows.

[0066] Specifically, calculate the standard deviations of the two orthogonal components separately, for example, the orthogonal components. The standard deviation is Orthogonal components The standard deviation is .use Perform normalized weighting and adopt Remove DC component The fused signal is obtained, which is the signal of frame overlap.

[0067] The values ​​of the fused signals within all sliding windows are summed to obtain a cumulative sum. This cumulative sum is then divided by the number of overlaps to obtain a continuous one-dimensional rPPG pulse signal. Alternatively, the fused signals within all sliding windows are multiplied by their corresponding weights and then summed to obtain a weighted sum. This weighted sum is then divided by the total weights to obtain a continuous one-dimensional rPPG pulse signal. It should be noted that the weights corresponding to the fused signals within each sliding window can be the same or different.

[0068] Optionally, the specific implementation of step 104 above includes: Determine the optimal shift offset between the rPPG pulse signal and the true pulse signal; based on the optimal shift offset, trim the rPPG pulse signal to obtain a trimmed rPPG pulse signal; filter the trimmed rPPG pulse signal to obtain a filtered rPPG pulse signal; estimate the heart rate based on the filtered rPPG pulse signal.

[0069] Specifically, after obtaining the rPPG pulse signal and the real pulse signal, the rPPG pulse signal and the real pulse signal can be detrended, filtered and smoothed respectively, and then standardized.

[0070] The optimal shift offset between the rPPG pulse signal and the real pulse signal is calculated by using a cross-correlation function. Specifically, the maximum number of frames that can be shifted, max_lag, is defined. The shift value that maximizes the correlation between the signals is found by iterating through [-max_lag, max_lag], which is best_lag.

[0071] Based on the optimal shift offset, the overlapping portion of the rPPG pulse signal and the real pulse signal is trimmed to obtain trimmed rPPG pulse signals and trimmed real pulse signals, aligning the rPPG pulse signals with the real pulse signals in the time dimension. For example, based on the optimal shift offset, the overlap length is calculated, and based on the overlap length, the overlapping portion is trimmed. If the rPPG pulse signal comes first and the real pulse signal comes second, then the latter part of the rPPG pulse signal and the former part of the real pulse signal are taken, and the overlapping portion is extracted.

[0072] Figure 2 This is a schematic diagram comparing the rPPG pulse signal obtained using the standard POS method with the real pulse signal, provided in an embodiment of this application. Figure 3 This is a schematic diagram comparing the rPPG pulse signal and the real pulse signal provided in the embodiments of this application. It should be noted that... Figure 2 and Figure 3 These are comparison images of the rPPG of the first 800 frames of subject1 in the UBFC-RPPG dataset with the actual signal. Figure 3 As can be seen, the rPPG pulse signal cropped in this application is superior to the standard pos method in terms of waveform morphology and phase alignment.

[0073] By applying a bandpass filter (0.75-3Hz) to the cropped rPPG pulse signal, a filtered rPPG pulse signal can be obtained. The heart rate range corresponding to the bandpass filter (0.75-3Hz) is 45 to 180 beats per minute (bpm), covering the typical heart rate range from resting state to after strenuous exercise. Based on the filtered rPPG pulse signal, the heart rate can be estimated.

[0074] Optionally, estimating the heart rate based on the filtered rPPG pulse signal includes: Peak detection is performed on the filtered rPPG pulse signal to obtain the peak value of the filtered rPPG pulse signal; the heart rate is estimated based on the interval between all adjacent peak values.

[0075] Specifically, the `find_peaks` function is optimized to perform peak detection on the filtered rPPG pulse signal, obtaining the peak values ​​of the filtered rPPG pulse signal. Constraints are applied to these peak values, filtering out unreasonable "pseudo-peaks." The interval between adjacent peaks can be either a frame interval or a time interval. When the interval is a frame interval, the frame interval is divided by the frame rate to obtain the time interval between adjacent peaks; for example, with a frame rate of 30, this means 30 frames per second. The average of the time intervals corresponding to all adjacent peaks is determined, and the ratio of 60 seconds to the average of the time intervals corresponding to all adjacent peaks is used to determine the heart rate. Alternatively, when the interval is a time interval, the average of the time intervals corresponding to all adjacent peaks is determined, and the ratio of 60 seconds to the average of the time intervals corresponding to all adjacent peaks is used to determine the heart rate.

[0076] It should be noted that peak constraint methods include the following: 1. Minimum Interpeak Distance Constraint. To avoid "overly dense false peaks" caused by high-frequency noise, such as detecting more than 3 peaks within 1 second, which far exceeds the upper limit of the physiological heart rate, a conservative threshold of 0.4 seconds between adjacent peaks is used, with a maximum heart rate of 150 bpm. Basis: The resting heart rate range for adults is 40-170 bpm, corresponding to a minimum heart rate interval of approximately 0.35 seconds (170 bpm). The minimum interval between all adjacent peaks represents the minimum heart rate interval, i.e., how many seconds it takes for the heart to beat once. fs represents the frame rate; for example, a frame rate of fs = 30 means 30 frames per second. a represents the minimum interval between all adjacent peaks, for example, 0.4 seconds. dmin represents the minimum number of frames between two adjacent peaks. Peaks with an interval less than 0.4 seconds between adjacent peaks are filtered out.

[0077] 2. Prominence Adaptive Constraint. Filters out low-amplitude "weak noise peaks," such as spurious peaks formed by minute fluctuations in illumination, whose amplitudes are much lower than the true pulse peaks. ,in, Indicates the significance of the minimum peak value. , This represents the standard deviation of the rPPG pulse signal. , This represents the rPPG pulse signal in the i-th frame. This represents the mean of the rPPG pulse signal across all frames, where N represents the total number of frames. The minimum vertical distance between each peak and its adjacent left and right troughs must be greater than [value missing]. The minimum vertical distance between the adjacent left and right valley values ​​is less than or equal to The peak values ​​are filtered out.

[0078] 3. Absolute constraint on peak height. The true pulse peak of the rPPG pulse signal originates from the periodic increase in blood volume, and its amplitude is necessarily higher than the signal baseline (mean). By setting a height threshold, all invalid fluctuations below the baseline can be filtered out, avoiding the misjudgment of the "noise inflection point of the falling edge of the signal" as the pulse peak. The height threshold is the signal mean. , This indicates a height threshold; each peak height must be greater than [a certain value]. less than or equal to The peak values ​​are filtered out.

[0079] Figure 4 This is a schematic diagram comparing the peak values ​​of the rPPG pulse signal obtained using the standard POS method and the real pulse signal, provided in an embodiment of this application. Figure 5 This is a schematic diagram comparing the peak values ​​of the rPPG pulse signal and the real pulse signal provided in an embodiment of this application. Figure 4 and Figure 5 Both comparisons are of peak values ​​in the first 800 frames on subject1. It can be seen that the standard POS method is susceptible to spurious peak interference, while the method provided in this application effectively suppresses redundant peak values ​​and improves the accuracy of heart rate estimation.

[0080] Optionally, estimating the heart rate based on the filtered rPPG pulse signal includes: The heart rate frequency of the filtered rPPG pulse signal is extracted using Fast Fourier Transform (FFT); the heart rate is estimated based on the heart rate frequency.

[0081] Specifically, the length of the filtered rPPG pulse signal is determined to be greater than or equal to a preset threshold. If it is less than the preset threshold, an invalid result is returned directly because the FFT spectrum of a short signal is easily affected by noise and cannot reflect the true heart rate frequency. If it is greater than or equal to the preset threshold, the filtered rPPG pulse signal is divided into multiple segments, for example, each segment has a length of nperseg = 240 frames (8 seconds) and an inter-segment overlap of noverlap = 120 frames (50% overlap). A Hanning window is added to each segment (to suppress spectral leakage and avoid frequency aliasing). An FFT is performed on each windowed rPPG pulse signal segment to calculate the power spectral density (PSD), and the PSD of all segments is averaged (to reduce noise) to obtain the average value. Then, each rPPG pulse signal segment is de-DC (detrend='constant'), that is, each rPPG pulse signal segment is subtracted from the average value to obtain the DC-DC value of each rPPG pulse signal segment, in order to eliminate the interference of baseline drift on the spectrum.

[0082] Each segment of the rPPG pulse signal after DC removal is filtered, retaining only the frequency and power spectrum of 0.75~2.5Hz (corresponding to heart rate 45~150bpm), filtering out interference from respiration (<0.75Hz) and high-frequency noise (>2.5Hz). The power spectral density of each filtered rPPG pulse signal is then logarithmically (log10) transformed to suppress low-amplitude interference peaks such as respiratory peaks, highlighting the dominant heart rate frequency. The frequency (argmax) corresponding to the maximum value after the logarithmic transformation is determined; this frequency is the dominant heart rate frequency. The heart rate (BPM) is obtained by multiplying this dominant heart rate frequency (Hz) by 60 and rounded to three decimal places.

[0083] Figure 6 This is a schematic diagram of the heart rate frequency of a real pulse signal provided in the embodiments of this application, such as... Figure 6 As shown, the frequency of the heart rate master frequency is 1.75Hz, and the heart rate obtained by 1.75×60 is 105.0bpm. The power value corresponding to the frequency of the heart rate master frequency is 1.43. Figure 7 This is a schematic diagram of the heart rate dominant frequency of the rPPG pulse signal provided in the embodiments of this application, as shown below. Figure 7 As shown, the frequency of the heart rate master frequency is 1.75Hz, and the heart rate obtained by 1.75×60 is 105.0bpm. The power value corresponding to the frequency of the heart rate master frequency is 2.5.

[0084] Figure 8 This is a second schematic flowchart of the heart rate estimation method provided in the embodiments of this application, as shown below. Figure 8 As shown, it includes steps 801-814.

[0085] Step 801: Read the face video frame sequence.

[0086] Step 802: Determine if there is significant shaking in the face video. If there is significant shaking in the face video, proceed to step 803; if there is no significant shaking in the face video, proceed to step 804.

[0087] Step 803: Use the target model to perform global motion compensation and stabilization processing on the face video.

[0088] Step 804: For the first face image in each frame of the face video, perform facial landmark detection on the first face image to obtain facial landmarks; determine target landmarks based on the facial landmarks; generate an affine matrix by least squares fitting based on the target landmarks, the facial landmarks and the preset target points; align the first face image based on the affine matrix to obtain the second face image.

[0089] Step 805: Perform spatial filtering on the second face image to obtain the filtered face image.

[0090] Step 806: Divide the filtered face image into regions of interest to obtain multiple regions of interest corresponding to the first face image; calculate the weighted average pixel value based on each region of interest and the preset weight corresponding to each region of interest.

[0091] Step 807: Determine the weighted average pixel value time series tensor based on the weighted average pixel values ​​corresponding to each of the first face images.

[0092] Step 808: Perform mean normalization on the multiple weighted average pixel values ​​corresponding to the weighted average pixel value time series tensor in each sliding window to obtain the processed signal.

[0093] Step 809: Project the processed signal onto two orthogonal components of the skin color space; determine the fused signal based on the two orthogonal components and the standard deviation corresponding to each orthogonal component.

[0094] Step 810: Determine the rPPG pulse signal based on the fused signals within all sliding windows.

[0095] Step 811: Determine the optimal shift offset between the rPPG pulse signal and the real pulse signal; based on the optimal shift offset, trim the overlapping part of the rPPG pulse signal and the real pulse signal to obtain the trimmed rPPG pulse signal.

[0096] Step 812: Filter the cropped rPPG pulse signal to obtain the filtered rPPG pulse signal.

[0097] Step 813: Perform peak detection on the filtered rPPG pulse signal to obtain the peak value of the filtered rPPG pulse signal; estimate the heart rate based on the interval between all adjacent peak values.

[0098] Step 814: Extract the heart rate frequency of the filtered rPPG pulse signal using FFT; estimate the heart rate based on the heart rate frequency.

[0099] This application can also calculate various evaluation indicators between the predicted heart rate value and the reference heart rate, such as mean absolute error (MAE), root mean square error (RMSE), Pearson correlation coefficient, and signal to interference plus noise ratio (SNR), for performance comparison.

[0100] The technical solution of this application significantly improves the accuracy and robustness of rPPG heart rate estimation in complex environments, with the following specific advantages: (1) The accuracy of heart rate estimation is significantly improved. By optimizing key aspects such as video stabilization, ROI extraction, signal alignment, and peak detection, this approach achieves better estimation accuracy than traditional POS methods on multiple datasets.

[0101] On the publicly available dataset UBFC-RPPG, the peak detection error is reduced by approximately 63.60% compared to the standard POS method, and the FFT heart rate estimation error is reduced by approximately 1.39%. On a self-built complex environment dataset, the peak detection error is reduced by approximately 67.01%, and the FFT heart rate estimation error is reduced by approximately 50.25%. This indicates that the proposed method achieves better accuracy in complex environments.

[0102] Table 1. Comparison Results of MAE

[0103] (2) Enhanced robustness in complex environments By assigning ROI weights and stabilizing the video, noise interference caused by facial occlusion, camera shake, and head movement is effectively reduced. Combined with affine transformation face alignment, the stability of the ROI region is significantly improved. Especially on self-built complex datasets (containing shake, occlusion, and dynamic backgrounds), the error is significantly reduced, which solves the problem of poor generalization of traditional POS methods to a certain extent and is suitable for real-world applications.

[0104] The heart rate estimation device provided in this application is described below. The heart rate estimation device described below can be referred to in correspondence with the heart rate estimation method described above.

[0105] Figure 9 This is a schematic diagram of the heart rate estimation device provided in the embodiments of this application, as shown below. Figure 9 As shown, the heart rate estimation device 900 includes: a first determining module 901, a second determining module 902, a processing module 903, and an estimation module 904; wherein, The first determining module 901 is used to determine the weighted average pixel value corresponding to the first face image for each frame of the first face image in the face video. The second determining module 902 is used to determine a weighted average pixel value time series tensor based on the weighted average pixel value corresponding to each of the first face images; Processing module 903 is used to process the weighted average pixel value time series tensor using a sliding window method to obtain the rPPG pulse signal; The estimation module 904 is used to estimate the heart rate based on the rPPG pulse signal.

[0106] The heart rate estimation device provided in this application determines the weighted average pixel value corresponding to each frame of a first face image in a face video; based on the weighted average pixel values ​​corresponding to each of the first face images, it determines a weighted average pixel value time series tensor; it processes the weighted average pixel value time series tensor using a sliding window method to obtain a remote photoplethysmography (rPPG) pulse signal; and it estimates the heart rate based on the rPPG pulse signal. By determining the weighted average pixel value time series tensor using the weighted average pixel values ​​corresponding to each of the first face images, and based on the rPPG pulse signal obtained using the sliding window method, it achieves accurate heart rate estimation, improving the accuracy and efficiency of heart rate estimation.

[0107] Optionally, the first determining module 901 is specifically used for: Determine multiple regions of interest corresponding to the first face image; The weighted average pixel value is calculated based on each region of interest and the preset weight corresponding to each region of interest.

[0108] Optionally, the first determining module 901 is further configured to: Facial landmark detection is performed on the first face image to obtain facial landmarks; Based on the aforementioned facial key points, target key points are determined; Based on target key points, facial key points and preset target points, an affine matrix is ​​generated by least squares fitting. Based on the affine matrix, the first face image is aligned to obtain the second face image; Spatial filtering is applied to the second face image to obtain the filtered face image; The filtered face image is divided into regions of interest to obtain multiple regions of interest corresponding to the first face image.

[0109] Optionally, the processing module 903 is specifically used for: The weighted average pixel values ​​corresponding to the time series tensor of the weighted average pixel value in each sliding window are subjected to mean normalization to obtain the processed signal. The processed signal is projected onto two orthogonal components of the skin color space; The rPPG pulse signal is determined based on the two orthogonal components.

[0110] Optionally, the processing module 903 is further configured to: The fused signal is determined based on the two orthogonal components and the standard deviation corresponding to each orthogonal component; The rPPG pulse signal is determined based on the fused signals within all sliding windows.

[0111] Optionally, the estimation module 904 is specifically used for: Determine the optimal shift offset between the rPPG pulse signal and the real pulse signal; Based on the optimal shift offset, the overlapping portion of the rPPG pulse signal and the real pulse signal is cropped to obtain the cropped rPPG pulse signal. The cropped rPPG pulse signal is filtered to obtain the filtered rPPG pulse signal; Heart rate is estimated based on the filtered rPPG pulse signal.

[0112] Optionally, the estimation module 904 is further configured to: Peak detection is performed on the filtered rPPG pulse signal to obtain the peak value of the filtered rPPG pulse signal; Heart rate is estimated based on the intervals between all adjacent peaks.

[0113] Optionally, the estimation module 904 is further configured to: The heart rate frequency of the filtered rPPG pulse signal was extracted using Fast Fourier Transform (FFT). The heart rate is estimated based on the stated heart rate frequency.

[0114] Optionally, the heart rate estimation device 900 further includes: The detection module is used to detect whether the face video is shaking; The removal module is used to perform global motion compensation and stabilization processing on the face video using a target model when the shaking level of the face video exceeds a preset value.

[0115] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10As shown, the electronic device may include a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute a heart rate estimation method. This method includes: for each frame of a face video, determining the weighted average pixel value corresponding to the first face image; determining a weighted average pixel value time series tensor based on the weighted average pixel values ​​corresponding to each of the first face images; processing the weighted average pixel value time series tensor using a sliding window method to obtain a remote photoplethysmography (rPPG) pulse signal; and estimating the heart rate based on the rPPG pulse signal.

[0116] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the heart rate estimation method provided by the above methods. The method includes: determining a weighted average pixel value corresponding to each frame of a first face image in a face video; determining a weighted average pixel value time series tensor based on the weighted average pixel values ​​corresponding to each of the first face images; processing the weighted average pixel value time series tensor using a sliding window to obtain a remote photoplethysmography (rPPG) pulse signal; and estimating the heart rate based on the rPPG pulse signal.

[0118] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the heart rate estimation method provided by the methods described above. The method includes: for each frame of a first face image in a face video, determining a weighted average pixel value corresponding to the first face image; determining a weighted average pixel value time series tensor based on the weighted average pixel values ​​corresponding to each of all the first face images; processing the weighted average pixel value time series tensor using a sliding window method to obtain a remote photoplethysmography (rPPG) pulse signal; and estimating the heart rate based on the rPPG pulse signal.

[0119] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A heart rate estimation method, characterized by, The method comprises: For each first face image of the face video, determining a weighted average pixel value corresponding to the first face image; Based on the weighted average pixel value corresponding to each of the first face images, a weighted average pixel value time sequence tensor is determined; The weighted average pixel value time sequence tensor is processed in a sliding window manner to obtain a remote photoplethysmography rPPG pulse signal; Based on the rPPG pulse signal, the heart rate is estimated.

2. The heart rate estimation method of claim 1, characterized in that, The determination of the weighted average pixel value corresponding to the first face image comprises: A plurality of regions of interest corresponding to the first face image are determined; Based on each region of interest and the preset weight corresponding to each region of interest, the weighted average pixel value is calculated.

3. The heart rate estimation method of claim 2, wherein, The determination of the plurality of regions of interest corresponding to the first face image comprises: Face key point detection is performed on the first face image to obtain face key points; Based on the face key points, target key points are determined; Based on the target key points, face key points, and preset target points, an affine matrix is generated by least squares fitting; Based on the affine matrix, the first face image is aligned to obtain a second face image; Spatial filtering is performed on the second face image to obtain a filtered face image; The filtered face image is divided into regions of interest to obtain a plurality of regions of interest corresponding to the first face image.

4. The heart rate estimation method of claim 1, wherein, The processing of the weighted average pixel value time sequence tensor in a sliding window manner to obtain the rPPG pulse signal comprises: The plurality of weighted average pixel values corresponding to the weighted average pixel value time sequence tensor in each sliding window are subjected to mean normalization processing to obtain a processed signal; The processed signal is projected onto two orthogonal components of a skin color space; Based on the two orthogonal components, the rPPG pulse signal is determined.

5. The heart rate estimation method of claim 4, wherein, The determination of the rPPG pulse signal based on the two orthogonal components comprises: Based on the two orthogonal components and the standard deviation corresponding to each orthogonal component, a fusion signal is determined; Based on the fusion signals in all sliding windows, the rPPG pulse signal is determined.

6. The heart rate estimation method of claim 1, wherein, The estimation of the heart rate based on the rPPG pulse signal comprises: An optimal shift offset between the rPPG pulse signal and a real pulse signal is determined; Based on the optimal shift offset, the overlapping part of the rPPG pulse signal and the real pulse signal is cropped to obtain a cropped rPPG pulse signal; The cropped rPPG pulse signal is filtered to obtain a filtered rPPG pulse signal; Based on the filtered rPPG pulse signal, the heart rate is estimated.

7. The heart rate estimation method of claim 6, wherein, The estimation of the heart rate based on the filtered rPPG pulse signal comprises: Peak value detection is performed on the filtered rPPG pulse signal to obtain the peak value of the filtered rPPG pulse signal; Based on the interval between all adjacent peak values, the heart rate is estimated.

8. The heart rate estimation method of claim 6, wherein, The estimation of the heart rate based on the filtered rPPG pulse signal comprises: The heart rate main frequency of the filtered rPPG pulse signal is extracted by fast Fourier transform FFT; estimate a heart rate based on the heart rate dominant frequency.

9. The heart rate estimation method according to any of claims 1 to 8, characterized in that, The method further includes: detecting whether the face video is shaking; in a case where a shaking degree of the face video exceeds a preset value, performing global motion compensation and stabilization processing on the face video by using a target model.

10. Heart rate estimation apparatus, characterized in that comprise: a first determination module configured to determine, for each first face image in the face video, a weighted average pixel value corresponding to the first face image; a second determination module configured to determine a weighted average pixel value time sequence tensor based on the weighted average pixel value corresponding to each of the first face images; a processing module configured to process the weighted average pixel value time sequence tensor in a sliding window manner to obtain an rPPG pulse signal; an estimation module configured to estimate a heart rate based on the rPPG pulse signal.

Images