Video heart rate measurement method based on time-frequency domain wiener filtering

Abstract

The application discloses a video heart rate measurement method based on time-frequency domain Wiener filtering, and steps of the method comprise the following steps: firstly, dividing a face into multiple ROIs and extracting a pulse signal of each ROI through a pulse extraction algorithm; then, converting the pulse signal and a facial region feature point trajectory signal into a time-frequency domain through synchronous compression wavelet transform, taking the facial region feature point trajectory signal as a motion noise reference, and applying Wiener filtering to the pulse signal for denoising in the time-frequency domain; finally, searching for a heart rate component in the time-frequency space by utilizing a quasi-periodic property of the heart rate, obtaining a time-frequency spectrum line of the heart rate, and calculating an instantaneous and average heart rate value. The application can remove motion noise existing in non-contact heart rate detection, so that more accurate heart rate measurement is obtained.

Description

Technical Field

[0001] This invention belongs to the technical field of biomedical signal processing, and particularly relates to a video heart rate measurement method based on time-frequency domain Wiener filtering. Background Technology

[0002] Heart rate, as an important physiological parameter of the human body, is of great significance for monitoring human health. Traditional clinical heart rate detection methods require contacting multiple parts of the body with the twelve leads of an electrocardiograph, which is complicated, lacks automation, and requires a high level of professional knowledge from the user, making it unsuitable for heart rate detection in special scenarios.

[0003] Photoplethysmography (PPG) is another commonly used method for heart rate detection. It's relatively simple to operate, but its biggest drawback, similar to that of an electrocardiograph (ECG), is the need for contact with the body. In recent years, with the rapid development of portable imaging devices, especially smartphones and laptops, there's a trend towards converting traditional contact PPG to imaging photoplethysmography (iPPG). The main principle of iPPG is that the periodic contraction and relaxation of the heart causes periodic changes in the volume of blood vessels on the body surface. This, in turn, leads to periodic changes in the hemoglobin content of the blood, causing variations in the absorption of incident light and ultimately, changes in the intensity of reflected light. These subtle changes are captured through video images, and heart rate can be obtained through facial recognition, blind source separation, and signal processing. Compared to PPG, iPPG's advantage lies in requiring only a low-cost digital camera, eliminating the need for a contact probe or dedicated light source, making it simple yet effective. However, most current methods for implementing iPPG technology are only applicable when the test subject remains relatively still. In practical applications, it is difficult for subjects to remain completely still; head movements (rigid motion) or facial expressions (non-rigid motion) are inevitable, introducing motion noise. Since the change in blood vessel volume caused by heartbeats leads to a much smaller change in hemoglobin absorption, the magnitude of this change is far less than the change caused by motion noise. Therefore, eliminating the influence of this motion noise is of significant research importance for the widespread application of non-contact video heart rate detection methods. Summary of the Invention

[0004] To address the shortcomings of the aforementioned technologies, this invention provides a video heart rate measurement method based on time-frequency domain Wiener filtering, aiming to remove the influence of motion noise on non-contact video heart rate detection, thereby improving the accuracy of non-contact video heart rate detection.

[0005] The present invention adopts the following technical solution to solve the technical problem:

[0006] The video heart rate measurement method based on time-frequency domain Wiener filtering of this invention is characterized by the following steps:

[0007] Step 1: Obtain T total The video images are processed in frames and then windowed to obtain W video windows; a face feature point detection algorithm is used to analyze the T values ​​within the w-th video window. window Frame-by-frame facial feature point detection is performed on the video images to obtain the set of facial feature points for each frame of the video image within the w-th video window, and K facial regions of interest are determined in each frame of the video image within the w-th video window.

[0008] Step 2: Calculate the average pixel values ​​of the R, G, and B color channels of the k-th facial region of interest in the t-th frame of the video image within the w-th video window, to obtain the color channel signal corresponding to the k-th facial region of interest in the t-th frame of the video image within the w-th window. in, This represents the signal obtained by averaging all pixel values ​​in the R color channel of the k-th facial region of interest in the t-th frame of the video image within the w-th video window. This represents the signal obtained by averaging all pixel values ​​in the G color channel of the k-th facial region of interest in the t-th frame of the video image within the w-th video window. This represents the average signal of all pixel values ​​in the B color channel of the k-th facial region of interest in the t-th frame of the video image within the w-th video window; 1≤k≤K; 1≤t≤T window 1≤w≤W; T is the transpose of the matrix; thus, we obtain T within the w-th video window. window The signal obtained by averaging all pixel values ​​in the R color channel of the k-th facial region of interest in the video frame. T in the wth video window window The signal obtained by averaging all pixel values ​​in the G color channel of the k-th facial region of interest in the video frame. T in the wth video window window The signal obtained by averaging all pixel values ​​in the B color channel of the k-th facial region of interest in the video frame.

[0009] Step 3: Use the pulse extraction algorithm to extract the pulses within the w-th video window. window R signal of the k-th facial region of interest in the video frame image G signal B signal Processing is performed to obtain T within the w-th video window. window Noisy impulse signal of the k-th facial region of interest in a frame of video image

[0010] Step 4: Based on T in the w-th video window window The motion trajectory of the facial feature points in the k-th region of interest of the face in the video frame is defined as T within the w-th video window. window Motion reference signal of the k-th facial region of interest in a frame of video image Among them, M w，k (t) represents the motion reference signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window. This represents the motion trajectory signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window, along the X-axis. This represents the motion trajectory signal of the k-th facial region of interest in the t-th frame of a video image within w video windows, along the Y-axis. This represents the motion trajectory signal of the k-th facial region of interest in the t-th frame of the video image within w video windows, along the Z-axis, thus obtaining the T-th frame of the video window. window The motion trajectory signal of the k-th facial region of interest in a frame of video image on the X-axis. T in the wth video window window The motion trajectory signal of the k-th facial region of interest in the video frame along the Y-axis. Indicates T within the w-th video window window The motion trajectory signal of the k-th facial region of interest in a frame of video image along the Z-axis.

[0011] Step 5: Use synchronous compressed wavelet transform to separate the noisy pulse signals. and motion reference signal By converting to the time-frequency domain, the time-frequency spectrum of the noisy pulse signal is obtained. Time spectrum of the moving reference signal Among them, Sx w，k (t) represents the time spectrum of the noisy impulse signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window, Sn w，k (t) represents the time spectrum of the motion reference signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window;

[0012] Step 6: For each and After normalization, the normalized time-frequency spectrum of the noisy pulse signal is obtained. The time spectrum after normalization of the motion reference signal Among them, SX w，k (t) represents the normalized time-frequency spectrum of the noisy impulse signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window, SN w，k(t) represents the normalized time-frequency spectrum of the motion reference signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window;

[0013] Step 7: Calculate the weights of the first time-frequency domain Wiener filter F1 and the second time-frequency domain Wiener filter F2 using equations (1) and (2), respectively:

[0014]

[0015]

[0016] In equations (1) and (2), C is a constant; F1 w，k (t) represents the weights of filter F1 for the k-th facial region of interest in the t-th frame of the video image within the w-th video window; F2 w，k (t) represents the weight of filter F2 for the k-th facial region of interest in the t-th frame of the video image within the w-th video window; SX i，k (t) represents the normalized time-frequency spectrum of the noisy impulse signal of the k-th facial region of interest in the t-th frame of the video image within the i-th video window, SN i，k (t) represents the normalized time-frequency spectrum of the motion reference signal of the k-th facial region of interest in the t-th frame of the video image within the i-th video window, F2. i，t (t) represents the weight of filter F2 for the k-th facial region of interest in the t-th frame of the video image within the i-th video window;

[0017] Step 8: Combine the weights of F1 and F2 using equation (3) to obtain the weights of filter F3;

[0018] F3 w，k (t)=a*F1 w，t (t)+b*F2 w，t (t) (3)

[0019] In equation (4), F3 w，k (t) represents the weight of filter F3 in the k-th facial region of interest of the t-th frame video image within the w-th video window; a represents the weight of filter F1 in F3, and b represents the weight of filter F2 in F3;

[0020] Step 9: Use equation (4) to obtain the time spectrum of the k-th facial region of interest in the t-th frame of the video image within the w-th video window. The clean time spectrum SC after filtering is then obtained. w，k (t):

[0021] SC w，k (t)=F3 w，t (t)·SX w，t (t) (4)

[0022] Step 10: For SC w，k (t) After windowing again, the time-spectral graphs of N sub-windows of the k-th facial region of interest in the t-th frame video image within the w-th video window are obtained. in This represents the time-spectral graph of the nth sub-window of the k-th facial region of interest in the t-th frame of the video image within the w-th video window; thus, the time-spectral graph of the T-th frame within the w-th video window is obtained. window Temporal spectrogram of all sub-windows under the k-th facial region of interest in a frame of video image

[0023] Step 11: From Extract the Q-root time-frequency ridge with the highest energy from the time-frequency plot of the nth sub-window of the k-th facial region of interest within the w-th video window of the t-th frame. in, Let Q be the highest-energy time-frequency ridge in the time-spectrum graph of the nth sub-window of the k-th facial region of interest in the video image of the t-th frame within the w-th video window. The variance of each ridge is calculated to obtain the variance of the Q highest-energy time-frequency ridges in the time-spectrum graph of the nth sub-window of the k-th facial region of interest in the w-th video window. in, This represents the variance of the q-th time-frequency ridge with the highest energy in the time-spectrum graph of the n-th sub-window of the k-th facial region of interest within the w-th video window. The minimum variance value is selected as the variance of the temporal spectrogram of the nth sub-window of the k-th facial region of interest within the w-th video window, denoted as .

[0024] Step 12: Sum the minimum variance values ​​of the N sub-windows of the k-th facial region of interest under the w-th video window to obtain the variance (VAR) of the k-th facial region of interest within the w-th video window. w，k The enhanced time spectrum MS of the video image in the t-th frame within the w-th video window is obtained by summing the time spectra of the top Δ facial regions of interest with the smallest variances among the K facial regions of interest within the w-th video window. w (t); where Δ represents the threshold;

[0025] Step 13: Enhance the time-frequency spectrum (MS) of the t-th frame video image within the w-th time-frequency window. w (t) After windowing again, the time-spectrum diagrams of the N sub-windows of the t-th frame video image within the w-th video window are obtained. in, The time-spectral graph representing the nth sub-window of the t-th frame video image within the w-th video window;

[0026] Step 14: From Extract the Q-root time-frequency ridge with the highest energy from the time-frequency plot of the nth sub-window within the w-th video window, specifically from the t-th frame of the video image. in, Let Q represent the highest-energy time-frequency ridge in the time-spectrum graph of the nth sub-window within the w-th video window (frame t). After calculating the variance of each ridge, we obtain the variance of the Q highest-energy time-frequency ridge in the time-spectrum graph of the nth sub-window within the w-th video window. in, This represents the variance of the q-th time-frequency ridge with the highest energy in the time-frequency spectrum of the n-th sub-window within the w-th video window. Select the frequency ridge line corresponding to the minimum variance and use it as the frequency ridge line of the nth sub-window of the t-th frame video image within the w-th video window, denoted as

[0027] Step 15: Extract the frequency ridges of the N sub-windows of the t-th frame video image within the w-th video window. After sequential concatenation, the minimum variance time-frequency ridge FF of the t-th frame video image within the w-th video window is obtained. w (t);

[0028] The minimum variance time-frequency ridge FF w The median of (t) is used as the heart rate frequency hr for the w-th video window. w Thus, the heart rate value (HR) of the w-th video window is obtained. w .

[0029] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the video heart rate measurement method, and the processor is configured to execute the program stored in the memory.

[0030] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, performs the steps of the video heart rate measurement method.

[0031] Compared with existing technologies, the beneficial effects of this invention are reflected in:

[0032] 1. This invention extracts coarse pulse signals using the POS algorithm and uses facial feature point trajectories as motion reference signals. The WSST transform is used to convert the coarse pulse signals and motion reference signals to the time-frequency domain, and time-frequency Wiener filtering is employed to eliminate rigid and non-rigid facial movements. Compared to traditional optical model-based methods, this invention achieves robust heart rate measurement even with strong motion noise.

[0033] 2. This invention uses a multi-region joint time-frequency domain Wiener filtering method to extract heart rate components. Since the non-rigid motion noise interference is different in different regions, while the heart rate components are the same, the facial region of interest is divided into multiple regions. The time-frequency spectrum after removing rigid motion noise is filtered according to the continuity of heart rate to obtain the average time-frequency spectrum. This not only effectively extracts heart rate components, but also obtains instantaneous heart rate. Attached Figure Description

[0034] Figure 1 This is a flowchart of the method of the present invention;

[0035] Figure 2 This is a schematic diagram of facial feature points used in the present invention to divide the region of interest on the face;

[0036] Figure 3 This is a schematic diagram illustrating the division of the facial region of interest according to the present invention. Detailed Implementation

[0037] In this embodiment, a video heart rate measurement method based on time-frequency domain Wiener filtering first divides the face into several sub-regions. For each sub-region, a noisy pulse signal is extracted using the POS algorithm. The coordinate motion trajectory of the facial feature points corresponding to that sub-region is used as a motion reference signal. Then, synchronous compressed wavelet transform is used to convert the noisy pulse signal and the motion reference signal to the time-frequency space. Time-frequency domain Wiener filtering is then introduced to further denoise the noisy pulse signal obtained from POS using anti-motion rPPG. Finally, utilizing the quasi-periodic property of heart rate, heart rate components are searched in the filtered time-frequency spectrum. Time-frequency ridges are extracted from the time-frequency spectrum of selected facial sub-regions with good signal quality to estimate instantaneous and average heart rates. Specifically, as... Figure 1 As shown, the method includes:

[0038] Step 1: Obtain T total The video images are framed and windowed to obtain W video windows; for example... Figure 2 As shown, the Mediapipe method is used to process the T within the w-th video window. window Frame-by-frame facial feature point detection is performed on the video images to obtain the set of facial feature points for each frame of the video image within the w-th video window, such as... Figure 3 As shown, K facial interest regions are uniformly distributed on the face in each frame of video image within the w-th video window, and K is 145.

[0039] Step 2: Calculate the average pixel values ​​of the R, G, and B color channels of the k-th facial region of interest in the t-th frame of the video image within the w-th video window, to obtain the color channel signal corresponding to the k-th facial region of interest in the t-th frame of the video image within the w-th window. in, This represents the signal obtained by averaging all pixel values ​​in the R color channel of the k-th facial region of interest in the t-th frame of the video image within the w-th video window. This represents the signal obtained by averaging all pixel values ​​in the G color channel of the k-th facial region of interest in the t-th frame of the video image within the w-th video window. This represents the average signal of all pixel values ​​in the B color channel of the k-th facial region of interest in the t-th frame of the video image within the w-th video window; 1≤k≤K; 1≤t≤T window 1≤w≤W; T is the transpose of the matrix; thus, we obtain T within the w-th video window. window The signal obtained by averaging all pixel values ​​in the R color channel of the k-th facial region of interest in the video frame. T in the wth video window window The signal obtained by averaging all pixel values ​​in the G color channel of the k-th facial region of interest in the video frame. T in the wth video window window The signal obtained by averaging all pixel values ​​in the B color channel of the k-th facial region of interest in the video frame.

[0040] Step 3: Use common signal processing methods to initially extract the noisy pulse signal from the RGB signal. This is to eliminate the influence of non-motion factors on the signal and improve the signal-to-noise ratio of the heart rate signal. Use the POS algorithm to process the T signals within the w-th video window. window R signal of the k-th facial region of interest in the video frame image G signal B signal Processing is performed to obtain T within the w-th video window. window Noisy impulse signal of the k-th facial region of interest in a frame of video image

[0041] Step 4: Based on T in the w-th video window window The motion trajectory of the facial feature points in the k-th region of interest of the face in the video frame is defined as T within the w-th video window. window Motion reference signal of the k-th facial region of interest in a frame of video image Among them, M w，k (t) represents the motion reference signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window. This represents the motion trajectory signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window, along the X-axis. This represents the motion trajectory signal of the k-th facial region of interest in the t-th frame of a video image within w video windows, along the Y-axis. This represents the motion trajectory signal of the k-th facial region of interest in the t-th frame of the video image within w video windows, along the Z-axis, thus obtaining the T-th frame of the video window. window The motion trajectory signal of the k-th facial region of interest in a frame of video image on the X-axis. T in the wth video window window The motion trajectory signal of the k-th facial region of interest in the video frame along the Y-axis. Indicates T within the w-th video window window The motion trajectory signal of the k-th facial region of interest in a frame of video image along the Z-axis.

[0042] Step 5: Synchronous compressed wavelet transform is a time-frequency transformation method that can transform signals from the time domain to the time-frequency domain. Compared with other time-frequency transformation methods, it has higher time-frequency resolution. The synchronous compressed wavelet transform method is used to transform the noisy pulse signal... and motion reference signal By converting to the time-frequency domain, the time-frequency spectrum of the noisy pulse signal is obtained. Time spectrum of the moving reference signal Among them, Sx w，k (t) represents the time spectrum of the noisy impulse signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window, Sn w，k (t) represents the time spectrum of the motion reference signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window;

[0043] Step 6: For each and After normalization, the normalized time-frequency spectrum of the noisy pulse signal is obtained. The time spectrum after normalization of the motion reference signal Among them, SX w，k (t) represents the normalized time-frequency spectrum of the noisy impulse signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window, SN w，k (t) represents the normalized time-frequency spectrum of the motion reference signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window;

[0044] Step 7: The Wiener filter is a common speech denoising algorithm, widely used in the field of speech denoising. The weights of the first time-frequency domain Wiener filter F1 and the second time-frequency domain Wiener filter F2 are calculated using equations (1) and (2), respectively:

[0045]

[0046]

[0047] In equations (1) and (2), C is a constant; the weights of the F1 filter are determined by spectral subtraction, and the weights of the F2 filter are obtained by recursively calculating the parameters of the previous filter. w，k (t) represents the weights of filter F1 for the k-th facial region of interest in the t-th frame of the video image within the w-th video window; F2 w，k (t) represents the weight of filter F2 for the k-th facial region of interest in the t-th frame of the video image within the w-th video window; SX i，k (t) represents the normalized time-frequency spectrum of the noisy impulse signal of the k-th facial region of interest in the t-th frame of the video image within the i-th video window, SN i，k (t) represents the normalized time-frequency spectrum of the motion reference signal of the k-th facial region of interest in the t-th frame of the video image within the i-th video window, F2. i，t (t) represents the weight of filter F2 for the k-th facial region of interest in the t-th frame of the video image within the i-th video window.

[0048] Step 8: Combine the weights of F1 and F2 using equation (3) to obtain the weights of filter F3;

[0049] F3 w，k (t)=a*F1 w，t (t)+b*F2 w，t (t) (3)

[0050] In equation (4), F3 w，k (t) represents the weight of filter F3 in the k-th facial region of interest of the t-th frame video image within the w-th video window; a represents the weight of filter F1 in F3, a equals , and b represents the weight of filter F2 in F3.

[0051] Step 9: Use equation (4) to obtain the time spectrum of the k-th facial region of interest in the t-th frame of the video image within the w-th video window. The clean time spectrum SC after filtering is then obtained. w，k (t):

[0052] SC w，k (t)=F3 w，t (t)·SX w，t (t) (4)

[0053] Step 10: For SC w，k (t) After windowing again, the time-spectral graphs of N sub-windows of the k-th facial region of interest in the t-th frame video image within the w-th video window are obtained. in The time-spectral graph represents the nth sub-window of the k-th facial region of interest in the t-th frame of the video image within the w-th video window;

[0054] Thus, the T value within the w-th video window is obtained. window Temporal spectrogram of all sub-windows under the k-th facial region of interest in a frame of video image

[0055] Step 11: Based on the continuity and quasi-periodic characteristics of heart rate, the heart rate should be a continuous and smooth curve in the time-frequency domain. Noise, however, often has complex frequency components and short durations, appearing as discontinuous points in the time-frequency domain. Considering the difference in the morphology of heart rate components and noise in the time-frequency domain, variance is used to identify and filter heart rate components. From Extract the Q-root time-frequency ridge with the highest energy from the time-frequency plot of the nth sub-window of the k-th facial region of interest within the w-th video window of the t-th frame. in, Let Q be the highest-energy time-frequency ridge in the time-spectrum graph of the nth sub-window of the k-th facial region of interest in the video image of the t-th frame within the w-th video window. The variance of each ridge is calculated to obtain the variance of the Q highest-energy time-frequency ridges in the time-spectrum graph of the nth sub-window of the k-th facial region of interest in the w-th video window. in, This represents the variance of the q-th time-frequency ridge with the highest energy in the time-frequency spectrum of the n-th sub-window of the k-th facial region of interest within the w-th video window. The minimum variance value is selected as the variance of the temporal spectrum of the nth sub-window of the k-th facial region of interest in the t-th frame of the video image within the w-th video window, denoted as .

[0056] Step 12: Sum the minimum variance values ​​of the N sub-windows of the k-th facial region of interest under the w-th video window to obtain the variance (VAR) of the k-th facial region of interest in the t-th frame of the video image within the w-th video window. w，k The enhanced time spectrum MS of the video image in the t-th frame within the w-th video window is obtained by summing the time spectra of the top Δ facial regions of interest with the smallest variances among the K facial regions of interest within the w-th video window. w (t); where Δ represents the threshold, and in this embodiment, Δ = 40.

[0057] Step 13: Enhance the time-frequency spectrum (MS) of the t-th frame video image within the w-th time-frequency window. w (t) After windowing again, the time-spectral graphs of the N sub-windows of the t-th frame video image within the w-th video window are obtained. in, The time-spectral graph representing the nth sub-window of the t-th frame video image within the w-th video window;

[0058] Step 14: From Extract the Q-root time-frequency ridge with the highest energy from the time-frequency plot of the nth sub-window within the w-th video window, specifically from the t-th frame of the video image. in, Let Q represent the highest-energy time-frequency ridge in the time-spectrum graph of the nth sub-window within the w-th video window (frame t). After calculating the variance of each ridge, we obtain the variance of the Q highest-energy time-frequency ridge in the time-spectrum graph of the nth sub-window within the w-th video window. in, This represents the variance of the q-th time-frequency ridge with the highest energy in the time-frequency spectrum of the n-th sub-window within the w-th video window. Select the frequency ridge line corresponding to the minimum variance and use it as the frequency ridge line of the nth sub-window of the t-th frame video image within the w-th video window, denoted as

[0059] Step 15: Extract the frequency ridges of the N sub-windows of the t-th frame video image within the w-th video window. After sequential concatenation, the minimum variance time-frequency ridge FF of the t-th frame video image within the w-th video window is obtained. w (t); the FF obtained at this time w (t) represents the instantaneous heart rate component; we obtain the mean heart rate by calculating its median. The minimum variance time-frequency ridge FF is then used. w The median of (t) is used as the heart rate frequency hr for the w-th video window. w Thus, the heart rate value HR of the w-th video window is obtained. w .

[0060] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0061] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

Claims

1. A video heart rate measurement method based on time-frequency domain Wiener filtering, characterized in that, The procedure is as follows: Step 1: Obtain T total The video images are processed by dividing them into frames and then windowing them to obtain W video windows; A face feature point detection algorithm is used to analyze the T-values ​​within the w-th video window. window Frame-by-frame facial feature point detection is performed on the video images to obtain the set of facial feature points for each frame of the video image within the w-th video window, and K facial regions of interest are determined in each frame of the video image within the w-th video window. Step 2: Calculate the average pixel values ​​of the R, G, and B color channels of the k-th facial region of interest in the t-th frame of the video image within the w-th video window, to obtain the color channel signal corresponding to the k-th facial region of interest in the t-th frame of the video image within the w-th window. in, This represents the signal obtained by averaging all pixel values ​​in the R color channel of the k-th facial region of interest in the t-th frame of the video image within the w-th video window. This represents the signal obtained by averaging all pixel values ​​in the G color channel of the k-th facial region of interest in the t-th frame of the video image within the w-th video window. This represents the average signal of all pixel values ​​in the B color channel of the k-th facial region of interest in the t-th frame of the video image within the w-th video window; 1≤k≤K; 1≤t≤T window 1≤w≤W; T is the transpose of the matrix; thus, we obtain T within the w-th video window. window The signal obtained by averaging all pixel values ​​in the R color channel of the k-th facial region of interest in the video frame. T in the wth video window window The signal obtained by averaging all pixel values ​​in the G color channel of the k-th facial region of interest in the video frame. T in the wth video window window The signal obtained by averaging all pixel values ​​in the B color channel of the k-th facial region of interest in the video frame. Step 3: Use the pulse extraction algorithm to extract the pulses within the w-th video window. window R signal of the k-th facial region of interest in the video frame image G signal B signal Processing is performed to obtain T within the w-th video window. window Noisy impulse signal of the k-th facial region of interest in a frame of video image Step 4: Based on T within the w-th video window window The motion trajectory of the facial feature points in the k-th region of interest of the face in the video image is defined, and T is defined within the w-th video window. window Motion reference signal of the k-th facial region of interest in a frame of video image Among them, M w,k (t) represents the motion reference signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window. This represents the motion trajectory signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window, along the X-axis. This represents the motion trajectory signal of the k-th facial region of interest in the t-th frame of a video image within w video windows, along the Y-axis. This represents the motion trajectory signal of the k-th facial region of interest in the t-th frame of the video image within w video windows, along the Z-axis, thus obtaining the T-th frame of the video window. window The motion trajectory signal of the k-th facial region of interest in a frame of video image on the X-axis. T in the wth video window window The motion trajectory signal of the k-th facial region of interest in the video frame along the Y-axis. Indicates T within the w-th video window window The motion trajectory signal of the k-th facial region of interest in a frame of video image along the Z-axis. Step 5: Use synchronous compressed wavelet transform to separate the noisy pulse signals. and motion reference signal By converting to the time-frequency domain, the time-frequency spectrum of the noisy pulse signal is obtained. Time spectrum of the moving reference signal Among them, Sx w,k (t) represents the time spectrum of the noisy impulse signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window, Sn w,k (t) represents the time spectrum of the motion reference signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window; Step 6: For each and After normalization, the normalized time-frequency spectrum of the noisy pulse signal is obtained. The time spectrum after normalization of the motion reference signal Among them, SX w,k (t) represents the normalized time-frequency spectrum of the noisy impulse signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window, SN w,k (t) represents the normalized time-frequency spectrum of the motion reference signal of the k-th facial region of interest in the t-th frame of the video image within the w-th video window; Step 7: Calculate the weights of the first time-frequency domain Wiener filter F1 and the second time-frequency domain Wiener filter F2 using equations (1) and (2), respectively: In equations (1) and (2), C is a constant; F1 w,k (t) represents the weights of filter F1 for the k-th facial region of interest in the t-th frame of the video image within the w-th video window; F2 w,k (t) represents the weight of filter F2 for the k-th facial region of interest in the t-th frame of the video image within the w-th video window; SX i,k (t) represents the normalized time-frequency spectrum of the noisy impulse signal of the k-th facial region of interest in the t-th frame of the video image within the i-th video window, SN i,k (t) represents the normalized time-frequency spectrum of the motion reference signal of the k-th facial region of interest in the t-th frame of the video image within the i-th video window, F2. i,t (t) represents the weight of filter F2 for the k-th facial region of interest in the t-th frame of the video image within the i-th video window; Step 8: Combine the weights of F1 and F2 using equation (3) to obtain the weights of filter F3; F3 w,k (t)=a*F1 w,t (t)+b*F2 w,t (t) (3) In equation (4), F3 w,k (t) represents the weight of filter F3 in the k-th facial region of interest of the t-th frame video image within the w-th video window; a represents the weight of filter F1 in F3, and b represents the weight of filter F2 in F3; Step 9: Use equation (4) to obtain the time spectrum of the k-th facial region of interest in the t-th frame of the video image within the w-th video window. The clean time spectrum SC after filtering is then obtained. w,k (t): SC w,k (t)=F3 w,t (t)·SX w,t (t) (4) Step 10: For SC w,k (t) After windowing again, the time-spectral graphs of N sub-windows of the k-th facial region of interest in the t-th frame video image within the w-th video window are obtained. in This represents the time-spectral graph of the nth sub-window of the k-th facial region of interest in the t-th frame of the video image within the w-th video window; thus, the time-spectral graph of the T-th frame within the w-th video window is obtained. window Temporal spectrogram of all sub-windows under the k-th facial region of interest in a frame of video image Step 11: From Extract the Q-root time-frequency ridge with the highest energy from the time-frequency plot of the nth sub-window of the k-th facial region of interest within the w-th video window of the t-th frame. in, Let Q be the highest-energy time-frequency ridge in the time-spectrum graph of the nth sub-window of the k-th facial region of interest in the video image of the t-th frame within the w-th video window. The variance of each ridge is calculated to obtain the variance of the Q highest-energy time-frequency ridges in the time-spectrum graph of the nth sub-window of the k-th facial region of interest in the w-th video window. in, This represents the variance of the q-th time-frequency ridge with the highest energy in the time-frequency spectrum of the n-th sub-window of the k-th facial region of interest within the w-th video window. The minimum variance value is selected as the variance of the temporal spectrum of the nth sub-window of the k-th facial region of interest within the w-th video window, denoted as . Step 12: Sum the minimum variance values ​​of the N sub-windows of the k-th facial region of interest under the w-th video window to obtain the variance (VAR) of the k-th facial region of interest within the w-th video window. w,k The enhanced time spectrum MS of the video image in the t-th frame within the w-th video window is obtained by summing the time spectra of the top Δ facial regions of interest with the smallest variances among the K facial regions of interest within the w-th video window. w (t); where Δ represents the threshold; Step 13: Enhance the time-frequency spectrum (MS) of the t-th frame video image within the w-th time-frequency window. w (t) After windowing again, the time-spectral graphs of the N sub-windows of the t-th frame video image within the w-th video window are obtained. in, The time-spectral graph representing the nth sub-window of the t-th frame video image within the w-th video window; Step 14: From Extract the Q-root time-frequency ridge with the highest energy from the time-frequency plot of the nth sub-window within the w-th video window, specifically from the t-th frame of the video image. in, Let Q represent the highest-energy time-frequency ridge in the time-spectrum graph of the nth sub-window within the w-th video window (frame t). After calculating the variance of each ridge, we obtain the variance of the Q highest-energy time-frequency ridge in the time-spectrum graph of the nth sub-window within the w-th video window. in, This represents the variance of the q-th time-frequency ridge with the highest energy in the time-frequency spectrum of the n-th sub-window within the w-th video window. Select the frequency ridge line corresponding to the minimum variance and use it as the frequency ridge line of the nth sub-window of the t-th frame video image within the w-th video window, denoted as Step 15: Extract the frequency ridges of the N sub-windows of the t-th frame video image within the w-th video window. After sequential concatenation, the minimum variance time-frequency ridge FF of the t-th frame video image within the w-th video window is obtained. w (t); The minimum variance time-frequency ridge FF w The median of (t) is used as the heart rate frequency hr for the w-th video window. w Thus, the heart rate value (HR) of the w-th video window is obtained. w .

2. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the video heart rate measurement method of claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the video heart rate measurement method of claim 1.

Images