Heartbeat estimation method and device based on face video RPPG signal

Abstract

The invention provides a heartbeat estimation method and device based on a face video RPPG signal, and relates to the field of physiological parameter estimation, and the method comprises the steps: obtaining each frame of RPPG signal based on an RGB value from a to-be-detected face video; performing principal component analysis on the RPPG signal based on the RGB value in the preset time window to obtain a pulse RPPG signal; converting the pulse RPPG signal from a time domain to a frequency domain, and determining a heartbeat estimated value in a heartbeat effective frequency interval; determining a signal-to-noise ratio according to the pulse RPPG signal and the heartbeat estimation value; and outputting a heartbeat estimation value corresponding to the pulse RPPG signal under the condition that the signal-to-noise ratio is greater than a preset threshold value. The heartbeat estimation accuracy is effectively improved, low-quality signals can be reliably filtered out, wrong heart rate data are prevented from being output, and the overall performance and practicability of the system are remarkably improved.

Description

Technical Field

[0001] The present disclosure relates to the field of physiological parameter estimation, and in particular to a heartbeat estimation method and device based on RPPG signals of face videos. Background Art

[0002] In recent years, the technology of estimating heartbeat based on remote photoplethysmography (RPPG) signals of face videos has attracted extensive attention and research. It can monitor heart rate contactlessly by capturing tiny color changes caused by facial blood flow, which has the advantages of convenience and non-invasiveness.

[0003] However, the RPPG signal processing in the related art fails to make full use of advanced signal processing techniques to improve the accuracy of heartbeat estimation, nor does it set an effective mechanism to determine the confidence of the estimated heartbeat. When environmental factors or other external interference affect the quality of the RPPG signal, the system is often unable to accurately determine which heartbeat estimates are reliable, and may output erroneous or inaccurate heart rate data. Therefore, an improved heartbeat estimation method is urgently needed to improve the accuracy of heartbeat estimation and effectively manage signal quality issues. Summary of the invention

[0004] The embodiments of the present disclosure provide a heartbeat estimation method and device based on RPPG signals of face videos, aiming to solve the problems existing in the above-mentioned background technologies.

[0005] In order to solve the above technical problems, the present disclosure is implemented as follows:

[0006] In a first aspect, an embodiment of the present disclosure provides a heartbeat estimation method based on a face video RPPG signal, the method comprising:

[0007] From the face video to be detected, obtain the RPPG signal based on the RGB value of each frame;

[0008] Performing principal component analysis on the RPPG signal based on RGB values ​​within a preset time window to obtain a pulse RPPG signal;

[0009] Transforming the pulse RPPG signal from the time domain to the frequency domain, and determining a heartbeat estimation value within a heartbeat effective frequency interval;

[0010] Determining a signal-to-noise ratio based on the pulse RPPG signal and the heartbeat estimate;

[0011] When the signal-to-noise ratio is greater than a preset threshold, the heartbeat estimation value corresponding to the pulse RPPG signal is output.

[0012] Optionally, obtaining the RPPG signal based on the RGB value of each frame from the face video to be detected includes:

[0013] Get the face video to be detected;

[0014] Performing face detection on the face video frame by frame to determine face regions of multiple frames;

[0015] Perform skin segmentation on the face area of ​​each frame to obtain multiple valid skin pixels in each frame;

[0016] The RGB values ​​of multiple valid skin pixels in each frame are averaged to obtain an RPPG signal based on the RGB value of each frame.

[0017] Optionally, performing principal component analysis on the RPPG signal based on RGB values ​​within a preset time window to obtain the pulse RPPG signal includes:

[0018] Organizing the RPPG signal based on the RGB value of each frame in a preset time window into an RPPG signal matrix;

[0019] The principal component analysis is performed on the RPPG signal matrix to convert the three-dimensional RPPG signal based on RGB values ​​into a one-dimensional pulse RPPG signal.

[0020] Optionally, transforming the pulse RPPG signal from the time domain to the frequency domain, and determining a heartbeat estimation value within a heartbeat effective frequency interval, comprises:

[0021] The pulse RPPG signal is converted from the time domain to the frequency domain by fast Fourier transform to obtain a corresponding pulse spectrum signal;

[0022] Determine the peak frequency corresponding to the peak value of the pulse spectrum signal within the effective heartbeat frequency interval;

[0023] A heartbeat estimate is determined based on the peak frequency.

[0024] Optionally, determining a signal-to-noise ratio according to the pulse RPPG signal and the heartbeat estimation value comprises:

[0025] In the pulse spectrum signal, determining the peak frequency corresponding to the heartbeat estimation value as the fundamental frequency;

[0026] According to the fundamental frequency, a plurality of harmonics are determined from the pulse spectrum signal, wherein the frequency difference of each harmonic is an integer multiple of the fundamental frequency;

[0027] Calculating the spectrum values ​​of the fundamental frequency and each harmonic, and superimposing the spectrum values ​​of the fundamental frequency and each harmonic to obtain a synthetic pulse signal;

[0028] subtracting the synthesized pulse signal from the pulse RPPG signal to extract a noise signal;

[0029] The signal-to-noise ratio is calculated based on the synthesized pulse signal and the noise signal using an SNR algorithm.

[0030] Optionally, when the signal-to-noise ratio is greater than a preset threshold, outputting a heartbeat estimation value corresponding to the pulse RPPG signal includes:

[0031] When the signal-to-noise ratio is greater than the preset threshold, marking the heartbeat estimation value corresponding to the pulse RPPG signal as a valid heartbeat, and outputting the valid heartbeat;

[0032] When the signal-to-noise ratio is less than the preset threshold, the heartbeat estimation value corresponding to the pulse RPPG signal is marked as an invalid heartbeat, the invalid heartbeat is discarded, and the last determined valid heartbeat is output.

[0033] Optionally, performing face detection on the face video frame by frame to determine face regions of multiple frames includes:

[0034] Using a lightweight Yunet algorithm to perform face detection on each frame of the face video to determine the position and boundary of the face area in each frame;

[0035] According to the position and boundary of the face region of each frame, the face regions of multiple frames are determined.

[0036] Optionally, skin segmentation is performed on the face region of each frame to obtain multiple valid skin pixels of each frame, including:

[0037] Use skin segmentation algorithm to classify pixels in the face area of ​​each frame to distinguish skin area from non-skin area;

[0038] Based on the color and texture features, effective skin pixels in each frame of the image are extracted in the skin area.

[0039] In a second aspect, an embodiment of the present disclosure provides a heartbeat estimation device based on a face video RPPG signal, the device comprising:

[0040] An acquisition module is used to obtain the RPPG signal based on the RGB value of each frame from the face video to be detected;

[0041] An analysis module is used to perform principal component analysis on the RPPG signal based on the RGB value within a preset time window to obtain a pulse RPPG signal;

[0042] A determination module, used to transform the pulse RPPG signal from the time domain to the frequency domain, and determine the heartbeat estimation value within the heartbeat effective frequency interval;

[0043] A confidence module, used to determine a signal-to-noise ratio based on the pulse RPPG signal and the heartbeat estimation value;

[0044] The output module is used to output the heartbeat estimation value corresponding to the pulse RPPG signal when the signal-to-noise ratio is greater than a preset threshold.

[0045] Optionally, the acquisition module includes:

[0046] The face video acquisition submodule is used to acquire the face video to be detected;

[0047] A face region determination submodule, used to perform face detection on the face video frame by frame to determine face regions of multiple frames;

[0048] The effective skin pixel determination submodule is used to perform skin segmentation on the face area of ​​each frame to obtain multiple effective skin pixels of each frame;

[0049] The mean calculation submodule is used to calculate the mean of the RGB values ​​of multiple valid skin pixels in each frame to obtain an RPPG signal based on the RGB value of each frame.

[0050] Optionally, the analysis module includes:

[0051] An organization submodule, used for organizing the RPPG signal based on the RGB value of each frame in a preset time window into an RPPG signal matrix;

[0052] The conversion submodule is used to perform principal component analysis on the RPPG signal matrix and convert the three-dimensional RPPG signal based on RGB values ​​into a one-dimensional pulse RPPG signal.

[0053] Optionally, the determining module includes:

[0054] A conversion submodule, used to convert the pulse RPPG signal from the time domain to the frequency domain through fast Fourier transform to obtain a corresponding pulse spectrum signal;

[0055] A peak frequency determination submodule, used to determine the peak frequency corresponding to the peak value of the pulse spectrum signal within the effective heartbeat frequency interval;

[0056] The heartbeat estimation value determination submodule is used to determine the heartbeat estimation value according to the peak frequency.

[0057] Optionally, the confidence module includes:

[0058] A fundamental frequency determination submodule, used for determining the peak frequency corresponding to the heartbeat estimation value in the pulse spectrum signal as the fundamental frequency;

[0059] A harmonic determination submodule, used to determine a plurality of harmonics from the pulse spectrum signal according to the fundamental frequency, wherein the frequency difference of each harmonic is an integer multiple of the fundamental frequency;

[0060] A superposition submodule, used for calculating the spectrum values ​​of the fundamental frequency and each harmonic, and superimposing the spectrum values ​​of the fundamental frequency and each harmonic to obtain a synthetic pulse signal;

[0061] A noise extraction submodule, used for subtracting the synthesized pulse signal from the pulse RPPG signal to extract a noise signal;

[0062] The signal-to-noise ratio calculation submodule is used to calculate the signal-to-noise ratio according to the synthesized pulse signal and the noise signal by using an SNR algorithm.

[0063] Optionally, the output module includes:

[0064] The effective heartbeat submodule is used to mark the heartbeat estimation value corresponding to the pulse RPPG signal as a valid heartbeat when the signal-to-noise ratio is greater than the preset threshold, and output the valid heartbeat;

[0065] The invalid heartbeat submodule is used to mark the heartbeat estimation value corresponding to the pulse RPPG signal as an invalid heartbeat when the signal-to-noise ratio is less than the preset threshold, discard the invalid heartbeat, and output the last determined valid heartbeat.

[0066] Optionally, the face region determination submodule includes:

[0067] A face detection unit, used to perform face detection on each frame of the face video using a lightweight Yunet algorithm to determine the position and boundary of the face area of ​​each frame;

[0068] The face region determination unit is used to determine the face regions of multiple frames according to the position and boundary of the face region of each frame.

[0069] Optionally, the effective skin pixel determination submodule includes:

[0070] A distinguishing unit, used to classify pixels in the face area of ​​each frame using a skin segmentation algorithm to distinguish skin areas from non-skin areas;

[0071] The extraction unit is used to extract effective skin pixel points in each frame of the image in the skin area based on color and texture features.

[0072] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of a heartbeat estimation method based on RPPG signals of face video are implemented.

[0073] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0074] The heartbeat estimation method based on the face video RPPG signal proposed in the present disclosure extracts the RGB value in each frame of video and performs principal component analysis within a preset time window to obtain the pulse RPPG signal. By converting the pulse RPPG signal from the time domain to the frequency domain, the heartbeat estimation value can be accurately determined within the effective heartbeat frequency range. In addition, the present disclosure evaluates the reliability of the heartbeat estimation by calculating the signal-to-noise ratio, and outputs the heartbeat estimation value only when the signal-to-noise ratio is greater than a preset threshold. The accuracy of the heartbeat estimation is effectively improved, especially in the face of environmental interference or external noise, and low-quality signals can be reliably filtered out to prevent the output of erroneous heart rate data, which significantly improves the overall performance and practicality of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0076] Figure 1 It is a schematic diagram of the steps of a heartbeat estimation method based on RPPG signals of face video provided by an embodiment of the present disclosure;

[0077] Figure 2 It is a structural block diagram of a heartbeat estimation device based on RPPG signals of face video provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0078] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0079] The present disclosure aims to solve the problems of insufficient accuracy of heartbeat estimation and weak signal quality control in related technologies. The embodiments of the present disclosure provide a method that can stably and efficiently estimate heartbeat in a complex environment by combining video signal processing, principal component analysis and frequency domain analysis technology. Through multi-step signal processing and optimization algorithms, the present disclosure can not only accurately extract RPPG signals related to heartbeat, but also effectively evaluate the credibility of the signal to ensure that the output heartbeat estimate has high reliability. The various steps and implementation plans of the present disclosure will be described in detail below.

[0080] Figure 1 FIG. 1 is a schematic diagram of the steps of a heartbeat estimation method based on RPPG signals of face video provided by an embodiment of the present disclosure. Figure 1 As shown, the method includes:

[0081] Step S101, obtaining an RPPG signal based on RGB values ​​for each frame from a face video to be detected.

[0082] Step S102, performing principal component analysis on the RPPG signal based on the RGB value within a preset time window to obtain a pulse RPPG signal.

[0083] Step S103, transforming the pulse RPPG signal from the time domain to the frequency domain, and determining a heartbeat estimation value within a heartbeat effective frequency interval.

[0084] Step S104, determining a signal-to-noise ratio according to the pulse RPPG signal and the heartbeat estimation value.

[0085] Step S105: when the signal-to-noise ratio is greater than a preset threshold, output the heartbeat estimation value corresponding to the pulse RPPG signal.

[0086] Involving step S101, the purpose is to extract each frame of RPPG (remote photoplethysmography) signal based on RGB values ​​from the face video to be detected. The RPPG signal extracts blood flow information by analyzing the slight changes in light reflected by the skin area in the video. Since each beat of the heart causes changes in the blood volume on the skin surface, these changes will affect the intensity of the light reflected from the skin. By analyzing the subtle fluctuations in the intensity of these reflected lights, physiological parameters such as heart rate can be inferred. In each frame of the face video, each pixel has a corresponding RGB (red, green, blue) value, which represents the color information of the pixel. The RPPG signal based on RGB values ​​refers to extracting the RGB color values ​​of the skin area in the video frame and analyzing its changes over time, thereby extracting physiological signals related to the heartbeat. Specifically, the skin pixels in each video frame will change the intensity of the reflected light as the blood flow changes caused by the periodic contraction and relaxation of the heart. This change manifests itself as a slight fluctuation in color value in the RGB channel, especially in the red channel, which is usually most obvious because red light can more easily penetrate the skin and be absorbed or reflected by the blood. By analyzing these fluctuations, a time-varying RPPG signal can be extracted, that is, an RPPG signal based on RGB values.

[0087] Regarding step S102, in the embodiment of the present disclosure, the preset time window length can be set to 10-30 seconds, and the specific length depends on the frame rate of the camera and the frequency range of the heart rate. For example, assuming that the frame rate of the camera is 30FPS, and the preset time window is 15 seconds, then 450 frames of video data will be accumulated within the preset time window, and the frame data can provide sufficient information for principal component analysis. It should be noted that the stability of the signal and the influence of noise must also be considered when selecting the time window. A time window that is too short may not accurately capture the periodic changes of the heartbeat, while a time window that is too long may introduce more motion artifacts and noise. Therefore, a window length of 15 seconds is an optional compromise that can provide sufficient heartbeat cycle information while avoiding excessive noise interference.

[0088] Principal component analysis (PCA) is a data dimensionality reduction technology, which is mainly used for feature extraction and removal of redundant information of high-dimensional data. In the embodiment disclosed herein, principal component analysis PCA is used to extract the most representative pulse signal component from the RPPG signal based on RGB values ​​to obtain the pulse RPPG signal, which is a signal directly related to the heartbeat cycle after noise reduction and redundancy removal. By principal component analysis, noise and irrelevant components are removed, so that the pulse RPPG signal can more accurately reflect the true situation of the heartbeat.

[0089] Involving step S103, the pulse RPPG signal is transformed from the time domain to the frequency domain. Since the pulse RPPG signal is a typical periodic signal, in the time domain, this periodicity becomes less obvious due to various noises and interferences. However, in the frequency domain, the periodic signal will appear at a specific frequency in the form of a peak, which can more intuitively identify the heartbeat frequency. Therefore, the present disclosure considers capturing the main frequency of the heartbeat in the frequency domain to determine the heartbeat estimate.

[0090] The effective heartbeat frequency range refers to the frequency range in which a person's heartbeat frequency usually appears within the normal physiological range. The effective heartbeat frequency range depends on the normal heart rate range under the physiological state of the human body. For example, the normal heart rate of an adult is between 60 and 100 beats per minute. Converting this heart rate range into a frequency range (i.e., the number of heartbeats per second) can obtain the corresponding frequency range. The frequency corresponding to 60BPM is 1Hz (1 time per second), and the frequency corresponding to 100BPM is 1.67Hz (about 1.67 times per second). Therefore, the effective heartbeat frequency range can be set between 1Hz and 1.67Hz. For specific groups of people (such as athletes, children, or the elderly), the heartbeat frequency may deviate from this range, so in practical applications, the effective heartbeat frequency range will be appropriately adjusted. For example, the heart rate of an athlete may be lower than 60BPM, while the heart rate of a child may be higher than 100BPM. After the frequency domain transformation, the main frequency is found in the effective heartbeat frequency range, and the heartbeat estimate is further determined to determine whether it is reliable in the subsequent confidence.

[0091] Involving step S104 and step S105, the signal-to-noise ratio (SNR) is a measure of the signal strength relative to the background noise strength, and is used to evaluate the quality of the signal. In the disclosed embodiment, the signal-to-noise ratio is used to distinguish between a true heartbeat signal and a noise signal. The higher the value of the signal-to-noise ratio, the better the signal quality, that is, the higher the ratio of the true heartbeat signal to the noise signal. The confidence level is determined by the SNR, and a high signal-to-noise ratio corresponds to a high confidence level, because the quality of the signal is good, and the currently estimated heartbeat estimate is more reliable. Through a large number of experiments and statistical analysis, a preset threshold value (for example, 3.1) can be determined. When the signal-to-noise ratio is greater than this threshold, the confidence level is high, and the heartbeat estimate at this time can be considered to be credible, and it is output to improve the accuracy and robustness of the overall algorithm.

[0092] The heartbeat estimation method based on the face video RPPG signal proposed in the present disclosure extracts the RGB value in each frame of video and performs principal component analysis within a preset time window to obtain the pulse RPPG signal. By converting the pulse RPPG signal from the time domain to the frequency domain, the heartbeat estimation value can be accurately determined within the effective heartbeat frequency range. In addition, the present disclosure evaluates the reliability of the heartbeat estimation by calculating the signal-to-noise ratio, and outputs the heartbeat estimation value only when the signal-to-noise ratio is greater than a preset threshold. The accuracy of the heartbeat estimation is effectively improved, especially in the face of environmental interference or external noise, and low-quality signals can be reliably filtered out to prevent the output of erroneous heart rate data, which significantly improves the overall performance and practicality of the system.

[0093] Exemplarily, the method of obtaining the RPPG signal based on RGB value for each frame from the face video to be detected includes: obtaining the face video to be detected; performing face detection on the face video frame by frame to determine face areas of multiple frames; performing skin segmentation on the face area of ​​each frame to obtain multiple valid skin pixels for each frame; and performing mean calculation on the RGB values ​​of the multiple valid skin pixels for each frame to obtain the RPPG signal based on RGB value for each frame.

[0094] Acquire real-time face video to be detected from a camera or other video acquisition device. The video can be a face video recorded under various background and lighting conditions. Video is usually acquired at multiple frames per second (e.g., 30 frames per second or higher) to capture subtle changes in the face at different times. For each frame of video image acquired, a face detection algorithm is applied to detect and determine the face area in the image. In each frame of the image, the detected face area can be represented by a rectangular frame, which defines the part of the image containing the face, and this part will be the main object of subsequent processing.

[0095] In each frame of the image, the detected face area is further segmented to distinguish the skin pixels from the non-skin pixels in the image. And among the multiple skin pixels, the valid skin pixels are determined. The valid pixels are unobstructed pixels and can be used for subsequent signal extraction, because the RPPG signal depends on the color change of the valid skin pixels. For the valid skin pixels segmented in each frame, the RGB values ​​of these valid skin pixels are extracted. The RGB value of each pixel represents the color information of the pixel, where R represents the red channel, G represents the green channel, and B represents the blue channel. The RGB values ​​of all valid skin pixels in each frame of the image are averaged, that is, the R, G, and B values ​​of all skin pixels are averaged, which will produce a single RGB triplet, representing the overall skin color of the frame of the image. The average RGB value of each frame is used as the RPPG signal value of the frame. As mentioned above, the beating of the heart causes periodic changes in the blood volume in the skin, and these changes will be reflected in the reflected light of the skin, resulting in slight fluctuations in the RGB values. By recording these fluctuations, a time-varying RPPG signal is generated, which is further used to estimate the heart rate.

[0096] Exemplarily, the principal component analysis is performed on the RPPG signal based on RGB values ​​within a preset time window to obtain a pulse RPPG signal, including: organizing each frame of the RPPG signal based on RGB values ​​within the preset time window into an RPPG signal matrix; performing principal component analysis on the RPPG signal matrix to convert the three-dimensional RPPG signal based on RGB values ​​into a one-dimensional pulse RPPG signal.

[0097] A certain amount of video frame data is accumulated within a preset time window (for example, 15 seconds). Assuming that the frame rate of the camera is 30FPS, there will be 450 frames of video data within 15 seconds. For each frame of the image, the RPPG signal based on the RGB value is a vector containing three components: R, G, and B. By arranging the RPPG signals based on the RGB values ​​of all frames within the preset time window vertically, an RPPG signal matrix in matrix form can be obtained. Assuming that each RPPG signal vector is R, G, and B, then for a time window of 450 frames, the RPPG signal matrix will be a 450x3 matrix, where 450 represents time (number of frames) and 3 represents the RGB channels.

[0098] The principal component analysis process is described below. For the three-dimensional RPPG signal based on RGB values, principal component analysis achieves dimensionality reduction by finding the direction with the largest variability in the data set (i.e., principal component) and projecting the original data onto these principal components. First, subtract the mean of each column (i.e., the signal of each color channel) to centralize the data and ensure that the results of principal component analysis will not be affected by the absolute value of the signal. Next, the covariance matrix of the signal matrix is ​​calculated to describe the linear correlation between different channels (R, G, B). The eigenvalues ​​and eigenvectors are solved for the covariance matrix. The eigenvalue represents the variability of the data in the direction of the eigenvector, and the eigenvector represents the direction of the principal component. Finally, the eigenvector with the largest eigenvalue is selected as the principal component direction, and the original three-dimensional signal is projected onto this direction to obtain a one-dimensional pulse RPPG signal.

[0099] Through principal component analysis, the original three-dimensional RGB-based RPPG signal is converted into a one-dimensional pulse RPPG signal. The one-dimensional pulse RPPG signal retains the most important information related to the pulse in the original data, while removing excess noise and redundant components, making the subsequent heart rate estimation more accurate and stable, and extracting the signal component that best reflects the changes in the heartbeat cycle from the high-dimensional data.

[0100] Exemplarily, the pulse RPPG signal is transformed from the time domain to the frequency domain, and a heartbeat estimation value is determined within the effective heartbeat frequency range, including: converting the pulse RPPG signal from the time domain to the frequency domain through fast Fourier transform to obtain a corresponding pulse spectrum signal; determining the peak frequency corresponding to the peak value of the pulse spectrum signal within the effective heartbeat frequency range; and determining the heartbeat estimation value based on the peak frequency.

[0101] As mentioned above, the pulse RPPG signal appears as a time-varying periodic waveform in the time domain, which is usually affected by various factors (such as noise, motion, etc.), making its periodicity less obvious. It is difficult to clearly see the heartbeat frequency directly from the time domain signal, so it is converted to the frequency domain to more clearly analyze its periodic components. Fast Fourier Transform (FFT) is an efficient algorithm used to calculate the discrete Fourier transform (DFT) of a signal and convert the signal from the time domain to the frequency domain. In the frequency domain, the periodicity of the signal will be manifested in the form of spectrum peaks, which correspond to the main frequency components of the pulse spectrum signal. Assume that a time domain pulse RPPG signal x(t) has been obtained, and x(t) has accumulated multiple frames of data (for example, 450 frames) in the time window. Apply the FFT algorithm to convert x(t) into the corresponding pulse spectrum signal X(f), where f is the frequency variable, X(f) is the signal amplitude of the frequency, and is displayed as |X(f1)|, |X(f2)|, |X(f3)|...|X(fn)| at different frequencies. Among them, each X(fi) represents the signal amplitude of the corresponding frequency fi. In the pulse spectrum signal, only the frequency components within the effective frequency interval are concerned to exclude noise or other non-heartbeat related signal interference. In the obtained frequency domain signal X(f), the signal amplitude within the effective frequency interval is searched, and the maximum amplitude point within the effective frequency interval is determined. The frequency corresponding to the maximum amplitude point is the peak frequency, which is the strongest periodic component in the signal and is likely to be related to the heartbeat. Assuming that in the effective frequency interval of the heartbeat, the maximum peak occurs at 1.2Hz, and the signal amplitude at this frequency point is the largest, then 1.2Hz is the determined peak frequency.

[0102] Convert the peak frequency in the frequency domain to heart rate, expressed as the number of heartbeats per minute. The specific calculation formula is heartbeat estimate (BPM) = peak frequency (Hz) × 60. Continuing with the above example, if the peak frequency is 1.2Hz, the corresponding heartbeat estimate is 1.2×60=72 beats / minute (BPM), indicating that the user's heart rate is estimated to be 72 beats / minute in the current time window.

[0103] After converting the pulse RPPG signal from the time domain to the frequency domain through fast Fourier transform, the main periodic components of the signal can be clearly seen. By finding the maximum peak frequency in the effective heartbeat frequency range and converting it into a heartbeat estimation value, the user's heart rate can be accurately inferred. The above steps convert the complex time domain signal analysis problem into a peak detection problem of the frequency domain signal, making the heart rate estimation more intuitive and accurate.

[0104] Exemplarily, a signal-to-noise ratio is determined based on the pulse RPPG signal and the heartbeat estimation value, including: in the pulse spectrum signal, determining the peak frequency corresponding to the heartbeat estimation value as the fundamental frequency; based on the fundamental frequency, determining multiple harmonics from the pulse spectrum signal, wherein the frequency difference of each harmonic is an integer multiple of the fundamental frequency; calculating the spectrum values ​​of the fundamental frequency and each harmonic, and superimposing the spectrum values ​​of the fundamental frequency and each harmonic to obtain a synthesized pulse signal; subtracting the synthesized pulse signal from the pulse RPPG signal to extract a noise signal; and using an SNR algorithm to calculate the signal-to-noise ratio based on the synthesized pulse signal and the noise signal.

[0105] The fundamental frequency is the frequency corresponding to the estimated value of the heartbeat in the pulse RPPG signal. Assuming that the heartbeat frequency found after FFT transformation is 1.2Hz, this frequency is the fundamental frequency. The fundamental frequency is the basis for calculating harmonics. The subsequent steps are based on this fundamental frequency to find harmonics and then synthesize the pulse signal. Harmonics are components with integer multiples of the fundamental frequency. For a fundamental frequency of 1.2Hz, the frequencies of the harmonics will be 2.4Hz (2 times the fundamental frequency), 3.6Hz (3 times the fundamental frequency), 4.8Hz (4 times the fundamental frequency), etc. In the actual pulse spectrum signal, in addition to the fundamental frequency, there are often multiple harmonic components, reflecting the periodic characteristics of the heartbeat signal. Assuming that the fundamental frequency is 1.2Hz, you will look for the harmonic spectrum values ​​corresponding to frequency points such as 2.4Hz, 3.6Hz, and 4.8Hz. Calculating the spectrum amplitude corresponding to the fundamental frequency and its harmonic frequencies in the pulse spectrum signal reflects the energy of the signal at these frequency points. The spectrum values ​​of the fundamental frequency and each harmonic are superimposed to generate a synthesized pulse signal. The synthesized signal should mainly reflect the periodic signal component of the heartbeat. If the spectrum value of the fundamental frequency 1.2Hz is X(f1), and the spectrum values ​​of its harmonics 2.4Hz and 3.6Hz are X(f2) and X(f3), then the synthesized pulse signal can be expressed as X(f1)+X(f2)+X(f3). The noise signal is the component of the pulse RPPG signal that does not belong to the heartbeat signal, which may be caused by motion artifacts, changes in illumination, etc. Subtract the synthesized pulse signal from the original pulse RPPG signal, and the remaining part is the noise signal. If the original pulse RPPG signal is RPPG_original, the noise signal is RPPG_original-synthesized pulse signal.

[0106] The signal-to-noise ratio SNR is calculated by the following formula:

[0107]

[0108] In the formula, ∑(synthesized pulse signal 2 ) is the total energy of the synthesized pulse signal, ∑(noise signal 2) is the sum of the energy of the noise signal. If the energy of the synthetic pulse signal is 100 and the energy of the noise signal is 10, then the signal-to-noise ratio is:

[0109]

[0110] The value of the signal-to-noise ratio directly affects the confidence of heartbeat estimation. The present invention generates a synthetic pulse signal through fundamental frequency and harmonic analysis, extracts the noise signal, and calculates the signal-to-noise ratio using the SNR algorithm, thereby effectively distinguishing between real heartbeat signals and noise interference, thereby improving the accuracy of heartbeat estimation based on face video.

[0111] Exemplarily, when the signal-to-noise ratio is greater than a preset threshold, the heartbeat estimation value corresponding to the pulse RPPG signal is output, including: when the signal-to-noise ratio is greater than the preset threshold, marking the heartbeat estimation value corresponding to the pulse RPPG signal as a valid heartbeat, and outputting the valid heartbeat; when the signal-to-noise ratio is less than the preset threshold, marking the heartbeat estimation value corresponding to the pulse RPPG signal as an invalid heartbeat, discarding the invalid heartbeat, and outputting the last determined valid heartbeat.

[0112] A reference value that determines whether the heartbeat estimate is credible based on a preset threshold. It determines which heartbeat estimates can be accepted and output and which need to be discarded to avoid misjudgment under noise interference.

[0113] When the signal-to-noise ratio is greater than the preset threshold, it means that the heartbeat component in the signal is dominant and the noise has little effect. At this time, the current heartbeat estimate is marked as "valid heartbeat". Assuming that after calculation, the current signal-to-noise ratio is 4.0dB, which is higher than the set 3.1dB threshold, the current heartbeat estimate is considered reliable. After marking as a valid heartbeat, the current heartbeat estimate is output, and the output form of the heartbeat estimate can be heart rate (such as 70BPM, indicating 70 heartbeats per minute).

[0114] When the signal-to-noise ratio is less than the preset threshold, it means that the noise component is large and the heartbeat signal is unreliable. At this time, the current heartbeat estimate is marked as "invalid heartbeat". For invalid heartbeats, the current estimate is not directly output, but discarded to avoid erroneous output due to noise interference. Go back to the last valid heartbeat when the signal-to-noise ratio was greater than the threshold, and output this value as the current heart rate estimate to ensure that a stable heart rate estimate can be provided even in the case of noise interference.

[0115] Exemplarily, performing face detection on the face video frame by frame to determine face areas of multiple frames includes: using a lightweight Yunet algorithm to perform face detection on each frame of the face video to determine the position and boundaries of the face area of ​​each frame; determining the face areas of multiple frames based on the position and boundaries of the face area of ​​each frame.

[0116] Yunet is a lightweight face detection algorithm that aims to detect faces in video frames efficiently and quickly. The Yunet algorithm can achieve real-time face detection in a low computing resource environment. It has the characteristics of lightweight model and fast detection speed. It can reduce the demand for computing resources while maintaining high accuracy, making it suitable for various devices, including mobile devices and embedded systems. Therefore, the embodiment of the present disclosure preferably uses the lightweight Yunet algorithm for face detection.

[0117] The Yunet algorithm analyzes each frame of the image through feature extraction and classification steps, identifies the location of the face, and marks the face area in the form of a bounding box. The bounding box is represented by a rectangle and defines the range of the face in the image. The face video consists of multiple consecutive frames of images. The Yunet algorithm will independently detect each frame of the image to obtain the position information of the face in each frame. As the video plays, the algorithm will track and record the changes in the position of the face in each frame. By analyzing the position and boundaries of the face detected in each frame, the consistency of the face area in multiple frames is determined. Optionally, during the detection process, the face area in consecutive frames is compared to determine whether the position of the face in the video remains stable or moves.

[0118] Exemplarily, skin segmentation is performed on the face area of ​​each frame to obtain multiple valid skin pixels in each frame, including: using a skin segmentation algorithm to classify pixels in the face area of ​​each frame to distinguish between skin areas and non-skin areas; based on color and texture features, valid skin pixels in each frame of the image are extracted in the skin area.

[0119] The detected face area is segmented, and valid skin pixels are extracted from each frame, which are used to calculate the RGB RPPG signal in the subsequent steps. Optionally, a skin segmentation algorithm based on a color model is used, such as a method based on YCbCr, HSV or Lab color space, because these color spaces can better distinguish skin from non-skin areas. For example, in the YCbCr space, the Cr and Cb components can effectively distinguish skin tones.

[0120] The skin segmentation algorithm analyzes the color features of pixels in each frame of the image and divides the pixels into skin areas and non-skin areas. Specifically, the skin segmentation algorithm uses a predefined skin color range to classify pixels. Pixels within the skin color range are marked as skin pixels, and other pixels are marked as non-skin pixels. For example, in a frame of image, suppose the pixels in the face area have a Cr value between 135 and 180 in the YCbCr color space and a Cb value between 85 and 135. The skin segmentation algorithm scans each pixel in the face area, and if the Cr and Cb values ​​of a pixel fall within the above range, the pixel will be marked as a skin pixel.

[0121] Within the skin region, skin pixels are further refined by color features. For example, the intensity of the red component (R) of the RGB color space is further utilized, combined with other color space features, to exclude possible misclassified pixels to ensure that the extracted skin pixels are more representative. For example, within the skin region, it is further required that the value of the red component R is higher than 120, and the value of R is greater than the values ​​of G and B. This additional restriction helps to remove mislabeled non-skin pixels and ensure that the segmented pixels more accurately represent the skin.

[0122] In addition to color features, texture features can also help further refine skin segmentation. Optionally, texture analysis can use methods such as local binary patterns (LBP) or gray-level co-occurrence matrix (GLCM) to help distinguish real skin texture from background texture. In the segmented skin area, LBP is used to analyze the texture and extract pixels with skin texture characteristics. This step can further eliminate light reflections or shadow areas that may be mixed in, making the final skin pixels more accurate. After dual screening of color and texture features, the remaining skin pixels in each frame of the image are valid skin pixels. The valid pixels will be used for the calculation of RPPG signals in subsequent steps.

[0123] By performing frame-by-frame skin segmentation on the face area, using algorithms based on color and texture features, the skin pixels in each frame can be effectively extracted. After double screening, these face pixels become valid skin pixels, providing high-quality input data for subsequent RGB RPPG signal extraction and heartbeat estimation. Accurate skin segmentation can not only improve the accuracy of heartbeat estimation, but also effectively reduce the interference caused by the external environment and face movement.

[0124] Figure 2 FIG. 1 is a structural block diagram of a heartbeat estimation device based on RPPG signals of face video provided by an embodiment of the present disclosure. Figure 2 As shown, the device comprises:

[0125] The acquisition module 201 is used to acquire the RPPG signal based on the RGB value of each frame from the face video to be detected;

[0126] An analysis module 202 is used to perform principal component analysis on the RPPG signal based on the RGB value within a preset time window to obtain a pulse RPPG signal;

[0127] A determination module 203, configured to transform the pulse RPPG signal from the time domain to the frequency domain, and determine a heartbeat estimation value within a heartbeat effective frequency interval;

[0128] A confidence module 204, configured to determine a signal-to-noise ratio based on the pulse RPPG signal and the heartbeat estimation value;

[0129] The output module 205 is used to output the heartbeat estimation value corresponding to the pulse RPPG signal when the signal-to-noise ratio is greater than a preset threshold.

[0130] Exemplarily, the acquisition module includes:

[0131] The face video acquisition submodule is used to acquire the face video to be detected;

[0132] A face region determination submodule, used to perform face detection on the face video frame by frame to determine face regions of multiple frames;

[0133] The effective skin pixel determination submodule is used to perform skin segmentation on the face area of ​​each frame to obtain multiple effective skin pixels of each frame;

[0134] The mean calculation submodule is used to calculate the mean of the RGB values ​​of multiple valid skin pixels in each frame to obtain an RPPG signal based on the RGB value of each frame.

[0135] Exemplarily, the analysis module includes:

[0136] An organization submodule, used for organizing the RPPG signal based on the RGB value of each frame in a preset time window into an RPPG signal matrix;

[0137] The conversion submodule is used to perform principal component analysis on the RPPG signal matrix and convert the three-dimensional RPPG signal based on RGB values ​​into a one-dimensional pulse RPPG signal.

[0138] Exemplarily, the determining module includes:

[0139] A conversion submodule, used to convert the pulse RPPG signal from the time domain to the frequency domain through fast Fourier transform to obtain a corresponding pulse spectrum signal;

[0140] A peak frequency determination submodule, used to determine the peak frequency corresponding to the peak value of the pulse spectrum signal within the effective heartbeat frequency interval;

[0141] The heartbeat estimation value determination submodule is used to determine the heartbeat estimation value according to the peak frequency.

[0142] Exemplarily, the confidence module includes:

[0143] A fundamental frequency determination submodule, used for determining the peak frequency corresponding to the heartbeat estimation value in the pulse spectrum signal as the fundamental frequency;

[0144] A harmonic determination submodule, used to determine a plurality of harmonics from the pulse spectrum signal according to the fundamental frequency, wherein the frequency difference of each harmonic is an integer multiple of the fundamental frequency;

[0145] A superposition submodule, used for calculating the spectrum values ​​of the fundamental frequency and each harmonic, and superimposing the spectrum values ​​of the fundamental frequency and each harmonic to obtain a synthetic pulse signal;

[0146] A noise extraction submodule, used for subtracting the synthesized pulse signal from the pulse RPPG signal to extract a noise signal;

[0147] The signal-to-noise ratio calculation submodule is used to calculate the signal-to-noise ratio according to the synthesized pulse signal and the noise signal by using an SNR algorithm.

[0148] Exemplarily, the output module includes:

[0149] The effective heartbeat submodule is used to mark the heartbeat estimation value corresponding to the pulse RPPG signal as a valid heartbeat when the signal-to-noise ratio is greater than the preset threshold, and output the valid heartbeat;

[0150] The invalid heartbeat submodule is used to mark the heartbeat estimation value corresponding to the pulse RPPG signal as an invalid heartbeat when the signal-to-noise ratio is less than the preset threshold, discard the invalid heartbeat, and output the last determined valid heartbeat.

[0151] Exemplarily, the face region determination submodule includes:

[0152] A face detection unit, used to perform face detection on each frame of the face video using a lightweight Yunet algorithm to determine the position and boundary of the face area of ​​each frame;

[0153] The face region determination unit is used to determine the face regions of multiple frames according to the position and boundary of the face region of each frame.

[0154] Exemplarily, the effective skin pixel determination submodule includes:

[0155] A distinguishing unit, used to classify pixels in the face area of ​​each frame using a skin segmentation algorithm to distinguish skin areas from non-skin areas;

[0156] The extraction unit is used to extract effective skin pixel points in each frame of the image in the skin area based on color and texture features.

[0157] An embodiment of the present disclosure also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various processes in the above-mentioned embodiment of a heartbeat estimation method based on RPPG signals of face videos are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0158] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods and devices. Therefore, the embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] The embodiments of the present disclosure are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing terminal device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0160] Although the preferred embodiments of the present disclosure have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present disclosure.

[0161] Finally, it should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "include..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements. The above is a detailed introduction to a heartbeat estimation method and device based on a face video RPPG signal provided by the present disclosure. This article uses specific examples to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present disclosure; at the same time, for those of ordinary skill in the art, according to the idea of ​​the present disclosure, there will be changes in the specific implementation method and scope of application. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

Claims

1. A heartbeat estimation method based on RPPG signal of face video, characterized in that: The method comprises: From the face video to be detected, obtain the RPPG signal based on the RGB value of each frame; Performing principal component analysis on the RPPG signal based on RGB values ​​within a preset time window to obtain a pulse RPPG signal; Transforming the pulse RPPG signal from the time domain to the frequency domain, and determining a heartbeat estimation value within a heartbeat effective frequency interval; Determining a signal-to-noise ratio based on the pulse RPPG signal and the heartbeat estimate; When the signal-to-noise ratio is greater than a preset threshold, the heartbeat estimation value corresponding to the pulse RPPG signal is output.

2. The method according to claim 1, characterized in that The step of obtaining the RPPG signal based on the RGB value of each frame from the face video to be detected includes: Get the face video to be detected; Performing face detection on the face video frame by frame to determine face regions of multiple frames; Perform skin segmentation on the face area of ​​each frame to obtain multiple valid skin pixels in each frame; The RGB values ​​of multiple valid skin pixels in each frame are averaged to obtain an RPPG signal based on the RGB value of each frame.

3. The method according to claim 1, characterized in that The method of performing principal component analysis on the RPPG signal based on the RGB value within the preset time window to obtain the pulse RPPG signal includes: Organizing the RPPG signal based on the RGB value of each frame in a preset time window into an RPPG signal matrix; The principal component analysis is performed on the RPPG signal matrix to convert the three-dimensional RPPG signal based on RGB values ​​into a one-dimensional pulse RPPG signal.

4. The method according to claim 1, characterized in that: The pulse RPPG signal is transformed from the time domain to the frequency domain, and a heartbeat estimation value is determined within a heartbeat effective frequency interval, including: The pulse RPPG signal is converted from the time domain to the frequency domain by fast Fourier transform to obtain a corresponding pulse spectrum signal; Determine the peak frequency corresponding to the peak value of the pulse spectrum signal within the effective heartbeat frequency interval; A heartbeat estimate is determined based on the peak frequency.

5. The method according to claim 4, characterized in that Determining a signal-to-noise ratio according to the pulse RPPG signal and the heartbeat estimation value includes: In the pulse spectrum signal, determining the peak frequency corresponding to the heartbeat estimation value as the fundamental frequency; According to the fundamental frequency, a plurality of harmonics are determined from the pulse spectrum signal, wherein the frequency difference of each harmonic is an integer multiple of the fundamental frequency; Calculating the spectrum values ​​of the fundamental frequency and each harmonic, and superimposing the spectrum values ​​of the fundamental frequency and each harmonic to obtain a synthetic pulse signal; subtracting the synthesized pulse signal from the pulse RPPG signal to extract a noise signal; The signal-to-noise ratio is calculated based on the synthesized pulse signal and the noise signal using an SNR algorithm.

6. The method according to claim 1, characterized in that When the signal-to-noise ratio is greater than a preset threshold, outputting a heartbeat estimation value corresponding to the pulse RPPG signal includes: When the signal-to-noise ratio is greater than the preset threshold, marking the heartbeat estimation value corresponding to the pulse RPPG signal as a valid heartbeat, and outputting the valid heartbeat; When the signal-to-noise ratio is less than the preset threshold, the heartbeat estimation value corresponding to the pulse RPPG signal is marked as an invalid heartbeat, the invalid heartbeat is discarded, and the last determined valid heartbeat is output.

7. The method according to claim 2, characterized in that The performing face detection on the face video frame by frame to determine face regions of multiple frames includes: Using a lightweight Yunet algorithm to perform face detection on each frame of the face video to determine the position and boundary of the face area in each frame; According to the position and boundary of the face region of each frame, the face regions of multiple frames are determined.

8. The method according to claim 2, characterized in that: Perform skin segmentation on the face area of ​​each frame to obtain multiple valid skin pixels in each frame, including: Use skin segmentation algorithm to classify pixels in the face area of ​​each frame to distinguish skin area from non-skin area; Based on the color and texture features, effective skin pixels in each frame of the image are extracted in the skin area.

9. A heartbeat estimation device based on RPPG signal of face video, characterized in that: The device comprises: An acquisition module is used to obtain the RPPG signal based on the RGB value of each frame from the face video to be detected; An analysis module is used to perform principal component analysis on the RPPG signal based on the RGB value within a preset time window to obtain a pulse RPPG signal; A determination module, used to transform the pulse RPPG signal from the time domain to the frequency domain, and determine the heartbeat estimation value within the heartbeat effective frequency interval; A confidence module, used to determine a signal-to-noise ratio based on the pulse RPPG signal and the heartbeat estimation value; The output module is used to output the heartbeat estimation value corresponding to the pulse RPPG signal when the signal-to-noise ratio is greater than a preset threshold.

10. An electronic device, characterized in that: It comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement a heartbeat estimation method based on a face video RPPG signal as described in any one of claims 1-8.

Images