A fast heart rate detection method under complex light sources based on face recognition
Through a fast heart rate detection method based on face recognition, a universal camera is used to achieve non-invasive measurement of heart rate in complex light source environments, solving the accuracy of the existing technology center rate monitoring under complex light sources, and improving the accuracy and flexibility of detection Supports multi-person monitoring and accuracy correction.
Patent Information
- Application Number
- CN202011561785.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-25
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-12-25
AI Technical Summary
The existing human heart rate monitoring based on RGB cameras is difficult to ensure the accuracy of detection data in complex light source environments, which limits its application range.
Using a fast heart rate detection method based on face recognition, through three steps: signal source selection, signal preprocessing and data judgment, a universal camera is used to realize non-invasive rapid and continuous measurement of heart rate in complex light source environments, including real-time acquisition of video stream data and acquisition Skin exposure sample data, noise suppression and filtering, and short-time Fourier transform to calculate heart rate.
It realizes accurate detection of heart rate under complex light sources, improves the time and measurement accuracy of data locking, supports multi-person monitoring, and can adaptively constrain the influence of light sources, enhancing the reliability and flexibility of detection.
Smart Images

Figure CN114680853B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of biomedical engineering, and in particular to a rapid heart rate detection method based on a general camera. Background Art
[0002] At present, human heart rate monitoring based on RGB cameras needs to be achieved by continuously collecting 15-30 seconds of data and performing data processing and analysis in a stable and sufficient light environment. If there is a complex light source with flicker, there may be problems with the accuracy of the detection data, thus limiting the scope of application of this type of detection. It is necessary to improve it. Summary of the invention
[0003] The present invention aims to provide a fast heart rate detection method under complex light sources based on face recognition, so as to realize non-invasive, rapid, continuous and non-contact measurement of heart rate, and realize detection by working in a complex light source environment through a universal camera, which has the characteristics of rich usage scenarios and low implementation cost.
[0004] The technical solution provided by the present invention is:
[0005] A rapid heart rate detection method under complex light sources based on face recognition, which sequentially includes three steps: signal source selection, signal preprocessing and data judgment;
[0006] 1) The signal source selection includes:
[0007] (1) Collect video stream data from general cameras in real time and perform face recognition on each frame;
[0008] (2) Obtain three main image regions of interest (ROIs) of the forehead and cheeks through HaarCascade;
[0009] (3) Scan pixel by pixel from left to right and from top to bottom in each main ROI using a 15x15 pixel window; obtain skin exposure sample data through the same window in two consecutive frames;
[0010] (4) Keep the window with positive SE and determine it as the sub-ROI;
[0011] (5) Obtain the facial grayscale value outside the main ROI and record the grayscale difference (NF) between two consecutive frames for noise model analysis.
[0012] 2) Signal preprocessing:
[0013] (1) Calculate the mean of the green components of multiple sub-ROIs and add them to the signal preprocessing array (SF) in FIFO mode according to the frame sequence;
[0014] (2) In SF, preliminary noise suppression is achieved by filtering out values that deviate greatly from the main value;
[0015] (3) Standardize and detrend the noise-pretreated SF;
[0016] (4) Use a Butterworth bandpass filter of order 5 or above to filter the SF, remove low-frequency and high-frequency noise interference, and increase the gain of the reasonable heart rate range of the human body;
[0017] (5) The NF (non-ROI facial grayscale signal) is transformed into the frequency domain to obtain the real-time noise signal model of the ambient light;
[0018] 3) Data judgment:
[0019] (1) Based on the preprocessed SF time domain signal, a short-time Fourier transform (STFT) is performed using a Hamming window of 1.5 seconds to 2 seconds;
[0020] (2) The maximum power spectrum (PF) of SF can be continuously obtained through short-time Fourier transform, and then the heart rate (BPM) can be obtained by calculation, BPM = PF*60;
[0021] (3) In the heart rate value array based on the time series, the maximum likelihood heart rate value (ML_BPM) is obtained by calculating the array centroid;
[0022] (4) extracting the historical heart rate data of the relevant human body from the database according to the FaceID obtained by face recognition;
[0023] (5) Compare ML_BPM with historical data, select the data in the ML_BPM array that has the highest overlap with the historical data range, complete the heart rate value judgment, and update the database.
[0024] The skin exposure sample data (SE) is determined by the red channel and the green channel and the brightness in the window, that is, SE = (-1)*Δr*Δg / gray,
[0025] Where: Δr is the mean difference of the red component in the same window of two frames;
[0026] Δg is the mean difference of the green component in the same window of two frames;
[0027] gray is the grayscale mean in the same window of two frames.
[0028] The beneficial effects of the present invention are:
[0029] 1. Based on fast face tracking, obtain the main ROI. Based on the data of skin exposure samples, extract one or more sub-ROIs with high signal-to-noise ratio in the main ROI to perform RPPG calculation; effectively avoid the influence of glasses, beards, masks, shadows and light makeup, and effectively improve the data locking time;
[0030] 2. Binding heart rate data with FaceID can define a new facial heart rate feature matrix; it can support serial or parallel monitoring of multiple people, track heart rate changes of specific faces, and iteratively correct the measurement accuracy based on the data records of specific faces;
[0031] 3. Adaptively constrain the influence of variable light sources, collect and calculate the changes in light sources in non-ROI areas and static areas; based on the RPPG data of the ROI area, use the stroboscopic gain rate for numerical correction in the frequency domain. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 is a flow chart of an embodiment of the present invention.
[0033] Figure 2 It is a schematic diagram of selecting the interest regions of the three main facial images in the present invention.
[0034] Figure 3 It is a schematic diagram of signal source selection in the present invention.
[0035] Figure 4 It is a schematic diagram of the short-time Fourier transform experimental results in the present invention.
[0036] Figure 5 It is a schematic diagram of the experimental results of the pulse wave in the present invention. DETAILED DESCRIPTION
[0037] The present invention will be further described below in conjunction with the embodiments shown in the accompanying drawings of the specification, but the present invention is not limited to the following embodiments.
[0038] The key technical points of the present invention are:
[0039] 1. Signal source acquisition of facial sub-ROI based on skin exposure sample data;
[0040] 2. Pulse wave denoising based on facial non-ROI data;
[0041] 3. Dynamic update of heart rate database based on face recognition.
[0042] The accompanying drawing shows a method for rapid heart rate detection under complex light sources based on face recognition, which sequentially includes three steps: signal source selection, signal preprocessing and data judgment;
[0043] 1) The signal source selection includes:
[0044] (1) Collect video stream data from general cameras in real time and perform face recognition on each frame;
[0045] (2) Obtain three main image regions of interest (ROIs) on the forehead and cheeks through HaarCascade; see Figure 2 );
[0046] (3) Scan each primary ROI pixel by pixel from left to right and from top to bottom using a 15x15 pixel window (see Figure 3 ); obtain skin exposure sample data through the same window of two consecutive frames;
[0047] The skin exposure sample data (SE) is determined by the red channel, green channel and brightness in the window, that is, SE = (-1)*Δr*Δg / gray, where Δr is the mean difference of the red component in the same window of two frames, Δg is the mean difference of the green component in the same window of two frames, and gray is the grayscale mean in the same window of two frames;
[0048] (4) Keep the window with positive SE and determine it as the sub-ROI;
[0049] (5) Obtain the facial grayscale value outside the main ROI and record the grayscale difference (NF) between two consecutive frames for noise model analysis.
[0050] 2) Signal preprocessing:
[0051] (1) Calculate the mean of the green components of multiple sub-ROIs and add them to the signal preprocessing array (SF) in FIFO mode according to the frame sequence;
[0052] (2) In SF, preliminary noise suppression is achieved by filtering out values that deviate greatly from the main value;
[0053] (3) Standardize and detrend the noise-pretreated SF;
[0054] (4) Use a Butterworth bandpass filter of order 5 or above to filter the SF, remove low-frequency and high-frequency noise interference, and increase the gain of the reasonable heart rate range of the human body;
[0055] (5) The NF (non-ROI facial grayscale signal) is transformed into the frequency domain to obtain a real-time noise signal model of the ambient light; according to the stroboscopic gain rate (FJ) of different actual environments, real-time ambient light noise suppression in the frequency domain is achieved.
[0056] 3) Data judgment:
[0057] (1) Based on the pre-processed SF time domain signal, a short-time Fourier transform (STFT) is performed using a Hamming window of 1.5 seconds to 2 seconds; the transformation results are shown in Figure 4 );
[0058] (2) The maximum power spectrum (PF) of SF can be continuously obtained by short-time Fourier transform; see Figure 5 ), and then calculate the heart rate (BPM), BPM = PF * 60;
[0059] (3) In the heart rate value array based on the time series, the maximum likelihood heart rate value (ML_BPM) is obtained by calculating the array centroid;
[0060] (4) extracting the historical heart rate data of the relevant human body from the database according to the FaceID obtained by face recognition;
[0061] (5) Compare ML_BPM with historical data, select the data in the ML_BPM array that has the highest overlap with the historical data range, complete the heart rate value judgment, and update the database.
[0062] The present invention solves the problem of measuring heart rate in the presence of stroboscopic light and sufficient light sources, and can also realize the functions of 1-2 seconds of high heart rate screening and 3-5 seconds of heart rate value measurement.
Claims
1. A rapid heart rate detection method under complex light sources based on face recognition, which includes three steps of signal source selection, signal preprocessing and data judgment in sequence; 1) The signal source selection includes: (1) Collect video stream data from general cameras in real time and perform face recognition on each frame; (2) Obtain three main image regions of interest (ROIs) of the forehead and cheeks through HaarCascade; (3) Scan pixel by pixel from left to right and from top to bottom in each main ROI using a 15x15 pixel window; obtain skin exposure sample data SE through the same window of two consecutive frames; (4) Keep the window with positive SE and determine it as the sub-ROI; (5) Obtain the grayscale value of the face outside the main ROI and record the grayscale difference between two consecutive frames for noise model analysis; 2) Signal preprocessing: (1) Calculate the mean of the green components of multiple sub-ROIs and add them to the signal preprocessing array in FIFO mode according to the frame sequence; (2) In the signal preprocessing array, preliminary noise suppression is achieved by filtering out values that deviate greatly from the main value; (3) performing data processing on the signal preprocessing array after noise preprocessing and standardizing and detrending the data; (4) filtering the signal preprocessing array using a Butterworth bandpass filter of order 5 or higher; (5) Convert the non-ROI facial grayscale signal into frequency domain to obtain the real-time noise signal model of ambient light; 3) Data judgment: (1) Preprocessing the time domain signal of the array according to the preprocessed signal, and performing short-time Fourier transform using a Hamming window of 1.5 seconds to 2 seconds; (2) The spectrum PF of the maximum power of the signal preprocessing array can be continuously obtained through short-time Fourier transform, and then the heart rate BPM is obtained by calculation, BPM = PF*60; (3) In the heart rate value array based on the time series, the maximum likelihood heart rate value ML_BPM is obtained by calculating the centroid of the array; (4) extracting the historical heart rate data of the relevant human body from the database according to the FaceID obtained by face recognition; (5) Compare ML_BPM with historical data, select the data in the ML_BPM array that has the highest overlap with the historical data range, complete the heart rate value judgment, and update the database.
2. The method for rapid heart rate detection under complex light sources based on face recognition according to claim 1, characterized in that: The skin exposure sample data SE in step 1) is determined by the red channel and the green channel and the brightness in the window, that is, SE = (-1)*Δr*Δg / gray, Where: Δr is the mean difference of the red component in the same window of two frames; Δg is the mean difference of the green component in the same window of two frames; gray is the grayscale mean in the same window of two frames.
