A system and method for detecting heart rate when part of video information is missing

Abstract

A system and method for detecting heart rate when part of the video information is missing belongs to the field of physiological signal processing and smart education. The method is divided into three steps. First, the video frame is processed, including face detection (when no face is detected, it is replaced by eye detection), feature point marking, determination of facial area of ​​interest, skin pixel segmentation, extraction of R, G, B three-channel signals and preprocessing, and independent component analysis. Secondly, the BVP signal is extracted and processed, and finally the heart rate is calculated by the Fourier transform method. The present invention takes into account the high probability of missing facial video information during the learning process of students in online classes, and solves the problem that the heart rate cannot be detected due to the missing part of interest caused by the rotation of the head, and the head is suddenly too close to the camera, resulting in the inability to detect the heart rate by locating the face. This method is more practical than the previous method that requires static facial measurement, and is more in line with the application of actual teaching scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of physiological signal processing and smart education, and relates to a heart rate detection system and method in the case of head rotation and partial information missing in a video. Background Art

[0002] Physiological signals can provide objective data support for emotion recognition. At present, the main contact-type physiological signal detection devices include: smart bracelets, finger-clip pulse oximeters, etc., which are subject to certain limitations in use and may cause discomfort or resistance to the subjects. Non-contact physiological signal detection technology is not only convenient, but also can complete the detection without affecting the daily life of the subjects. Video-based physiological signal detection has been developed for more than ten years, and has made breakthrough progress in solving problems such as illumination changes, motion artifacts, and video compression, such as ensemble empirical mode decomposition, chroma-based methods, wavelet transforms, and bounded Kalman filtering technologies. However, most application models have many restrictions, such as: the physical constraints on the subjects are very strong, and the test video is required to include the entire front face; in special cases such as the face facing the camera sideways or the face disappearing, heart rate detection cannot be performed. The above models are difficult to measure heart rate under real conditions. The published patents closest to the present invention include a heart rate detection system and detection method based on face video images, a video-based human heart rate and facial blood volume accurate detection method and system, and a heart rate detection method based on an ordinary camera. The above-mentioned published patent requires the detection of a face to calculate the heart rate, and the detection is performed on the front face. One of the differences of this method is that when the distance is too close or the face is blocked and the face cannot be detected, an attempt is made to locate the region of interest through the human eye for heart rate detection. The second difference is that a symmetrical substitution method is adopted. When the head is turned horizontally between 30 and 45 degrees, the regions of interest that can be detected on the left and right cheeks are symmetrically replaced, which solves the problem that some regions of interest are missing and cannot be detected when the face is turned to the side. Therefore, the present invention is an innovation that applies the above-mentioned patent to specific scenarios to solve special problems. Summary of the invention

[0003] In view of the problem that in online teaching scenarios, students are too close to the camera, resulting in the traditional face detection algorithm failing to detect the face and the heart rate calculation being terminated, the present invention designs a method for locating the facial area of ​​interest through the human eye and calculating the heart rate based on this.

[0004] In order to solve the problem in traditional methods that if the camera cannot detect the whole face and part of the area of ​​interest is missing, the heart rate calculation is terminated, the present invention proposes a symmetrical substitution method. When the head is rotated between 30 and 45 degrees and the facial area of ​​interest is partially missing, the detected data from the left and right cheeks are symmetrically copied to the other side so that the calculation can continue.

[0005] The technical solution adopted by the present invention is as follows:

[0006] The non-contact heart rate measurement process adopted by the present invention is divided into three steps. First, the video frame is processed, including face detection (when no face is detected, it is replaced by eye detection), feature point marking, facial region of interest determination, skin pixel segmentation, extraction of R, G, B three-channel signals and preprocessing, and independent component analysis. Secondly, the BVP signal is extracted and processed, and finally the heart rate is calculated using the Fourier transform method.

[0007] The video non-contact heart rate detection system based on head rotation and partial information loss is characterized by comprising: a video acquisition module, a heart rate calculation module, and a result display storage module.

[0008] The video acquisition module is responsible for extracting data from the video frame. If the user clicks the "Camera" button, the computer camera is turned on, and the system captures the video frame and displays it. At this time, the video acquisition module only captures the video frame and displays it on the graphical interface, and calculates the spatial average value X of the three channels of R, G, and B in the three regions of interest in the image frame by frame. R , X G , X B , and store them in three preparation data queues respectively. Whether to enter the heart rate calculation module is determined by whether there are 600 data (30 frames per second, after our test, it is most appropriate to calculate the heart rate from 20 seconds). If there are 600 data, enter this module. If not, repeatedly calculate the spatial averages of the three channels of R, G, and B in the three regions of interest in the image frame by frame, and store them in three preparation data queues respectively.

[0009] The heart rate calculation module includes data preprocessing, heart rate calculation and calculation result correction. When the data length in the preparatory queue reaches 600, the heart rate calculation module extracts the first 600 data from the three preparatory data queues and stores them in the heart rate calculation data queue. After that, data is extracted every second, that is, the data extracted from the second heart rate calculation data queue is the 31st to 631st in the preparatory queue, and the data stored in the third heart rate calculation data queue is the 61st to 661st in the preparatory queue, and so on. There are only 600 data in total in each heart rate calculation data queue. These 600 data are preprocessed (detrended and standardized), and then independent component analysis is performed, and the BVP signal is found through correlation analysis. After filtering, the heart rate can be calculated through Fourier transform, and finally the calculation result is corrected.

[0010] The result display storage module displays the measured heart rate in a graphical interface and draws the filtered BVP signal and the waveform after Fourier transformation, as well as the heart rate change curve. In addition, the result display storage module needs to display frame images and store the heart rate data in CSV and Word files.

[0011] The video non-contact heart rate detection method based on head rotation and partial information loss includes the following steps:

[0012] Step 1: Process the frame image, locate the face frame and mark 68 feature points. If the face cannot be detected, locate the eye coordinates. (Currently, the commonly used feature points for face contour extraction are 5, 20, 58, 68, etc. We only used one of them.)

[0013] Step 2: determine the region of interest by the coordinates of facial feature points or the coordinates of the human eye.

[0014] Step 3: Determine whether the region of interest disappears. If it disappears, use the symmetric replacement method; if it does not disappear, go directly to step 4.

[0015] Step 4: Extract skin pixels in the region of interest.

[0016] Step 5: Separate the R, G, and B channels of the skin image to obtain three one-dimensional time series.

[0017] Step 6: Preprocess the three one-dimensional time series (detrend and standardize).

[0018] Step 7: Use the processed three-channel one-dimensional time series as the observation signal and use independent component analysis (ICA) to separate the source signal from the observation signal.

[0019] Step eight, select the independent source signal most correlated with the green channel as the BVP signal through the Pearson correlation coefficient.

[0020] Step nine, smooth the BVP signal and perform 0.75-4 Hz bandpass filtering.

[0021] Step 10: Use Fourier transform to obtain the frequency spectrum of the processed BVP signal.

[0022] Step 11: Find the frequency corresponding to the highest peak of the spectrum × 60 to obtain the heart rate measurement value.

[0023] The beneficial effects of the present invention are as follows: the non-contact heart rate detection method based on head rotation and partial information loss in the video proposed by the present invention takes into account the high probability of missing facial video information during students' learning in online classes, and solves the problem that the heart rate cannot be detected due to the missing part of interest caused by head rotation, and the heart rate cannot be detected by locating the face due to the head suddenly being too close to the camera. This method is more practical than the previous method that requires static facial measurement, and is more in line with the application of actual teaching scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a block diagram of the heart rate detection system of the present invention;

[0025] Figure 2 This is a block diagram of the video acquisition module of the present invention;

[0026] Figure 3 This is a block diagram of the heart rate calculation module of the present invention;

[0027] Figure 4 A block diagram of a result display storage module according to the present invention;

[0028] Figure 5 This is a flow chart of the heart rate detection method of the present invention;

[0029] Figure 6 This is a flow chart of locating a region of interest by human eyes according to the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0031] The block diagram of the video non-contact heart rate detection system based on head rotation and partial information loss is as follows: Figure 1 As shown, it includes: a video acquisition module, a heart rate calculation module, and a result display storage module.

[0032] The process of the video non-contact heart rate detection method based on head rotation and partial information loss is as follows Figure 5 As shown, the specific steps include:

[0033] Step 1: Extract the facial area related to the calculation from each frame of the video image. If a face is detected, use the Gradient Boosting Decision Tree (GBDT) algorithm to detect 68 facial feature points. If no face is detected, locate the coordinates of the human eye. Assume that the coordinates of point A of the left eye are (x1, y1), the coordinates of point B are (x2, y2), the coordinates of point C of the right eye are (x3, y3), and the coordinates of point D are (x4, y4). Determine the area of ​​interest through the human eye. Figure 6 shown.

[0034] Step 2: Determine the region of interest using the coordinates of the 68 feature points or the coordinates of the human eye. If feature points are used for positioning, the 17th, 31st, 28th, and 39th feature points associated with the left cheek ROI and the 26th, 35th, 28th, and 42nd feature points associated with the right cheek ROI are extracted, and the region of interest is determined based on the horizontal and vertical coordinates of the feature points. If eye coordinates are used for positioning, assuming that the coordinates of the region of interest are E(x5, y5) and F(x6, y6), the coordinates of the left and right eyes can be used to represent its position.

[0035]

[0036] Step 3: Determine whether the region of interest has disappeared. If it has disappeared, use the symmetric replacement method. If it has not disappeared, use the original region of interest and proceed to step 4. Calculate ROI left and ROI right (ROI left Represents the left region of interest, ROI right represents the area S of the right region of interest left and S right (This refers to the left and right cheeks in the video, which is the opposite of the actual situation), and then the area ratio is used to determine whether one side disappears. Assuming that the area ratio of the left cheek to the right cheek is a, when a is greater than 1.5, it means that the right cheek disappears; when a is less than 0.66, it means that the left cheek disappears; when a is between 0.66 and 1.5, it means that no area of ​​interest disappears.

[0037] a=S left / S right

[0038]

[0039] Step 4: Extract skin pixels in the region of interest. This step uses the skin color model established by Pitas et al. in HSV space. The image will be segmented into skin only if the following conditions are met at the same time.

[0040]

[0041] Step 5: RGB channel separation. Separate the RGB channels of the three ROI areas of each frame and calculate the spatial average of the three channels to obtain r i , g i 、b i , and then transform the entire video sequence into three one-dimensional signals as the original signal, recorded as X = [X R ,X G ,X B ] T , where X R , XG , X B They respectively represent the change curves of the characteristic values ​​of the three components R, G, and B of all image frames within a period of time.

[0042] Step 6: Preprocess (detrend, standardize) the three one-dimensional time series. Detrend is performed by subtracting an optimal (least squares) fitting line, plane or surface from the data, focusing the analysis on the fluctuation of the data trend itself, and standardizing the signal using the following formula.

[0043]

[0044] Among them, Y is the signal after standardization, μ and σ are the mean and standard deviation of the original data, respectively, and i represents the three channels of R, G, and B. Through standardization, the three channel signals will become signals with a mean of 0 and a standard deviation of 1.

[0045] Step 7: The processed three-channel signal is used as the observation signal, and then the source signal Y is separated from the mixed observation signal using independent component analysis FastICA to obtain three independent source signals component1, component2, and component3.

[0046] Step eight, performing correlation analysis on the three separated independent source signals and the green channel signal, and taking the corresponding independent source signal with the largest absolute value of the Pearson correlation coefficient as the pulse wave signal BVP.

[0047] Step nine, smoothing the BVP signal to obtain smooth_BVP, and performing 0.75-4 Hz bandpass filtering on the smooth_BVP signal to obtain bandpass_BVP.

[0048] Step 10: Use Fourier transform to obtain the frequency spectrum fft_BVP of the bandpass_BVP signal.

[0049] Step 11: Find the frequency f corresponding to the highest peak of the fft_BVP spectrum HR ×60 to get the heart rate measurement value HR estimation .

Claims

1. A system for detecting heart rate when part of video information is missing, characterized in that: include: Video acquisition module, heart rate calculation module, result display storage module; among them, The video acquisition module is responsible for extracting data from the video frame. If the user clicks the "Camera" button, the computer camera is turned on, and the system captures the video frame and displays it. At this time, the video acquisition module only captures the video frame and displays it on the graphical interface, and calculates the spatial average value X of the three channels of R, G, and B in the three regions of interest in the image frame by frame. R , X G , X B , and store them in three preparation data queues respectively; determine whether to enter the heart rate calculation module according to whether there are 600 data; if there are 600 data, enter the module; if not, repeatedly calculate the spatial average values ​​of the three channels of R, G, and B in the three regions of interest in the image frame by frame, and store them in three preparation data queues respectively; The heart rate calculation module includes data preprocessing, heart rate calculation and calculation result correction; when the data length in the preparatory queue reaches 600, the heart rate calculation module extracts the first 600 data from the three preparatory data queues and stores them in the heart rate calculation data queue; after that, data is extracted every second, that is, the data extracted from the second heart rate calculation data queue are the 31st to 631st in the preparatory queue, and the data stored in the third heart rate calculation data queue are the 61st to 661st in the preparatory queue, and so on; each time the heart rate calculation data queue has a total of only 600 data; after preprocessing these 600 data, independent component analysis is performed, and the BVP signal is found through correlation analysis. After filtering, the heart rate can be calculated through Fourier transform, and finally the calculation result is corrected; The result display storage module displays the measured heart rate in a graphical interface and draws the filtered BVP signal and the waveform after Fourier transformation, as well as the heart rate change curve; in addition, the result display storage module also needs to display frame images and store the heart rate data in CSV and Word files; Heart rate calculation specifically includes the following steps: Step 1: Extract the facial area related to the calculation from each frame of video image; if a face is detected, use the gradient enhancement decision tree algorithm to detect 68 facial feature points; if no face is detected, locate the coordinates of the eyes; assume that the coordinates of point A of the left eye are (x1, y1), the coordinates of point B are (x2, y2), the coordinates of point C of the right eye are (x3, y3), and the coordinates of point D are (x4, y4); Step 2: Determine the region of interest by the coordinates of 68 feature points or the coordinates of the human eye. If feature points are used for positioning, the 17th, 31st, 28th, and 39th feature points related to the left cheek ROI and the 26th, 35th, 28th, and 42nd feature points related to the right cheek ROI are extracted, and the region of interest is determined according to the horizontal and vertical coordinates of the feature points. If eye coordinates are used for positioning, assuming that the coordinates of the region of interest are E(x5, y5) and F(x6, y6), the coordinates of the left and right eyes can be used to represent its position. Step 3: Determine whether the region of interest has disappeared; if it has disappeared, use the symmetric replacement method; if it has not disappeared, use the original region of interest and proceed to step 4; calculate ROI left and ROI right The area S left and S right ROI left Represents the left region of interest, ROI right represents the right region of interest, which refers to the left and right cheeks in the video. The area ratio is then used to determine whether one cheek has disappeared. Assuming that the area ratio of the left cheek to the right cheek is a, when a is greater than 1.5, it means that the right cheek has disappeared; when a is less than 0.66, it means that the left cheek has disappeared; when a is between 0.66 and 1.5, it means that no region of interest has disappeared. a=S left / S right Step 4: Extract skin pixels in the region of interest. This step uses the skin color model established by Pitas et al. in the HSV space. The skin pixels will be segmented into skin only if the following conditions are met. Step 5: RGB channel separation: Separate the three ROI areas of each frame into RGB channels, and calculate the spatial average values ​​of the three channels to obtain r i , g i 、b i , and then transform the entire video sequence into three one-dimensional signals as the original signal, recorded as X = [X R ,X G ,X B ] T , where X R , X G , X B Respectively represent the change curves of the characteristic values ​​of the three components R, G, and B of all image frames over a period of time; Step 6: Preprocess the three one-dimensional time series, i.e., detrend and standardize them. Detrend is performed by subtracting an optimal fitting straight line, plane, or surface from the data, focusing the analysis on the fluctuation of the data trend itself, and standardizing the signal using the following formula; Among them, Y is the signal after standardization, μ and σ are the mean and standard deviation of the original data respectively, and i represents the three channels of R, G, and B. Through standardization, the three channel signals will become signals with a mean of 0 and a standard deviation of 1; Step 7: The processed three-channel signal is used as the observation signal, and then the source signal Y is separated from the mixed observation signal by independent component analysis FastICA to obtain three independent source signals component1, component2, and component3; Step 8: Perform correlation analysis on the three separated independent source signals and the green channel signal, and take the independent source signal corresponding to the largest absolute value of the Pearson correlation coefficient as the pulse wave signal BVP; Step nine, smoothing the BVP signal to obtain smooth_BVP, and performing 0.75-4 Hz bandpass filtering on the smooth_BVP signal to obtain bandpass_BVP; Step 10, using Fourier transform to obtain the frequency spectrum fft_BVP of the bandpass_BVP signal; Step 11: Find the frequency f corresponding to the highest peak of the fft_BVP spectrum HR ×60 to get the heart rate measurement value HR estimation .

Images