Remote heart rate measurement method with anti-motion interference based on region candidate frame generation and multi-scale ROI fusion

Abstract

A motion-interference-resistant remote heart rate measurement method based on regional candidate box generation and multi-scale ROI fusion, firstly, generates 25 bounding boxes of different sizes around the initial facial bounding box, extracts the pulse signal corresponding to each bounding box for the first signal optimal screening; secondly, uses the eight-neighborhood signal screening method to eliminate the motion jitter caused by the tracking algorithm; finally, constructs a multi-scale ROI video pyramid to improve the diversity of the signal and perform signal feature fusion. The present invention has a great improvement on the accuracy of heart rate measurement in motion scenes.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, signal processing and video heart rate detection, and in particular to a motion-interference-resistant remote heart rate measurement method based on regional candidate frame generation and multi-scale ROI fusion. Background Art

[0002] In recent years, remote detection of human heart rate in a non-contact manner has become an active research direction. Most existing methods focus on the signal separation of the original trajectory obtained from the facial region of interest (ROI), which itself has a crucial impact on the measurement accuracy. Existing remote heart rate detection methods cannot effectively handle ROI tracking issues such as inappropriateness for the subject's face, pixel offset when the head moves, and fluctuations when stationary, which seriously affect the pulse measurement accuracy. Summary of the invention

[0003] In order to overcome the shortcomings of the prior art, the present invention proposes a motion-interference-resistant remote heart rate measurement method based on region candidate frame generation and multi-scale ROI fusion. The method mainly screens out the ROI with the best signal through two region candidate frame generation and signal screening, and scales the ROI to generate multi-scale pulse signals for fusion, so as to improve the heart rate detection accuracy.

[0004] The technical solution used in the present invention is:

[0005] A motion-interference-resistant remote heart rate measurement method based on region candidate frame generation and multi-scale ROI fusion includes the following steps:

[0006] Step 1: Generate region candidate boxes. The process is as follows:

[0007] 1.1 Multi-aspect ratio sampling strategy

[0008] A multi-aspect ratio sampling strategy is used to generate a set of facial bounding boxes with different aspect ratios;

[0009] 1.2 Candidate box fine-tuning

[0010] Generate eight bounding boxes of the same size in the eight neighborhoods around the bounding box after the first screening, and extract the pulse signal respectively;

[0011] Step 2: Extract pulse signal. The process is as follows:

[0012] The pulse signals of different facial bounding boxes are extracted using the remote pulse signal extraction method. The plane orthogonal tiling method POS is used, which defines a vector [1,1,1] TOrthogonal projection plane to eliminate skin color dependence; in order to separate the pulse signal and motion artifacts, the original trajectory is projected onto two vectors on this plane, and the signal is processed by α-tuning to obtain the pulse signal;

[0013] Step 3: Pulse signal screening. The process is as follows:

[0014] 3.1 Maximum signal-to-noise ratio screening

[0015] According to different pulse signal qualities, the pulse signal with the best signal is selected from the pulse signals extracted from all bounding boxes, and the signal-to-noise ratio (SNR) calculation formula is used as the signal screening basis to select the pulse signal with the highest SNR;

[0016] 3.2 Maximum Weight Screening

[0017] Based on the screening method of mean square error (MSE), the facial candidate box corresponding to the signal with the largest weight is screened out according to the size of the weight value;

[0018] Step 4: Multi-scale signal fusion

[0019] The video sequence corresponding to the filtered bounding box is established as a multi-scale video sequence, and the pulse signal of each scale video sequence is extracted respectively, and then convex combination is performed to fuse the final pulse signal, and finally the final heart rate value is obtained after signal processing.

[0020] Furthermore, in step 1, the process of the multi-aspect ratio sampling strategy is:

[0021] Assume that the original facial bounding box is B0 = [x0, y0, w0, h0], where x0, y0, w0, h0 represent the horizontal coordinate, vertical coordinate, width and height of the upper left corner of the bounding box, respectively. The center point of the bounding box is calculated as:

[0022]

[0023] Generate a series of bounding box sets with different aspect ratios using the same bounding box center point and different aspect ratios Where N represents the number of generated bounding boxes, let B i =[x i ,y i ,w i ,h i ] represents the horizontal coordinate, vertical coordinate, width and height of the upper left corner point, then the width and height of the bounding box are calculated as follows:

[0024] w i =σ w ·w0

[0025] hi =σ h ·w i (2)

[0026] where σ w ,σ h The width coefficient and height coefficient of the aspect ratio of the generated bounding box are respectively used to adapt to the facial contours of different people. Considering that the height of most people’s facial contours is greater than the width, let σ w =[0.70,0.75,0.80,0.85,0.90],σ h =[1.1,1.2,1.3,1.4,1.5], Next, the generated bounding box B i The horizontal and vertical coordinates of the upper left corner are calculated as follows:

[0027]

[0028]

[0029] Furthermore, in step 1, the process of fine-tuning the bounding box is as follows:

[0030] The facial bounding box fine-tuning method generates eight bounding boxes of the same size in the eight neighborhoods around the bounding box after the first screening, and extracts the pulse signal respectively. Let C p0 =[x p0 ,y p0 ] is the coordinate of the center point of the facial bounding box after the first screening, and the 8 neighborhoods around it are selected with C pi The feature point separated by 7 pixels is the center point of the target bounding box. Let C pi =[x pi ,y pi ] is the coordinate of the i-th 8-neighborhood feature point, and its coordinate value is calculated as follows:

[0031] C pi =C p0 +p(i)·s (4)

[0032] Wherein p(i) is represented by {(-1,-1), (-1,0), ..., (0,1), (1,1)}, representing the relative position of pixels in 8 neighborhoods, and s=7 in the present invention.

[0033] Furthermore, in step 2, each generated candidate box uses the same pulse signal extraction method: POS, and the process is as follows:

[0034] First, all pixels in each channel of each bounding box are averaged to obtain the original measurement trajectory Where L represents the window length, and time normalization is applied to eliminate the skin color dependency.

[0035] I n =N·I (5)

[0036] in is a diagonal matrix whose ith diagonal gives the inverse of the mean of the ith row of I, that is:

[0037] N i =1 / μ(I i ) (6)

[0038] The time-normalized trajectory is then projected onto a plane whose normal vector has the same direction as the skin color vector [1,1,1] T The purpose is to define two projection axes on the plane to limit the most likely pulsation area. Based on the analysis of the pulsation intensity distribution and the requirement of mutual orthogonality, the projection axis is defined as P = [0, 1, -1; -2, 1, 1], and the projection signal is calculated as follows:

[0039] S1=I ng -I nb

[0040] S2=I ng +I nb -2I nr (7)

[0041] The signals in S1 and S2 are in phase, and the motion artifacts are in anti-phase. Therefore, the final pulse signal is obtained by α tuning.

[0042] p=S1+αS2 (8)

[0043] Where α = σ(s1(t)) / σ(s2(t)), σ(·) represents the standard deviation, and p is the final pulse signal.

[0044] In step 3, the process of maximum signal-to-noise ratio screening is:

[0045] For the predicted pulse signal First, the signal is converted into time-frequency using discrete Fourier transform (DFT), and then the ratio of the signal amplitude within the effective frequency range to the signal amplitude outside the effective frequency range is calculated, which is the signal-to-noise ratio (SNR):

[0046]

[0047] Where U(f) is a selective switching logic gate function that specifies the range of valid frequency signals:

[0048]

[0049] According to the SNR calculation formula, the facial bounding box corresponding to the signal with the highest SNR value is selected for bounding box fine-tuning.

[0050] In step 3, the maximum weight screening process is as follows:

[0051] The second screening requires selecting a bounding box with a better signal from the 9 fine-tuned bounding boxes. A screening method based on mean square error (MSE) is proposed. The calculation formula is as follows:

[0052]

[0053] Where k = 1, 2, 3…P, P is equal to 9, α k It is a special mean square error (MSE) function, and the calculation formula is as follows:

[0054]

[0055] Among them, y ref represents the reference signal, represents the average fusion of the pulse signals predicted by all bounding boxes, and the calculation formula is as follows:

[0056]

[0057] After two screenings in step 3, the bounding box corresponding to the best pulse signal is screened out, and a multi-scale ROI video pyramid is established on this basis to preliminarily separate motion artifacts into face videos of different scales. The final pulse signal is fused by the pulse signals extracted from the multi-scale ROI. Let w0 and h0 be the width and height of the selected bounding box respectively, then the width and height of the multi-scale bounding box are calculated as follows:

[0058] w l =w0·K l

[0059] h l =h0·K l (14)

[0060] Where K = 0.9 represents the scale factor that controls the ratio to the original image size. The anchor center of all scaled bounding boxes is the same as the anchor center of the unscaled bounding box. After the video pyramid is generated, the pulse signal of each layer of the video is extracted and fused. The fusion method is as follows:

[0061]

[0062] Where P represents the number of all layers of the video pyramid, λ l >0 and ∑λ l =0, p(t) is the final pulse signal;

[0063] The above operations are all performed within the time window, and the pulse signals obtained are all within the time window. The complete pulse signals are connected in series through overlapping and adding, and the final heart rate value is obtained through signal processing.

[0064] The beneficial effects of the present invention are as follows: by generating candidate regions and screening signals twice, the ROI with the best signal is screened out, and the ROI is scaled to generate multi-scale pulse signals for fusion, thereby improving the accuracy of heart rate detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a flow chart of a remote heart rate measurement method with motion interference resistance based on region candidate box generation and multi-scale ROI fusion.

[0066] Specific implementation process

[0067] The present invention will be described in further detail below in conjunction with the accompanying drawings.

[0068] Reference Figure 1 , a motion-interference-resistant remote heart rate measurement method based on region candidate frame generation and multi-scale ROI fusion, the method comprising the following steps:

[0069] Step 1: Generate region candidate boxes. The process is as follows:

[0070] 1.1 Multi-aspect ratio sampling strategy

[0071] Assume that the original facial bounding box is B0 = [x0, y0, w0, h0], where x0, y0, w0, h0 represent the horizontal coordinate, vertical coordinate, width and height of the upper left corner of the bounding box, respectively. The center point of the bounding box can be calculated as:

[0072]

[0073] Generate a series of bounding box sets with different aspect ratios using the same bounding box center point and different aspect ratios Where N represents the number of generated bounding boxes, let B i =[x i ,y i ,w i ,h i ] represents the horizontal coordinate, vertical coordinate, width and height of the upper left corner point, then the width and height of the bounding box are calculated as follows:

[0074] w i =σ w ·w0

[0075] h i =σ h ·wi (2)

[0076] where σ w ,σ h are the width coefficient and height coefficient that control the aspect ratio of the generated bounding box, in order to adapt to the facial contours of different people. Considering that the height of most people’s facial contours is greater than the width, let σ w =[0.70,0.75,0.80,0.85,0.90],σ h =[1.1,1.2,1.3,1.4,1.5]. Next, the generated bounding box B i The horizontal and vertical coordinates of the upper left corner are calculated as follows:

[0077]

[0078]

[0079] 1.2 Bounding Box Fine-tuning

[0080] The facial bounding box fine-tuning method generates eight bounding boxes of the same size in the eight neighborhoods around the bounding box after the first screening, and extracts the pulse signal respectively. p0 =[x p0 ,y p0 ] is the coordinate of the center point of the facial bounding box after the first screening, and the 8 neighborhoods around it are selected with C pi The feature points separated by 7 pixels are the center points of the target bounding box. pi =[x pi ,y pi ] is the coordinate of the i-th 8-neighborhood feature point, and its coordinate value is calculated as follows:

[0081] C pi =C p0 +p(i)·s (4)

[0082] Wherein p(i) is represented by {(-1,-1), (-1,0), ..., (0,1), (1,1)}, representing the relative position of pixels in 8 neighborhoods, and s=7 in the present invention.

[0083] Step 2: Extract pulse signal. The process is as follows:

[0084] The present invention uses the commonly used remote pulse signal extraction method POS to extract the pulse signal of each facial boundary box. This method first averages all pixels in each channel of each boundary box to obtain the original measurement trajectory Where L represents the window length, and time normalization is applied to eliminate the skin color dependency.

[0085] I n=N·I (5)

[0086] in is a diagonal matrix whose ith diagonal gives the inverse of the mean of the ith row of I, that is:

[0087] N i =1 / μ(I i ) (6)

[0088] The time-normalized trajectory is then projected onto a plane whose normal vector has the same direction as the skin color vector [1,1,1] T The purpose is to define two projection axes on the plane to limit the most likely pulsation area. Based on the analysis of the pulsation intensity distribution and the requirement of mutual orthogonality, the projection axis is defined as P = [0, 1, -1; -2, 1, 1]. The projection signal is calculated as follows:

[0089] S1=I ng -I nb

[0090] S2=I ng +I nb -2I nr (7)

[0091] The signals in S1 and S2 are in phase, and the motion artifacts are in anti-phase. Therefore, the final pulse signal is obtained by α tuning.

[0092] p=S1+αS2 (8)

[0093] Where α = σ(s1(t)) / σ(s2(t)), σ(·) represents the standard deviation, and p is the final pulse signal.

[0094] Step 3, pulse signal screening, the process is as follows:

[0095] 3.1 Maximum signal-to-noise ratio screening

[0096] For the predicted pulse signal First, the signal is converted into time-frequency using discrete Fourier transform (DFT), and then the ratio of the signal amplitude within the effective frequency range to the signal amplitude outside the effective frequency range is calculated, which is the signal-to-noise ratio (SNR):

[0097]

[0098] Where U(f) is a selective switching logic gate function that specifies the range of valid frequency signals:

[0099]

[0100] According to the SNR calculation formula, the facial bounding box corresponding to the signal with the highest SNR value is selected for bounding box fine-tuning.

[0101] 3.2 Maximum Weight Screening

[0102] The second screening requires selecting a bounding box with a better signal from the 9 fine-tuned bounding boxes. We propose a screening method based on mean square error (MSE), and the calculation formula is as follows:

[0103]

[0104] Where k = 1, 2, 3…P, P is equal to 9, α k It is a special mean square error (MSE) function, and the calculation formula is as follows:

[0105]

[0106] Among them, y ref represents the reference signal, represents the average fusion of the pulse signals predicted by all bounding boxes, and the calculation formula is as follows:

[0107]

[0108] Step 4: Multi-scale signal fusion

[0109] The final pulse signal is fused by extracting pulse signals from multi-scale ROIs. By building a video pyramid of multi-scale ROIs, motion artifacts are preliminarily separated into face videos of different scales. Let w0 and h0 be the width and height of the selected bounding box, respectively. The width and height of the multi-scale bounding box are calculated as follows:

[0110] w l =w0·K l

[0111] h l =h0·K l (14)

[0112] Where K = 0.9 represents the scale factor that controls the ratio to the original image size, and the anchor center of all scaled bounding boxes is the same as the anchor center of the unscaled bounding box. After the video pyramid is generated, the pulse signal of each layer of the video is extracted and fused. The fusion method is as follows:

[0113]

[0114] Where P represents the number of all layers of the video pyramid, λ l >0 and ∑λ l = 0. p(t) is the final pulse signal;

[0115] The above operations are all performed within the time window, and the pulse signals obtained are all within the time window. The complete pulse signals are connected in series through overlapping and adding, and the final heart rate value is obtained through signal processing.

[0116] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A motion-interference-resistant remote heart rate measurement method based on region candidate frame generation and multi-scale ROI fusion, characterized in that the method comprises the following steps: Step 1: Generate region candidate boxes. The process is as follows: 1.1 Multi-aspect ratio sampling strategy A multi-aspect ratio sampling strategy is used to generate a set of facial bounding boxes with different aspect ratios; 1.2 Candidate box fine-tuning Generate eight bounding boxes of the same size in the eight neighborhoods around the bounding box after the first screening, and extract the pulse signal respectively; Step 2: Extract pulse signal. The process is as follows: The pulse signals of different facial bounding boxes are extracted using the remote pulse signal extraction method. The plane orthogonal tiling method POS is used, which defines a vector [1,1,1] T Orthogonal projection plane to eliminate skin color dependence; in order to separate the pulse signal and motion artifacts, the original trajectory is projected onto two vectors on this plane respectively, and the signal is processed by α tuning to obtain the pulse signal; Step 3, pulse signal screening, the process is as follows: 3.1 Maximum signal-to-noise ratio screening According to different pulse signal qualities, the pulse signal with the best signal is selected from the pulse signals extracted from all bounding boxes, and the signal-to-noise ratio (SNR) calculation formula is used as the signal screening basis to select the pulse signal with the highest SNR; 3.2 Maximum Weight Screening Based on the screening method of mean square error (MSE), the facial candidate box corresponding to the signal with the largest weight is screened out according to the size of the weight value; Step 4: Multi-scale signal fusion The video sequence corresponding to the filtered bounding box is established as a multi-scale video sequence, and the pulse signal of each scale video sequence is extracted respectively, and then convex combination is performed to fuse the final pulse signal, and finally the final heart rate value is obtained through signal processing; In step 1, the process of the multi-aspect ratio sampling strategy is: Assume that the original facial bounding box is B0 = [x0, y0, w0, h0], where x0, y0, w0, h0 represent the horizontal coordinate, vertical coordinate, width and height of the upper left corner of the bounding box, respectively. The center point of the bounding box is calculated as: Generate a series of bounding box sets with different aspect ratios using the same bounding box center point and different aspect ratios Where N represents the number of generated bounding boxes, let B i =[x i ,y i ,w i ,h i ] represents the horizontal coordinate, vertical coordinate, width and height of the upper left corner point, then the width and height of the bounding box are calculated as follows: In i =σ w ·w0 h i =s h ·w i (2) where σ w ,σ h The width coefficient and height coefficient of the aspect ratio of the generated bounding box are respectively used to adapt to the facial contours of different people. Considering that the height of most people’s facial contours is greater than the width, let σ w =[0.70,0.75,0.80,0.85,0.90],σ h =[1.1,1.2,1.3,1.4,1.5], Next, the generated bounding box B i The horizontal and vertical coordinates of the upper left corner are calculated as follows:

2. The method for remote heart rate measurement with motion interference resistance based on region candidate frame generation and multi-scale ROI fusion as claimed in claim 1, characterized in that: In step 1, the process of fine-tuning the bounding box is as follows: The facial bounding box fine-tuning method generates eight bounding boxes of the same size in the eight neighborhoods around the bounding box after the first screening, and extracts the pulse signal respectively. Let C p0 =[x p0 ,y p0 ] is the coordinate of the center point of the facial bounding box after the first screening, and the 8 neighborhoods around it are selected with C pi The feature point separated by 7 pixels is the center point of the target bounding box. Let C pi =[x pi ,y pi ] is the coordinate of the i-th 8-neighborhood feature point, and its coordinate value is calculated as follows: C pi =C p0 +p(i)·s (4) Where p(i) is expressed as {(-1,-1),(-1,0),...,(0,1),(1,1)}, representing the relative position of the pixels in the 8 neighborhoods, where s=7.

3. The method for remote heart rate measurement with motion interference resistance based on region candidate frame generation and multi-scale ROI fusion as claimed in claim 1 or 2, characterized in that: In step 2, each generated candidate box uses the same pulse signal extraction method: POS, and the process is as follows: First, all pixels in each channel of each bounding box are averaged to obtain the original measurement trajectory Where L represents the window length, and time normalization is applied to eliminate the skin color dependency. I n =N·I (5) in is a diagonal matrix whose i-th diagonal gives the inverse of the mean of the i-th row of I, that is: N i =1 / μ(I i ) (6) The time-normalized trajectory is then projected onto a plane whose normal vector has the same direction as the skin color vector [1,1,1] T The purpose is to define two projection axes on the plane to limit the most likely pulsation area. Based on the analysis of the pulsation intensity distribution and the requirement of mutual orthogonality, the projection axis is defined as P = [0, 1, -1; -2, 1, 1], and the projection signal is calculated as follows: S1=I ng -IN nb S2=I ng +I nb -2I nr (7) The signals in S1 and S2 are in phase, and the motion artifacts are in anti-phase. Therefore, the final pulse signal is obtained by α tuning. p=S1+αS2 (8) where α=σ(s1(t)) / σ(s2(t)), σ(·) represents the standard deviation, and p is the final pulse signal.

4. The method for remote heart rate measurement with motion interference resistance based on region candidate frame generation and multi-scale ROI fusion as claimed in claim 1 or 2, characterized in that: In step 3, the process of maximum signal-to-noise ratio screening is: For the predicted pulse signal First, the signal is converted into time-frequency using discrete Fourier transform (DFT), and then the ratio of the signal amplitude within the effective frequency range to the signal amplitude outside the effective frequency range is calculated, which is the signal-to-noise ratio (SNR): Where U(f) is a selective switching logic gate function that specifies the range of valid frequency signals: According to the SNR calculation formula, the facial bounding box corresponding to the signal with the highest SNR value is selected for bounding box fine-tuning.

5. The method for remote heart rate measurement with motion interference resistance based on region candidate frame generation and multi-scale ROI fusion as claimed in claim 4, characterized in that: In step 3, the maximum weight screening process is as follows: The second screening requires selecting a bounding box with a better signal from the 9 fine-tuned bounding boxes. A screening method based on mean square error (MSE) is proposed. The calculation formula is as follows: Where k = 1, 2, 3…P, P is equal to 9, α k It is a special mean square error (MSE) function, and the calculation formula is as follows: Among them, y ref represents the reference signal, represents the average fusion of the pulse signals predicted by all bounding boxes, and the calculation formula is as follows:

6. The method for remote heart rate measurement with motion interference resistance based on region candidate frame generation and multi-scale ROI fusion as claimed in claim 5, characterized in that: After two screenings in step 3, the bounding box corresponding to the best pulse signal is screened out, and a multi-scale ROI video pyramid is established on this basis to preliminarily separate motion artifacts into face videos of different scales. The final pulse signal is fused by the pulse signals extracted from the multi-scale ROI. Let w0 and h0 be the width and height of the selected bounding box respectively, then the width and height of the multi-scale bounding box are calculated as follows: In l =w0 K l h l =h0·K l (14) Where K = 0.9 represents the scale factor that controls the ratio to the original image size. The anchor center of all scaled bounding boxes is the same as the anchor center of the unscaled bounding box. After the video pyramid is generated, the pulse signal of each layer of the video is extracted and fused. The fusion method is as follows: Where P represents the number of all layers of the video pyramid, λ l >0 and ∑λ l =0, p(t) is the final pulse signal; The above operations are all performed within the time window, and the pulse signals obtained are all within the time window. The complete pulse signals are connected in series through overlapping and adding, and the final heart rate value is obtained through signal processing.

Images