A heart rate detection method for compressed video

Abstract

A heart rate detection method for compressed video, comprising the following steps: 1) estimating skin reflection model parameters; 2) synthesizing a noise-free observation signal; 3) extracting a blood volume pulse (BVP) signal. Based on the skin reflection model, the model parameters are estimated using the observation signal, and finally a noise-free observation signal is generated, followed by a POS pulse signal extraction algorithm, and finally a pulse signal output is obtained. The present application can reduce the cost of video storage and transmission under the premise of ensuring the high quality of rPPG signal, and provide technical possibility for remote home health care, cloud video data processing and other remote application scenarios, greatly expand the practical application range of rPPG.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video heart rate detection, computer vision and signal processing, and particularly relates to a heart rate detection method for compressed video. BACKGROUND

[0002] Existing heart rate detection technologies are mostly contact methods, such as electrocardiography (ECG), photoplethysmography (PPG), etc. These methods measure accurate and information-rich physiological electrical signals by connecting the human skin to complex sensing devices. However, both methods require special equipment to contact the skin, which is costly, and long-term wearing can cause discomfort, thus having certain limitations in actual use scenarios. In recent years, remote photoplethysmography (rPPG) based on optical and physiological principles has attracted more and more attention. It refers to capturing the fluctuation of blood volume in the blood vessels under the skin tissue caused by the heart cycle through a camera to capture the face video of a person, analyzing the photoplethysmography signal related to the heart rate from the periodic changes in skin color, and realizing remote measurement of heart rate. This technology is also known as super-sensing heart rate detection, and has achieved rapid development due to its easy implementation, wide application, low cost, non-contact measurement and other advantages.

[0003] However, many previous rPPG measurement works have not considered a key problem: the influence of video compression on rPPG measurement, which is called video compression artifact. The rPPG method mainly relies on analyzing the subtle changes in the horizontal degree that cannot be seen by the naked eye in the face video, and these details are easily affected by video compression, resulting in information loss. In fact, most commercial cameras capture videos that are compressed by different compression codecs at different bit rates, and considering the convenience of storage and transmission on the Internet, video compression is inevitable in scenarios that require remote video transmission. Due to the needs of these practical applications, the original video will be compressed to a certain extent, resulting in a large amount of noise signal contained in the red and blue channels, which greatly disturbs the extraction of the pulse signal and reduces the accuracy of the rPPG algorithm in estimating the heart rate. Therefore, it is of great practical value to develop an rPPG method that works stably on compressed video. SUMMARY

[0004] In order to overcome the deficiencies of the prior art, the present application proposes a heart rate detection method for compressed video. The method is performed at the level of rPPG post-extraction biological signal processing, so that a high SNR pulse signal can be extracted even in the presence of a large amount of noise signal in the red and blue channels. Based on the skin reflection model, the model parameters are estimated using the observation signal, and finally a noise-free observation signal is generated. The POS pulse signal extraction algorithm is connected, and the heart rate signal output is finally obtained.

[0005] The technical scheme adopted by the present application to solve its technical problems is:

[0006] A heart rate detection method for compressed video, the method comprising the following steps:

[0007] Step 1: Estimate the skin reflection model parameters

[0008] The physiological indicator measurement algorithm based on rPPG reflects the change of blood volume by measuring the difference in skin absorption and reflection of different color lights, and the correlation degree of the signals contained in the RGB three channels and the heart rate signal is different, wherein the G channel contains the strongest heart rate signal, and the pulse signal intensity in the R and B channels is much lower than the time signal, so the existence of the pulse signal can be ignored in the process of estimating the time signal; by modeling the absorption and reflection of the skin to different color channels, using the difference and complementarity of the skin to different wavelength lights, combining multiple color channels to estimate the pulse signal, and synthesizing a noise-free observation signal, the heart rate physiological indicator is measured;

[0009] Step 2: Synthesize a noise-free observation signal

[0010] Based on the double reflection model, the reflection of each skin pixel in the recorded image sequence is defined as a time-varying function in the RGB channel, which is caused by body movement and blood pulsation, according to the skin reflection model, the model parameters estimated are substituted, and a noise-free original measurement signal is synthesized;

[0011] Step 3: Extract the blood volume pulse BVP signal

[0012] The blood volume pulse signal extraction adopts the plane orthogonal surface method POS, uses a plane orthogonal to the skin color in the time normalized RGB space to extract the pulse, and uses a projection matrix to project the RGB signal to a range most likely to contain the pulse signal.

[0013] Further, the process of step 1 is as follows:

[0014] Estimate the skin reflection model parameters:

[0015] First, according to the skin reflection model, the original measurement signal collected by a normal camera after time normalization (L is the window length) is as follows:

[0016]

[0017] Based on the dichromatic reflection model, C n(t) consists of specular and diffuse reflections from the skin surface, the specular reflection mainly contains motion-induced changes due to body motion affecting the geometry between the light source, the skin surface and the camera, i.e. unwanted motion artifacts; the specular reflection contains the mirror change signal, i.e. the non-pulse (inter-beat) signal, denoted as is the specular reflection related vector; a small portion of the incident light penetrates the skin to reach the micro blood vessels, this portion becomes the diffuse reflection, which contains the target pulse signal, denoted as is the vibration direction of the pulse; both components are proportional to the water level of the light intensity, thus modulated by the intensity change signal denoted as i(t);

[0018] For a noisy camera, assume that the green channel signal still satisfies equation (1), but in the blue and red channel signals, p(t) is not contained, only s(t) is contained;

[0019] Then the degenerated signal The RGB three-channel original measurement signal formula is as follows:

[0020] C nr (t) = i(t) + u sr s(t) (2)

[0021] C ng (t) = i(t) + u sg s(t) + u pg p(t) (3)

[0022] C nb (t) = i(t) + u sb s(t) (4)

[0023] The target is to estimate p(t) given C n (t), u s and u p , estimate u s and u p from the original measurement signal collected by a noiseless camera, and directly use the learned u s and u p ;

[0024] u s = [0.6265, 0.5479, 0.5526] (5)

[0025] u p = [0.3435, 0.7968, 0.4972] (6)

[0026] where it is required that u s and u p are unit vectors, i.e.:

[0027]

[0028] ①Estimation of s(t) by using C n Estimation of s(t)

[0029] Since the signal strength of p(t) is much lower than s(t) in red and blue channels, p(t) can be ignored in the process of estimating s(t), and i(t) is constant, i.e.

[0030]

[0031] Estimation of s(t) by least square method:

[0032]

[0033] Combining equation (7), the analytical solution of this problem is obtained:

[0034]

[0035] ②Estimation of p(t) by green channel signal;

[0036] According to the green channel signal formula 3, after simple derivation,

[0037]

[0038] Further, the process of step 2 is: synthesizing the noise-free observation signal, substituting the known u s , u p and the estimated s(t) and p(t) into equation (1), to obtain the calculation formula of the synthesized original measurement signal:

[0039]

[0040] Further, the noise-free original measurement signal obtained from step 1 and step 2 is used to extract the blood volume pulse BVP signal by POS algorithm, the process is as follows:

[0041] To extract the blood volume pulse BVP signal, the plane orthogonal surface method POS, which is a high-performance, simple algorithm and widely used, is used to extract the blood volume pulse (BVP) signal from the trajectory of remote photoplethysmography (rPPG); POS defines a projection plane orthogonal to the vector [1,1,1] T The plane is actually a plane orthogonal to the time-normalized skin color to eliminate the dependence of skin color;

[0042] For a single light source, the skin pulsation is usually the largest in the G channel, followed by the B channel and the R channel; based on such channel ranking and the requirement of the projection signal, the time-normalized trace is projected onto two vectors defined by the projection matrix: P p = [0, 1, -1; -2, 1, 1], where each row represents a mutually orthogonal projection axis, which actually combines the time-normalized RGB signals into:

[0043] S1(t) = I nG (t) - I nB (t) (14)

[0044] S2(t) = I nG (t) + I nB (t) - 2I nR (t) (15)

[0045] The specular reflection distortion and the pulse are physically separable on the projection plane, in order to separate the specular reflection and the pulse signal components, S1(t) and S2(t) need to be a-tuned:

[0046] h(t) = S1(t) + aS2(t) (16)

[0047] where a = σ(S1(t)) / σ(S2(t)), and σ(·) represents the standard deviation, h(t) is the extracted BVP signal, also called the POS feature; the alpha adjustment used has an attractive property: when the pulse variation dominates S(t), S1(t) and S2(t) appear in phase, adding the two in-phase signals together will increase the resulting signal strength, i.e. the value of a is not important at this time; when the specular reflection variation dominates S(t), S1(t) and S2(t) appear out of phase, a pulls / pushes the specular variation strength of one signal to the same level as the other, i.e. σ(S1) = σ(a-S2); adding two out-of-phase signals with the same amplitude will cancel the specular reflection distortion, however, when the pulse strength and the specular reflection strength are very close, its performance becomes suboptimal, i.e. a is driven by a mixture of the two, so it is not well-defined;

[0048] For a long-time detection video, the windowed outputs are concatenated to get the long-time pulse signal, given a video of length N, first the sequence is segmented into segments of length T, the proposed algorithm is applied to get the windowed outputs, and the overlap-and-add method is applied to get the final pulse signal output.

[0049] The beneficial effects of the present application mainly manifest in: under the premise of ensuring high quality of rPPG signals, the cost of video storage and transmission is reduced, which provides technical possibility for remote home health care, cloud video data processing and other remote application scenarios, greatly expanding the practical application range of rPPG. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 A flowchart of a heart rate detection method for compressed video.

[0051] Figure 2 An example diagram of a processing flow. DETAILED DESCRIPTION

[0052] The present application will be further described below with reference to the accompanying drawings.

[0053] Referring to Figure 1 and Figure 2 A heart rate detection method for compressed video includes the following steps:

[0054] Step 1, estimate the skin reflection model parameters

[0055] First, according to the skin reflection model, the original measurement signal collected by the normal camera after time normalization (L is the window length) is as follows:

[0056]

[0057] Based on the dichromatic reflection model, C n (t) is composed of specular reflection and diffuse reflection from the skin surface, due to the influence of body movement on the geometry between the light source, the skin surface and the camera, the specular reflection mainly contains the change caused by movement, that is, the unwanted motion artifact, the specular reflection contains (non-pulse) specular change signal (time signal), denoted as is the vector related to the specular reflection; a small part of incident light penetrates the skin to reach the microvessel, which becomes diffuse reflection, which contains the target pulse signal, denoted as is the vibration direction of the pulse, both components are proportional to the light intensity level, so they are modulated by the intensity change signal denoted as i(t);

[0058] For a camera with too much noise, assuming that the green channel signal still satisfies formula (1), but in the blue and red channel signals, p(t) is not contained, only s(t) is contained. Then the degenerate signal The formula of the RGB three-channel original measurement signal is as follows:

[0059] C nr (t) = i(t) + u sr · s(t) (2)

[0060] C ng (t) = i(t) + u sg · s(t) + u pg · p(t) (3)

[0061] C nb (t) = i(t) + u sb · s(t) (4)

[0062] The goal is to estimate p(t) given C n (t), u s and u p , estimate u s and u p from the raw measurement signal collected by the noiseless camera, and directly use the learned u s and u p .

[0063] u s = [0.6265, 0.5479, 0.5526] (5)

[0064] u p = [0.3435, 0.7968, 0.4972] (6)

[0065] where u s and u p are unit vectors, that is:

[0066]

[0067] ① Use C n to estimate s(t)

[0068] Due to video compression or camera quantization error, etc., there is still a lot of noise in the actual raw measurement signal obtained in the experiment, mainly manifested as burr, high frequency; In order to eliminate the influence of this part of noise on the algorithm, the raw measurement signal needs to be preprocessed. Since this part of noise belongs to high frequency noise, median filter is selected to filter each channel signal respectively; Because in the red and blue channels, the signal intensity (amplitude) of p(t) is much lower than that of s(t), in the process of estimating s(t), it is reasonable to assume that p(t) can be ignored, i(t) is a constant, that is

[0069] s(t) can be estimated by least squares method:

[0070] Combined with formula (7), the analytical solution of this problem is obtained:

[0071]

[0072] 2. Estimate p(t) from green channel signal.

[0073] According to the green channel signal formula 3, it can be derived simply that,

[0074]

[0075] Step 2, Synthesize the noise-free observation signal

[0076] Substitute the known u s , u p and the estimated s(t) and p(t) into formula (1), and the calculation formula of the synthesized original measurement signal can be obtained:

[0077]

[0078] Step 3, Extract the blood volume pulse (BVP) signal

[0079] The plane orthogonal surface method (POS) is used to extract the blood volume pulse (BVP) signal from the trajectory of the remote photoplethysmography (rPPG), which is a high-performance, simple algorithm and widely used. POS defines a projection plane that is orthogonal to the vector [1,1,1] T , which is actually a plane that is orthogonal to the time-normalized skin color to eliminate the dependence of the skin color.

[0080] Taking a single light source (such as a fluorescent lamp) as an example, the skin pulsation is usually the largest in the G channel, followed by the B channel and the R channel. Based on such channel ordering and the requirement of the projection signal, the time-normalized trajectory is projected onto two vectors defined by the projection matrix: P p = [0,1,-1; -2,1,1], where each row represents a mutually orthogonal projection axis, which actually combines the time-normalized RGB signals into:

[0081] S1(t) = I nG (t) - I nB (t) (14)

[0082] S2(t) = I nG (t) + I nB (t) - 2I nR (t) (15)

[0083] Mirror reflection distortion and pulse are physically separable on the projection plane. In order to separate the mirror reflection and pulse signal components, a-tuning processing is required for S1(t) and S2(t):

[0084] h(t) = S1(t) + aS2(t) (16)

[0085] where a = σ(St(t)) / σ(S2(t)) and σ(·) denotes the standard deviation, h(t) is the extracted BVP signal, also referred to as the POS feature. The alpha adjustment used has an attractive property: when pulsatility variations dominate S(t), S1(t) and S2(t) appear in phase. Adding the two in-phase signals together will increase the resulting signal strength, i.e. the value of a is not important at this time; when specular reflection variations dominate S(t), S1(t) and S2(t) appear out of phase. a can pull / push the specular variation strength of one signal to the same level as the other, i.e. σ(S1) = σ(a-S2). Adding two out-of-phase signals with the same amplitude will cancel out the specular reflection distortion, however, when the pulsatility strength and the specular reflection strength are very close, its performance becomes sub-optimal, i.e. a is driven by a mixture of both, thus not well defined.

[0086] The operations discussed above are all within the scope of a time window. For a long time detection video, the windowed outputs are concatenated to get a long time heart rate signal. Specifically, given a video of length N, first the sequence is segmented into segments of length T, the proposed algorithm is applied to get the windowed outputs, and the overlap adding method is applied to get the final heart rate output.

Claims

1. A heart rate detection method for compressed video, characterized by, The method comprises the following steps: Step 1: Estimate the skin reflection model parameters The rPPG-based physiological index measurement algorithm reflects the change of blood volume by measuring the difference in skin absorption and reflection of different color lights, and the correlation of the signals contained in the RGB three channels and the heart rate signal is different. The G channel contains the strongest heart rate signal, and the pulse signal intensity in the R and B channels is much lower than the time signal, so the existence of the pulse signal can be ignored in the process of estimating the time signal. By modeling the skin absorption and reflection of different color channels, the difference and complementarity of skin absorption of different wavelengths of light are utilized to combine multiple color channels to estimate the pulse signal and synthesize a noise-free observation signal, so as to measure the heart rate physiological index; Step 2: Synthesize a noise-free observation signal Based on the double reflection model, the reflection of each skin pixel in the recorded image sequence is defined as a time-varying function in the RGB channel, which is caused by body movement and blood pulsation. According to the skin reflection model, the model parameters estimated are substituted to synthesize a noise-free original measurement signal; Step 3: Extract the blood volume pulse BVP signal The blood volume pulse signal extraction adopts the plane orthogonal surface method POS, which uses a plane orthogonal to the skin color in the time-normalized RGB space to extract the pulse, and uses a projection matrix to project the RGB signal to a range most likely to contain the pulse signal; The process of step 1 is as follows: Estimate the skin reflection model parameters: First, according to the skin reflectance model, the original measurement signal collected by a normal camera after time normalization , L is the window length, as follows: （1）； Based on the dichromatic reflection model, The specular reflection consists of specular and diffuse reflection from the skin surface. Due to body motion affecting the geometry between the light source, the skin surface and the camera, the specular reflection mainly contains motion-induced changes, i.e. unwanted motion artifacts; the specular reflection contains specular changes signals, i.e. inter-pulse signals, denoted as , is the specular reflection related vector; a small portion of the incident light penetrates the skin to reach the microvessels, this portion becomes diffuse reflection, which contains the target pulse signal, denoted as , is the vibration direction of the pulse; both components are proportional to the water level of light intensity, so they are modulated by the intensity change signal denoted as ; For a camera with too much noise, assume that the green channel signal still satisfies equation (1), but in the blue and red channel signals, instead of , only is included; The degraded signal The RGB three-channel raw measurement signal is formulated as follows: （2）； （3）； （4）； The goal is to estimate , and given , estimate and from raw measurement signals collected by a noiseless camera, directly using learned and ; （5）； （6）； wherein the requirements and are unit vectors, i.e.: （7）； （8）； ①utilize estimate ; Since in the red-blue channel, the signal strength is much lower than in the red-green channel, the estimation of can be assumed to be negligible, is constant, i.e. （9）； By least square method : （10）； ②Combined with formula (7), the analytical solution of this problem is obtained: （11）； Estimation by green channel signal ; According to the green channel signal formula 3, after simple deduction, （12）。 2. The compression video oriented heart rate detection method of claim 1, wherein: The step 2 process is to synthesize a noise-free observation signal, substitute the known , and the estimated and into equation (1) to obtain the calculation formula of the synthesized original measurement signal: （13）。 3. The compression video oriented heart rate detection method according to claim 1 or 2, characterized in that: The noise-free original measurement signal obtained from step 1 and step 2 is used to extract the blood volume pulse BVP signal through the POS algorithm, and the process is as follows: To extract blood volume pulse (BVP) signal, a planar orthogonal surface (POS) method is used to extract blood volume pulse (BVP) signal from remote photoplethysmography (rPPG) traces. POS defines a projection plane that is orthogonal to a vector which is actually a plane orthogonal to the time-normalized skin color to eliminate the dependence of skin color. For a single light source, skin pulsation is usually the largest in the G channel, followed by the B channel and the R channel; Based on such channel ordering and projection signal requirements, the time-normalized trajectories are projected onto two vectors defined by the projection matrix: where each row represents one of the mutually orthogonal projection axes, which in effect combines the time-normalized RGB signals as: （14）； （15）； The specular reflection distortion and the pulse are physically separable on the projection plane, in order to separate the mirror reflection and the pulse signal component, and An α-tuning process is required: （16）； wherein simultaneously denotes the standard deviation, is the extracted BVP signal, also called the POS feature; using an alpha adjustment has an attractive property: when the pulsatile variation dominates , and appear in phase, adding the two in-phase signals together will increase the resulting signal strength, i.e. the value of a is not important at this time; when the specular reflection variation dominates , and appear out of phase, α pulling / pushing the specular variation strength of one signal to the same level as the other, i.e. ; adding two out-of-phase signals with the same amplitude will cancel the specular reflection distortion, however, when the pulsatile strength and the specular reflection strength are very close, its performance becomes sub-optimal, i.e. α is driven by a mix of both, thus not well-defined; For long time detection video, the windowed outputs are concatenated to get the long time pulse signal. Given a video of length , the sequence is first divided into segments of length , the proposed algorithm is applied to get the windowed outputs, and the overlap-add method is applied to get the final pulse signal output.

Images