Non-contact robust heart rate detection method and system for long-distance and large depth of field scenes

Abstract

The invention discloses a non-contact robust heart rate detection method and system for long-distance and large depth of field scenes, comprising the following steps: S1: in a long-distance scene, using an image acquisition device to collect facial video data of a user, and obtaining a close-up facial image of the user; S2: using a face detection algorithm to obtain a face area, and in a long-distance and high-dynamic scene, obtaining a stable area SROI in the face as a detection area, and using a stable region of interest tracking algorithm to determine the detection area in each frame of the image; S3: preprocessing, segmenting and dimensionality reduction of the SROI image sequence, and generating multi-channel original observation data from the SROI two-dimensional image sequence; S4: using the FastICA algorithm to separate multiple physiological signals from the multi-channel original observation signals, determining a BVP signal from the multi-channel physiological signals by calculating the short-time kurtosis method, and performing heart rate calculation.

Description

Technical Field

[0001] The present invention relates to the field of detection, recognition and analysis of human physiological information, and in particular to a non-contact robust heart rate detection method and system for long-distance and large depth-of-field scenes. Background Art

[0002] Heart rate is a basic vital sign of the human body and an important physiological indicator that reflects the health status of the human body. Convenient and efficient heart rate detection is not only of great significance for early prevention and diagnosis of diseases, but also plays an important role in daily health monitoring, sports performance evaluation, fatigue monitoring, psychological evaluation and other fields.

[0003] In the traditional medical field, contact-based heart rate detection is generally used. The disadvantage of this method is that it requires the use of professional equipment and the subjective cooperation of the user, resulting in a poor user experience and is not suitable for daily long-term heart rate monitoring. Imaging Photoplethysmography (IPPG, also known as rPPG, remote Photoplethysmography) technology is a method for extracting heart rate based on the periodic changes of blood flow signals reflected by body surface images. Compared with traditional heart rate monitoring technology, this method performs non-contact heart rate measurement based on body surface video image information. It is independent of professional equipment, easy to operate, low cost, good user experience, and easy for daily use.

[0004] Since it is highly dependent on blood flow change information in human body surface image sequences, the classic IPPG algorithm needs to collect close-up images of the human body surface (the camera should be within 30 cm from the face). However, in long-distance and large depth-of-field application scenarios such as interviews, trials, sports event broadcasts, and drone aerial photography, the image sensor is far away from the person and there are no objective conditions for close-range facial image collection, which limits the application of classic IPPG technology.

[0005] In order to realize non-contact heart rate detection in long-distance scenes, it is necessary to use a variable focal length optical sensor to obtain close-up images of the human body. However, due to the limited field of view of the optical sensor, a slight displacement of a person in a scene with a large depth of field will cause a violent shaking of the human body in the video image, making it difficult to obtain a stable body surface area for heart rate detection. Therefore, it is necessary to propose a non-contact robust heart rate detection method and system for long-distance and large depth of field scenes to solve the above problems. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a non-contact robust heart rate detection method and system for long-distance and large depth of field scenes, which can realize non-contact heart rate detection by acquiring a stable body surface area in long-distance scenes.

[0007] In order to solve the above technical problems, a technical solution adopted by the present invention is to provide a non-contact robust heart rate detection method for long-distance and large depth of field scenes, comprising the following steps:

[0008] S1: In a long-distance scenario, use an image acquisition device to collect facial video data of the user to obtain a close-up facial image of the user;

[0009] S2: Use the face detection algorithm to obtain the face area in the acquired close-up image of the user's face, and in the high-dynamic scene at a long distance, obtain the stable region SROI in the face as the detection area, and use the stable region of interest tracking algorithm to determine the detection area in each frame of the image;

[0010] S3: preprocessing, segmenting and dimension reduction of the SROI image sequence, and generating a multi-channel original observation signal from the SROI two-dimensional image sequence;

[0011] S4: The FastICA algorithm is used to separate multiple physiological signals from the multi-channel original observation signals, the BVP signal is determined from the multi-channel physiological signals separated by FastICA by calculating the short-time kurtosis, and the heart rate is calculated based on the BVP signal.

[0012] In a preferred embodiment of the present invention, in step S1, in a long-distance scene, when using an image acquisition device to collect facial video data of a user, the distance between the image acquisition device and the user's face is not less than 3 meters. Under this long-distance condition, the proportion of pixels in the facial area in the video frame is less than 8%.

[0013] Furthermore, the image acquisition device adopts a variable magnification high-definition camera or an optical zoom image sensor. After using optical zoom, the resolution of the acquired close-up face image is not less than 640*480, the pixel value of the face area in the video image accounts for more than 78%, and the video frame rate is not less than 24fps.

[0014] In a preferred embodiment of the present invention, in step S2, the method for obtaining a stable region SROI in the face as a detection region is: using a short-time spatial entropy algorithm to process continuous video frames of the face region, obtaining a high entropy region corresponding to dynamic facial changes in a continuous video image sequence, eliminating the high entropy region and then obtaining the intersection of low entropy regions in each frame, and using this region as a stable SROI region.

[0015] In a preferred embodiment of the present invention, in step S2, the stable region of interest tracking algorithm uses a particle filter algorithm to track the SROI region.

[0016] In a preferred embodiment of the present invention, in step S3, preprocessing the SROI image sequence includes video image sequence filtering, averaging and variance normalization.

[0017] In a preferred embodiment of the present invention, in step S3, the step of segmenting the SROI image sequence includes:

[0018] The SROI (size is M×N) of each frame is divided into three parts with equal spacing in the horizontal direction. Each part is further divided into three channels according to the RGB value of the image. In fact, 9 channels of data are obtained. Taking a video containing T frame images as an example, 9 one-dimensional vectors are obtained after dimensionality reduction:

[0019]

[0020] In a preferred embodiment of the present invention, in step S3, the step of reducing the dimension of the SROI image sequence includes:

[0021] Based on coherent averaging, the three color channel video data of R, G, and B in each region of the segmented SROI are processed by spatial pixel averaging according to the following formula:

[0022]

[0023] Where H and W are the length and width of the ROI area of ​​the facial video image, N is the total number of video image frames, and X n Represents the one-dimensional signal of the R, G, and B channels of the ROI area, x n (h, w) are the color intensity values ​​of the corresponding channels of the pixel point (h, w) in the nth frame, and h, w are the spatial coordinates of the image area.

[0024] In a preferred embodiment of the present invention, in step S4, the BVP signal is automatically selected from the multi-channel signal separated by FastICA using the short-time kurtosis algorithm, and the signal with the largest kurtosis value is the BVP signal. The kurtosis calculation method of the zero-mean signal z is as follows:

[0025]

[0026] In order to solve the above technical problems, another technical solution adopted by the present invention is to provide a non-contact robust heart rate detection system for long-distance and large depth of field scenes, comprising:

[0027] The facial video data acquisition module is used to collect facial video data of the user using an image acquisition device in a long-distance scene to obtain a close-up facial image of the user;

[0028] The facial stable region of interest detection module is used to obtain the face area using the face detection algorithm, and obtain the stable region SROI in the face as the detection area in a long-distance high-dynamic scene;

[0029] The facial stable region of interest tracking module is used to track the SROI region in each frame image using a stable region of interest tracking algorithm when the SROI region moves in the facial video;

[0030] A multi-channel original observation signal generation module is used to pre-process, segment and reduce the dimension of the SROI image sequence, and generate a multi-channel original observation signal from the SROI two-dimensional image sequence;

[0031] The BVP signal determination and heart rate calculation module is used to separate multiple physiological signals from multi-channel original observation signals using the FastICA algorithm, determine the BVP signal from the multi-channel physiological signals separated by FastICA by calculating the short-time kurtosis, and calculate the heart rate based on the BVP signal.

[0032] The beneficial effects of the present invention are as follows: in a scene with a large depth of field greater than 3 meters, the present invention obtains a close-up image of the subject's face through an optical zoom camera, obtains a stable facial region of interest under large depth of field conditions through face detection and a stable region of interest detection algorithm, and uses a region of interest tracking algorithm to maintain stable acquisition of the region, effectively solving the problem of instability of the acquired body surface area caused by SROI region movement caused by user head movement and relative position change between the user's head and the acquisition device in the facial video; multi-channel one-dimensional observation signals are generated by SROI image sequence segmentation and dimensionality reduction, multi-channel source signals are separated from the multi-channel observation signals with the help of the FastICA algorithm, and the BVP signal is selected using spectral kurtosis to estimate the heart rate, thereby realizing non-contact robust heart rate detection in long-distance and large depth of field scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of a non-contact robust heart rate detection method for long-distance and large depth of field scenes of the present invention;

[0034] Figure 2 It is a schematic diagram comparing the application scenarios of the present invention and the traditional application scenarios of IPPG;

[0035] Figure 3 It is a schematic diagram of the SROI area positioning and tracking method in a long-distance scene;

[0036] Figure 4 It is a schematic diagram of segmenting a facial image;

[0037] Figure 5It is a structural block diagram of the non-contact robust heart rate detection system for long-distance and large depth of field scenes. DETAILED DESCRIPTION

[0038] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0039] See also Figure 1 , the embodiment of the present invention includes:

[0040] A non-contact robust heart rate detection method for long-distance and large depth of field scenes comprises the following steps:

[0041] S1: In a long-distance scenario, use an image acquisition device to collect facial video data of the user to obtain a close-up facial image of the user;

[0042] Combination Figure 2 , the application scenario of the present invention (see Figure 2 (b)) and traditional application scenarios (see Figure 2 (a)) is a comparison of Figure 2 As shown. In a long-distance and large depth of field scene, the distance between the image acquisition device and the face is not less than 3 meters. Under this long-distance condition, the proportion of pixels in the face area in the video frame is less than 8%. With the help of an image acquisition device, a close-up image of the face is obtained in a long-distance and large depth of field scene. The user is required to keep the body as still as possible during the facial video acquisition process to reduce noise interference. Preferably, the image acquisition device uses a variable magnification high-definition camera or an optical zoom image sensor. After using optical zoom, the resolution of the acquired close-up face image is not less than 640*480, the proportion of pixels in the face area in the video image is higher than 78%, and the video frame rate is not less than 24fps.

[0043] S2: using a face detection algorithm (in the acquired close-up image of the user's face) to obtain the face area, and in a long-distance high-dynamic scene, obtain a stable region SROI in the face as the detection area, and use a stable region of interest tracking algorithm to determine the detection area in each frame of the image in subsequent processing; the specific steps include:

[0044] First, the Viola-Jones algorithm is used to detect the face area, and then the SROI is further determined from the detected face area;

[0045] Then, the stable region of interest (SROI) detection algorithm is used to obtain the stable facial region of interest under large depth of field conditions. The short-time spatial entropy algorithm is used to process the continuous video frames of the face area to obtain the high entropy value area corresponding to the dynamic changes of the face in the continuous video image sequence. After eliminating this high entropy area, the intersection of the low entropy value areas of each frame is obtained, and this area is used as the SROI. The specific steps are as follows:

[0046] Get the local color histogram of the face Where i and j represent the central pixel position, T represents the number of frames accumulated in the histogram, and L represents the side length of the face area rectangle (in pixels). Then, the probability density function of the pixel value at each pixel position of this local image can be obtained:

[0047]

[0048] The numerator represents a component of the normalized histogram of a certain pixel position, and the denominator represents the total cumulative histogram of a specific rectangular area after short-term accumulation. In practice, L×L×T pixels are used to calculate the color histogram H of the current point (i, j). i，j，q , where q = 1, ..., Q represents the number of quantization intervals of the color histogram. Therefore, each component of the color histogram obtained after each statistical operation is {H i，j，1 , ..., H i，j，Q}, the probability of any component q can be obtained by directly normalizing this local color histogram:

[0049]

[0050] The denominator in the above formula is the total number of short-term pixels in a specific local area. After processing using the above formula, the components of the histogram satisfy:

[0051]

[0052] Then the short-time spatial entropy corresponding to this point is obtained as

[0053]

[0054] (4) The left side E L i，j，T It is the short-term spatial entropy of the L×L rectangular window sliding across T frames at the pixel point (i, j). The short-term spatial entropy of each pixel constitutes an image. The bright area in this image is the dynamic area, and the stable static area is the black low-brightness area. The intersection of the minimum low-brightness inscribed area in the T-frame image is the SROI, as shown in Figure 3 shown.

[0055] Finally, the region of interest tracking (ROI tracking) algorithm is used to maintain stable acquisition of the region (SROI stable region):

[0056] In view of the movement of the SROI area in facial videos caused by the movement of the user's head and the change in the relative position between the user's head and the acquisition device, a particle filtering algorithm is used to track the SROI area.

[0057] S3: preprocessing, segmenting and dimension reduction of the SROI image sequence, and generating multi-channel original observation data from the SROI two-dimensional image sequence;

[0058] The SROI image sequence is preprocessed including video image sequence filtering, de-meaning and variance normalization. The specific steps are: using a 0.7-8 Hz bandpass filter to filter the coherent averaged signal (the one-dimensional signal after the image sequence is reduced in dimension), and using the Z_Score normalization method to normalize the three color channel video signals of R, G, and B respectively.

[0059] y(t)=(x(t)-μ) / σ (5)

[0060] Where μ and σ are the mean and standard deviation of the original data, respectively. When the original data is approximately normally distributed, the data above the mean μ is normalized to a positive value, while the data below μ is normalized to a negative value. At the same time, the expectation of Z_Score is 0 and the standard deviation is 1.

[0061] Combination Figure 4 , the step of segmenting the SROI image sequence comprises:

[0062] The SROI (size is M×N) of each frame is divided into three parts with equal spacing in the horizontal direction. Each part is further divided into three channels according to the RGB value of the image. In fact, 9 channels of data are obtained. Taking a video containing T frame images as an example, 9 one-dimensional vectors are obtained after dimensionality reduction:

[0063]

[0064] The step of reducing the dimension of the SROI image sequence comprises:

[0065] Based on coherent averaging, the three color channel video data of R, G, and B in each region of the segmented SROI are processed by spatial pixel averaging according to the following formula:

[0066]

[0067] Where H and W are the length and width of the ROI area of ​​the facial video image, N is the total number of video image frames, and X nRepresents the one-dimensional signal of the R, G, and B channels of the ROI area, x n (h, w) are the color intensity values ​​of the corresponding channels of the pixel point (h, w) in the nth frame, and h, w are the spatial coordinates of the image area.

[0068] S4: The FastICA algorithm is used to separate multiple physiological signals from the multi-channel original observation signals, the BVP signal is determined from the multi-channel physiological signals separated by FastICA by calculating the short-time kurtosis, and the heart rate is calculated based on the BVP signal.

[0069] The FastICA algorithm is used to separate BVP and other physiological signals and noise from the preprocessed R, G, and B color channel signals. Since the independent component analysis algorithm has inherent output sorting uncertainty, and considering that the BVP signal has a strong periodicity and a large kurtosis value, the short-time kurtosis algorithm is used to automatically select the BVP signal from the multi-channel signal separated by FastICA, and the signal with the largest kurtosis value is the BVP signal. The kurtosis calculation method of the zero-mean signal z is as follows:

[0070]

[0071] The BVP signal after blind source separation is filtered twice using a 4Hz low-pass filter, and the heart rate is estimated by finding the spectrum peak of the BVP.

[0072] See also Figure 5 The present invention also provides a non-contact robust heart rate detection system for long-distance and large depth of field scenes, including:

[0073] The facial video data acquisition module is used to collect facial video data of the user using an image acquisition device in a long-distance scene to obtain a close-up facial image of the user;

[0074] The facial stable region of interest detection module is used to obtain the face area using the face detection algorithm, and obtain the stable region SROI in the face as the detection area in a long-distance high-dynamic scene;

[0075] The facial stable region of interest tracking module is used to track the SROI region in each frame image using a stable region of interest tracking algorithm when the SROI region moves in the facial video;

[0076] A multi-channel original observation signal generation module is used to pre-process, segment and reduce the dimension of the SROI image sequence, and generate a multi-channel original observation signal from the SROI two-dimensional image sequence;

[0077] The BVP signal determination and heart rate calculation module is used to separate multiple physiological signals from multi-channel original observation signals using the FastICA algorithm, determine the BVP signal from the multi-channel physiological signals separated by FastICA by calculating the short-time kurtosis, and calculate the heart rate based on the BVP signal.

[0078] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A non-contact robust heart rate detection method for long-distance and large depth of field scenes, characterized in that: The following steps are involved: S1: In a long-distance scenario, use an image acquisition device to collect facial video data of the user to obtain a close-up facial image of the user; In a long-distance scene, when an image acquisition device is used to collect facial video data of a user, the distance between the image acquisition device and the user's face is not less than 3 meters, and under this long-distance condition, the proportion of pixels in the face area in the video frame is less than 8%; the image acquisition device uses a variable-magnification high-definition camera or an optical zoom image sensor. After using optical zoom, the resolution of the acquired close-up face image is not less than 640*480, the proportion of pixels in the face area in the video image is higher than 78%, and the video frame rate is not less than 24fps; S2: Use the face detection algorithm to obtain the face area in the acquired close-up image of the user's face, and in the high-dynamic scene at a long distance, obtain the stable area SROI in the face as the detection area, and use the stable region of interest tracking algorithm to determine the detection area in each frame image; the specific steps include: First, the Viola-Jones algorithm is used to detect the face area, and then the SROI is further determined from the detected face area; Then, a stable region of interest detection algorithm is used to obtain a stable facial region of interest under a large depth of field condition. The short-time spatial entropy algorithm is used to process the continuous video frames of the face area to obtain the high entropy region corresponding to the dynamic changes of the face in the continuous video image sequence. After eliminating this high entropy region, the intersection of the low entropy regions of each frame is obtained, and this region is used as the stable SROI region. Finally, the region of interest tracking algorithm is used to maintain stable acquisition of the stable SROI area; S3: Preprocessing, segmenting and reducing the dimension of the SROI image sequence, generating a multi-channel original observation signal from the SROI two-dimensional image sequence; the steps of segmenting the SROI image sequence include: The SROI of each frame is divided into three parts with equal spacing in the horizontal direction, and each part is divided into three channels according to the RGB value of the image. In fact, 9 channels of data are obtained. Taking a video containing T frame images as an example, 9 one-dimensional vectors are obtained after dimensionality reduction: The step of reducing the dimension of the SROI image sequence comprises: Based on coherent averaging, the three color channel video data of R, G, and B in each region of the segmented SROI are processed by spatial pixel averaging according to the following formula: Where H and W are the length and width of the ROI area of ​​the facial video image, N is the total number of video image frames, and X n Represents the one-dimensional signal of the R, G, and B channels of the ROI area, x n (h, w) are the color intensity values ​​of the corresponding channels of the pixel point (h, w) in the nth frame, and h, w are the spatial coordinates of the image area; S4: Use the FastICA algorithm to separate multiple physiological signals from the multi-channel original observation signal, determine the BVP signal from the multi-channel physiological signal separated by FastICA by calculating the short-time kurtosis, and calculate the heart rate based on the BVP signal; use the short-time kurtosis algorithm to automatically select the BVP signal from the multi-channel signal separated by FastICA, and the signal with the largest kurtosis value is the BVP signal. The kurtosis calculation method of the zero-mean signal z is as follows:

2. The non-contact robust heart rate detection method for long-distance and large depth of field scenes according to claim 1 is characterized in that: In step S2, the stable region of interest tracking algorithm uses a particle filter algorithm to track the SROI region.

3. The non-contact robust heart rate detection method for long-distance and large depth of field scenes according to claim 1 is characterized in that: In step S3, the SROI image sequence is preprocessed including video image sequence filtering, averaging and variance normalization.

4. A non-contact robust heart rate detection system for long-distance and large depth of field scenes, using the non-contact robust heart rate detection method for long-distance and large depth of field scenes as claimed in any one of claims 1 to 3, characterized in that: include: The facial video data acquisition module is used to collect facial video data of the user using an image acquisition device in a long-distance scene to obtain a close-up facial image of the user; The facial stable region of interest detection module is used to obtain the face area using the face detection algorithm, and obtain the stable region SROI in the face as the detection area in a long-distance high-dynamic scene; The facial stable region of interest tracking module is used to track the SROI region in each frame image using a stable region of interest tracking algorithm when the SROI region moves in the facial video; A multi-channel original observation signal generation module is used to pre-process, segment and reduce the dimension of the SROI image sequence, and generate a multi-channel original observation signal from the SROI two-dimensional image sequence; The BVP signal determination and heart rate calculation module is used to separate multiple physiological signals from multi-channel original observation signals using the FastICA algorithm, determine the BVP signal from the multi-channel physiological signals separated by FastICA by calculating the short-time kurtosis, and calculate the heart rate based on the BVP signal.

Images