Smart phone heart rate detection method based on rppg and deep neural network

Abstract

The invention relates to the field of heart rate signal detection, in particular to a non-contact heart rate signal detection method based on rPPG and a deep neural network, and the method comprises the steps: 1, obtaining color videos of faces of a plurality of target persons, and extracting the faces through a deep neural network model; 2, preprocessing the video data, and setting a facial region of interest (ROI); 3, a large amount of noise generated by the environment and shooting equipment is filtered out according to the minimum absorption rate of the blood to the red component; and 4, converting the time domain signal into a frequency domain signal by using a fast Fourier transform algorithm, further filtering the data, and taking the highest point of the small section of frequency as the heart rate. According to the method, the deep neural network technology is utilized, and the face tracking technology is combined, so that remote multi-target simultaneous heart rate acquisition can be realized by using a common camera, and the universality of rPPG is improved.

Description

Technical Field

[0001] The present invention belongs to the field of physiological signal detection, and in particular relates to a non-contact heart rate signal detection method based on rPPG and deep neural network. Background Art

[0002] Heart rate is an extremely important physiological indicator, which refers to the number of human heartbeat cycles per unit time. It can reflect the health status, breathing status, sympathetic nerve activity, emotional state and other information of the subject. Therefore, real-time heart rate monitoring is conducive to the prevention of cardiovascular diseases. With the advancement of technology, the measurement methods of physiological signals such as heart rate are constantly evolving. From the early pulse-taking method and the measurement method based on pressure sensors to the commonly used measurement methods based on ECG and PPG (PhotoPlethysmo Graphy), the measurement of physiological signals such as heart rate has become more convenient and accurate. Most of these methods are based on the biophysical mechanisms of two important physiological signals, ECG and BVP, for measurement.

[0003] In order to monitor heart rate changes in real time, these portable wearable devices on the market, such as electrocardiogram (ECG) and photoplethysmogram (PPG), have certain limitations. ECG extracts HR by sensing the human heart current through electrodes, but this causes discomfort to the subject, the operation process is cumbersome, and only one-to-one measurements can be performed. And the scope of application is narrow, for example, it is not applicable to newborns, burns, and trauma patients. PPG is a simple, low-cost HR acquisition method, but it is still a contact measurement method, and long-term measurement will still make the subject feel uncomfortable. In addition, the contact measurement based on PPG is not highly resistant to interference, and the interference caused by contact measurement to the subject cannot be eliminated.

[0004] In addition, some video-based methods for obtaining heart rate have emerged in recent years, such as image photoplethysmography (iPPG) and remote photoplethysmography (rPPG). iPPG is a method for obtaining PPG signals from non-contact video. It has the advantages of non-contact measurement, easy implementation and machine operation, but when using iPPG to detect heart rate, a standard white board needs to be placed, and the color change of the white board in the video is used to correct the color of the skin. However, it is not suitable for general environments, and it is difficult to achieve real-time monitoring in daily life, let alone detect heart rate from a distance. Remote photoplethysmography (rPPG) provides an innovative alternative to obtain rPPG signals. This technology uses a camera to detect subtle color changes on the surface of human skin and analyzes them to obtain heart rate.

[0005] In the early stages of rPPG research, researchers separated the three RGB channels in the video into three independent sources and considered one of them to be the waveform of the heart rate. Although good results were obtained through this method, on the one hand, it is not suitable for analysis in the case of poor video quality, and on the other hand, when the external noise is large, the failure to separate the noise leads to the inability to obtain an effective heart rate. Later, a method of obtaining heart rate based on a thermal camera using a PCA algorithm was introduced, but this method is costly and not suitable for daily use. In addition to the above methods, rPPG can also use Butterworth passband filters, random forest regression (FR) and fast Fourier transform (FFT) for heart rate measurement.

[0006] However, all these methods cannot effectively separate the heart rate signal from the noise brought by the complex environment and objects, and their lighting robustness and available distance are very limited, which restricts their practical application. There are two main problems with these methods. First, the effective measurement distance is too short to conveniently monitor the heart rate. Second, the detection effect under poor lighting conditions and drastic changes in illumination has not been fully studied. Summary of the invention

[0007] Purpose of the invention: The purpose of the present invention is to provide a non-contact heart rate signal detection method based on rPPG and deep neural network, to improve the robustness and accuracy of heart rate measurement, to increase its detection distance and to be able to adapt to poor lighting conditions.

[0008] Technical solution: A non-contact heart rate signal detection method based on rPPG and deep neural network of the present invention comprises the following steps:

[0009] Step 1, obtain color videos of the faces of several target persons and read frames from the videos by using OpenCV and a deep neural network (DNN) model, preprocess the video data, and use face recognition to extract facial information of multiple test subjects separately;

[0010] Step 2: For the extracted facial information, 68 facial feature points are obtained to construct the facial reference system of each subject, and based on this, the area with the most obvious facial skin color change and the most intense heartbeat signal is cut out as the processing region (ROI) as the application object of the subsequent algorithm;

[0011] Step 3: Since blood absorbs green more than red, the change of green component is approximated as noise. The obtained image data is divided into three channels, RGB, and the values ​​of the RGB channels of each image are subtracted from each other to obtain a waveform after simple smooth filtering, thereby filtering out a large amount of noise generated by the environment and the shooting equipment;

[0012] Step 4: Use the fast Fourier transform algorithm (FFT) to convert the time domain signal into a frequency domain signal, delete the heart rate values ​​that are not within the normal range of human heart rate of 40 / min to 120 / min as noise from the measured frequency domain signal, and take the highest frequency point after filtering as the heart rate.

[0013] Further, step 1 is specifically: obtaining color videos of the faces of several target persons and reading frames from the videos by using OpenCV and a deep neural network (DNN) model, preprocessing the video data, and extracting facial information of multiple test objects separately by using face recognition; specifically including the following steps:

[0014] Step 1.1, first define two string variables pb_file_path and pbtxt_file_path, which are used to store the paths of face detection model files and model structure files respectively. These files are deep neural network models trained using TensorFlow.

[0015] Step 1.2. Use OpenCV's cv::dnn::readNetFromTensorflow() function to read the deep neural network model from the TensorFlow model file and model structure file and load it into the cv::dnn::Net object.

[0015] Step 1.3. Create a VideoCapture object cap to capture video frames from the camera.

[0002] Step 1.4, enter the video capture loop, read a frame of video from the camera in each iteration, and store it in frame:

[0016] Step 1.5. Convert the current frame to the format required for neural network input. Use the cv::dnn::blobFromImage() function to convert the frame into a 4-dimensional blob, set the blob as the input of the neural network, and obtain the output of the model through forward propagation.

[0017] Step 1.6: Process the output results, extract the detected face information, and display the current frame with the face detection results.

[0018] Step 1.7, get the values ​​of R(k), G(k), and B(k). Specifically, k represents the number of frames of the image, R(k) is the sum of the ROIs in the red channel, G(k) is the sum of the ROIs in the green channel, and B(k) is the sum of the ROIs in the blue channel.

[0019] Further, step 2 is specifically as follows: for the acquired facial information, 68 facial feature points are respectively acquired to construct the facial reference system of each of the tested objects, and based on this, the area where the facial skin color change is most obvious and the heartbeat signal is most intense is cut out as the processing region (ROI) as the application object of the subsequent algorithm; specifically includes the following steps:

[0020] Step 2.1: Get the heart rate signal x per unit area (n) : Where n is the number of sequence images, x (n) is the heart rate signal obtained, R (i，j) With G (i，j) They are the red and green component values ​​of the i-th row and j-th column in the 10×10 area respectively.

[0021] Step 2.2, obtain the vector spectrum of the discrete signal. According to the fast Fourier transform formula, the heart rate signal can be obtained and then the signal is fast Fourier transformed to obtain the vector spectrum of the discrete signal: By taking the vector spectrum obtained by equation (2) and applying it to equation (3), we can obtain the wave contents of different frequencies in the current 10×10 region. M (N) =|X (k) |#(3)

[0022] Step 2.3, obtain the distribution map of the real heart rate wave content, find the signal sensitive area ROI, and according to formula (3), substitute the real heart rate into the formula to obtain the content of the real heart rate in the area, and then you can get the real heart rate wave content in the area. Through formula (4), copy the value to the pixel represented by the area in the distribution map (row m, column n), and after completing the calculation of all areas, you can get the distribution map of the real heart rate wave content. IMG (m，n) =M (Heart Rate) #(4) By reflecting the calculated results in the graph, we can get a heart rate distribution graph, where the color represents the content of the wave. It can be clearly seen that the β region is the area where the heart rate waveform is concentrated. This area is used as the ROI and the application object of the subsequent algorithm.

[0023] Furthermore, step 3 is specifically as follows: after determining the sensitive area of ​​the palm heart rate signal, superimpose the values ​​of all red channels in the area, and at the same time superimpose the values ​​of all green channels in the area, subtract the two values ​​and divide them by the area, and you can get the change in the heart rate signal value per unit area after the red and green channels are smoothed and filtered in the area. Divide the obtained image data into three channels, RGB, and subtract the values ​​of the red channel and the green channel of each image to get a waveform after a simple smoothing filter. It can be considered that this waveform removes some of the noise caused by changes in external and equipment factors. The formula used in this method is: Where n is the number of sequential images, x and y are the horizontal and vertical coordinates of the signal sensitive area, respectively. (n) is the heart rate signal obtained, R (x，y) With G (x，y) They are the red and green component values ​​of the x-th row and y-th column in the x×y area respectively.

[0024] Furthermore, step 4 is specifically: perform fast Fourier transform on the heart rate signal and modulus the obtained result to find the content of waves of different frequencies in the filtered waveform. Since the normal heart rate of a person is about 60 to 100 beats per minute, after filtering out the waves other than 40 / min to 120 / min, the highest point in this frequency range is the heart rate. Specifically, the following steps are included:

[0025] Step 4.1: The heart rate obtained from the filtered data x(n) is obtained by subtracting the G value from the R value after filtering the noise. FFT is used to obtain the vector spectrum of the heart rate signal. The FFT formula is as follows: Where n is the FFT resolution, and Y(k) is the vector sequence obtained by FFT calculation. Modulo operation is performed on Y(k) to obtain M(k) (frequency domain data of the filtered signal). The signal after frequency domain filtering is expressed as M(k) = |Y(k)|.

[0026] Step 4.2, M(k) represents the amount of signal in each given frequency band within the frequency range (Bd). The physiological range of human heart rate is between 40 and 220 beats per minute (bpm). Therefore, Hr L The value is set to 40bpm, Hr H Set to 220 bpm. Then, H and below HR L The heart rate value is removed from the measured frequency domain signal as noise. L and Hr H The maximum value of M(k) between the two is taken as the heart rate measured by rPPG.

[0027] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: 1. The present invention is easy to operate and low in cost. It relies on the camera equipment and computing platform of the smart phone, does not need to purchase additional hardware equipment, does not require professional knowledge to operate, and is easy to popularize;

[0028] 2. The invention has created a three-channel difference filtering algorithm, which can achieve extremely high stability even in the environment of sudden changes in ambient light. It is combined with the ICA algorithm to extract the main heartbeat signal, and has more relaxed environmental requirements and can adapt to some extreme environments;

[0029] 3. The present invention has more powerful measurement performance. Compared with similar international studies, the measurement distance of the present invention is nearly three times higher, while the necessary acquisition window time is shortened by 1 / 3;

[0030] 4. The present invention can realize multi-target heart rate measurement. Combining deep learning facial recognition technology with feature point monitoring, it can realize simultaneous measurement of multiple objects, greatly improving practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 is a flow chart of a method provided by an embodiment of the present invention; Figure 2 is a flow chart of preprocessing facial video provided by an embodiment of the present invention; Figure 3 The flowchart of selecting the signal sensitive area ROI provided by the embodiment of the present invention is Figure 4 The heart rate distribution diagram of the real heart rate wave content in step 3 provided in the embodiment of the present invention Figure 5 is a flow chart of signal collection and smoothing filtering provided by an embodiment of the present invention; Figure 6 is a flow chart of extracting heart rate provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0033] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0034] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0035] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0036] In the embodiments of the present invention, sometimes the subscripts such as W n It may be mistakenly written as a non-subscript form such as Wn. When the difference is not emphasized, the meanings they express are the same.

[0037] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0038] In this embodiment, a method for monitoring heart rate over a long distance based on rPPG is mainly to use a mobile phone to shoot a color video of a person's face within 4 meters, set a facial region of interest ROI, filter out a large amount of noise generated by the environment and the shooting equipment according to the absorption rate of blood to light of different wavelengths, find independent signals, use a fast Fourier transform algorithm (FFT) to convert the time domain signal into a frequency domain signal, further filter the data, and finally analyze the data to obtain an effective heart rate, such as Figure 1 As shown, the steps are as follows:

[0039] Step 1: First, use the smartphone camera to obtain color videos of the faces of several target persons. The half-minute to one-minute videos used for measurement are recorded by ordinary mobile phone cameras. The video resolution is 1080p and the frame rate is 60 frames. A total of five volunteers participated in the experiment, two women and three men, aged 20 to 47 years old, the frame rate is 30 frames per second, and the video length is about 1 minute.

[0040] Perform data preprocessing on face videos, such as Figure 2As shown, it includes: defining two string variables pb_file_path and pbtxt_file_path, which are used to store the paths of face detection model files and model structure files respectively. These files are deep neural network models trained using TensorFlow.

[0041] Use OpenCV's cv::dnn::readNetFromTensorflow() function to read the deep neural network model from the TensorFlow model file and model structure file and load it into the cv::dnn::Net object.

[0042] Create a VideoCapture object cap to capture video frames from the camera. Enter the video capture loop, read a frame of video from the camera in each iteration, and store it in frame:

[0043] Convert the current frame to the format required by the neural network input. Use The cv::dnn::blobFromImage() function converts the frame into a 4-dimensional blob, sets the blob as the input of the neural network, and obtains the output of the model through forward propagation. The output is processed to extract the detected face information and display the current frame with the face detection result.

[0044] Get the R(k), G(k) and B(k) values. Specifically, k represents the number of frames of the image, R(k) is the sum of the ROIs in the red channel, G(k) is the sum of the ROIs in the green channel, and B(k) is the sum of the ROIs in the blue channel.

[0045] Step 2: If Figure 3 As shown in the figure, for the acquired facial information, 68 facial feature points are obtained to construct the facial reference system of each object under test, and based on this, the area with the most obvious facial skin color change and the most intense heartbeat signal is cut out as the processing region (ROI) as the application object of the subsequent algorithm:

[0046] Step 2.1, in order to minimize the impact of facial tremors and camera noise. When measuring an area, calculate the average value of each pixel in the area. Since the measurement is an area, the noise will only affect the red component that accounts for a small proportion of the entire area. This can effectively reduce the red component noise caused by factors such as noise generated by some pixels and changes in the measurement area caused by facial tremors. Get the heart rate signal x per unit area (n) : Where n is the number of sequence images, x (n)is the heart rate signal obtained, R (i，j) With G (i，j) They are the red and green component values ​​of the i-th row and j-th column in the 10×10 area respectively.

[0047] Step 2.2, obtain the vector spectrum of the discrete signal. According to the fast Fourier transform formula, the heart rate signal can be obtained and then the signal is fast Fourier transformed to obtain the vector spectrum of the discrete signal: By taking the vector spectrum obtained by equation (2) and applying it to equation (3), we can obtain the wave contents of different frequencies in the current 10×10 region. M (N) =|X (k) |#(3)

[0048] Step 2.3, obtain the distribution map of the real heart rate wave content, find the signal sensitive area ROI, and according to formula (3), substitute the real heart rate into the formula to obtain the content of the real heart rate in the area, and then you can get the real heart rate wave content in the area. Through formula (4), copy the value to the pixel represented by the area in the distribution map (row m, column n), and after completing the calculation of all areas, you can get the distribution map of the real heart rate wave content. IMG (m,n) =M (Heart Rate) #(4) By reflecting the calculated results into the graph, we can get a heart rate distribution graph, such as Figure 4 As shown, the color represents the content of the wave. It can be clearly seen that the area where the heart rate waveform is concentrated is used as the ROI and the application object of the subsequent algorithm.

[0049] Step 3: After determining the sensitive area of ​​the palm heart rate signal, superimpose the values ​​of all red channels in the area, and also superimpose the values ​​of all green channels in the area. Subtract the two values ​​and divide them by the area to get the change in the heart rate signal value per unit area after the red and green channels are smoothed and filtered. Divide the obtained image data into three channels, RGB, and subtract the values ​​of the red channel and the green channel of each image to get a waveform after simple smooth filtering. It can be considered that this waveform removes some of the noise caused by changes in external and equipment factors. The formula used in this method is: Where n is the number of sequential images, x and y are the horizontal and vertical coordinates of the signal sensitive area, respectively. (n) is the heart rate signal obtained, R (x，y) With G (x，y)They are the red and green component values ​​of the x-th row and y-th column in the x×y area respectively.

[0050] Step 4: Perform a fast Fourier transform on the heart rate signal and find the modulus of the result to find the content of waves of different frequencies in the filtered waveform. Since the normal heart rate of a person is about 60 to 100 beats per minute, after filtering out the waves outside the range of 40 / min to 120 / min, the highest point in this frequency range is the heart rate. The specific steps include:

[0002] Step 4.1: The heart rate obtained from the filtered data x(n) is obtained by subtracting the G value from the R value after filtering the noise. FFT is used to obtain the vector spectrum of the heart rate signal. The FFT formula is as follows: Where n is the FFT resolution, and Y(k) is the vector sequence obtained by FFT calculation. Modulo operation is performed on Y(k) to obtain M(k) (frequency domain data of the filtered signal). The signal after frequency domain filtering is expressed as M(k) = |Y(k)|.

[0051] Step 4.2, M(k) represents the amount of signal in each given frequency band within the frequency range (Bd). The physiological range of human heart rate is between 40 and 220 beats per minute (bpm). Therefore, Hr L The value is set to 40bpm, Hr H Set to 220 bpm. Then, H and below HR L The heart rate value is removed from the measured frequency domain signal as noise. L and Hγ H The maximum value of M(k) between the two is taken as the heart rate measured by rPPG.

[0052] The above embodiments may be implemented in whole or in part through software, hardware (such as circuits), firmware or any other combination.

[0053] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0054] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0055] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0056] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0057] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0058] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0059] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A non-contact heart rate signal detection method based on rPPG and deep neural network, characterized in that: The steps include: Step 1, obtain color videos of the faces of several target persons and read frames and extract faces from the videos by using OpenCV and a deep neural network (DNN) model; Step 2: Preprocess the video data, use face recognition to extract facial information of multiple subjects separately, then obtain 68 facial feature points to build the facial reference system of each subject, and based on this, cut out the area with the most obvious facial skin color change and the most intense heartbeat signal as the processing region (ROI) as the application object of the subsequent algorithm; Step 3: Since blood absorbs green more than red, the change of green component is approximated as noise. The obtained image data is divided into three channels, RGB, and the values ​​of the RGB channels of each image are subtracted from each other to obtain a waveform after simple smooth filtering, thereby filtering out a large amount of noise generated by the environment and the shooting equipment; Step 4: Use the fast Fourier transform algorithm (FFT) to convert the time domain signal into a frequency domain signal, delete the heart rate values ​​that are not within the normal range of human heart rate of 40 / min to 120 / min as noise from the measured frequency domain signal, and take the highest frequency point after filtering as the heart rate.

2. According to claim 1, a non-contact heart rate signal detection method based on rPPG and deep neural network is characterized in that: Step 1 specifically includes: obtaining color videos of the faces of several target persons and reading frames from the videos by using OpenCV and a deep neural network (DNN) model, preprocessing the video data, and extracting facial information of multiple test subjects separately by using face recognition; specifically includes the following steps: Step 1.1, first define two string variables pb_file_path and pbtxt_file_path, which are used to store the paths of face detection model files and model structure files respectively. These files are deep neural network models trained using TensorFlow. Step 1.

2. Use OpenCV's cv::dnn::readNetFromTensorflow() function to read the deep neural network model from the TensorFlow model file and model structure file and load it into the cv::dnn::Net object. Step 1.

3. Create a VideoCapture object cap to capture video frames from the camera. Step 1.4, enter the video capture loop, read a frame of video from the camera in each iteration, and store it in frame: Step 1.

5. Convert the current frame to the format required for neural network input. Use the cv::dnn::blobFromImage() function to convert the frame into a 4-dimensional blob, set the blob as the input of the neural network, and obtain the output of the model through forward propagation. Step 1.6: Process the output results, extract the detected face information, and display the current frame with the face detection results. Step 1.7, get the values ​​of R(k), G(k), and B(k). Specifically, k represents the number of frames of the image, R(k) is the sum of the ROIs in the red channel, G(k) is the sum of the ROIs in the green channel, and B(k) is the sum of the ROIs in the blue channel.

3. The non-contact heart rate signal detection method based on rPPG and deep neural network according to claim 1, characterized in that: Step 2 is specifically as follows: for the acquired facial information, 68 facial feature points are obtained to construct the facial reference system of each of the tested objects, and based on this, the area with the most obvious facial skin color change and the most intense heartbeat signal is cut out as the processing region (ROI) as the application object of the subsequent algorithm; specifically, it includes the following steps: Step 2.1: Get the heart rate signal x per unit area (n) : Where n is the number of sequence images, x (n) is the heart rate signal obtained, R (i，j) With G (i，j) They are the red and green component values ​​of the i-th row and j-th column in the 10×10 area respectively. Step 2.2, obtain the vector spectrum of the discrete signal. According to the fast Fourier transform formula, the heart rate signal can be obtained and then the signal is fast Fourier transformed to obtain the vector spectrum of the discrete signal: By taking the vector spectrum obtained by equation (2) and applying it to equation (3), we can obtain the wave contents of different frequencies in the current 10×10 region. M (N) =|X (k) |#(3) Step 2.3, obtain the distribution map of the real heart rate wave content, find the signal sensitive area ROI, and according to formula (3), substitute the real heart rate into the formula to obtain the content of the real heart rate in the area, and then you can get the real heart rate wave content in the area. Through formula (4), copy the value to the pixel represented by the area in the distribution map (row m, column n), and after completing the calculation of all areas, you can get the distribution map of the real heart rate wave content. IMG (m,n) =M (HeartRate) #(4) By reflecting the calculated results in the graph, we can get a heart rate distribution graph, where the color represents the content of the wave. It can be clearly seen that the β region is the area where the heart rate waveform is concentrated. This area is used as the ROI and the application object of the subsequent algorithm.

4. The non-contact heart rate signal detection method based on rPPG and deep neural network according to claim 1, characterized in that: Step 3 is as follows: After determining the sensitive area of ​​the palm heart rate signal, superimpose the values ​​of all red channels in the area, and superimpose the values ​​of all green channels in the area, subtract the two values ​​and divide them by the area, and you can get the change in the heart rate signal value per unit area after the red and green channels are smoothed and filtered in the area. Divide the obtained image data into three channels, RGB, and subtract the values ​​of the red channel and the green channel of each image to get a waveform after a simple smoothing filter. It can be considered that this waveform removes some of the noise caused by changes in external and equipment factors. The formula used in this method is: Where n is the number of sequential images, x and y are the horizontal and vertical coordinates of the signal sensitive area, respectively. (n) is the heart rate signal obtained, R (x，y) With G (x，y) They are the red and green component values ​​of the x-th row and y-th column in the x×y area respectively.

5. The non-contact heart rate signal detection method based on rPPG and deep neural network according to claim 1, characterized in that: Step 4 is as follows: perform a fast Fourier transform on the heart rate signal and find the modulus of the result to find the content of waves of different frequencies in the filtered waveform. Since the normal heart rate of a person is about 60 to 100 beats per minute, after filtering out the waves other than 40 / min to 120 / min, the highest point in this frequency range is the heart rate. The specific steps include: Step 4.1: The heart rate obtained from the filtered data x(n) is obtained by subtracting the G value from the R value after filtering the noise. FFT is used to obtain the vector spectrum of the heart rate signal. The FFT formula is as follows: Where n is the FFT resolution, and Y(k) is the vector sequence obtained by FFT calculation. Modulo operation is performed on Y(k) to obtain M(k) (frequency domain data of the filtered signal). The signal after frequency domain filtering is expressed as M(k) = |Y(k)|. Step 4.2, M(k) represents the amount of signal in each given frequency band within the frequency range (Bd). The physiological range of human heart rate is between 40 and 220 beats per minute (bpm). Therefore, Hr L The value is set to 40bpm, Hr H Set to 220 bpm. Then, H and below HR L The heart rate value is removed from the measured frequency domain signal as noise. L and Hr H The maximum value of M(k) between the two is taken as the heart rate measured by rPPG.

Images