Method and system for non-contact accurate measurement of pulse wave and heart rate

Abstract

The invention provides a non-contact accurate pulse wave and heart rate measurement method and system, and the method comprises the following steps: shooting a wrist of a human body by using a high-frame-rate and high-bit-depth camera, and obtaining a raw picture sequence; selecting a region of interest, and obtaining G channel data in the region of interest; calculating a G channel mean value of each frame to obtain a mean value sequence; enabling the mean value sequence to pass through a band-pass filter to obtain pulse waves; the pulse waves pass through the trained multi-layer perceptron, so that more pulse wave details are recovered; the pulse wave main frequency is obtained through Fourier transform, and therefore the heart rate is obtained; the measured pulse wave and heart rate are displayed to a user through a display screen UI interface. According to the method and the system, the accuracy of non-contact measurement of the pulse wave and the heart rate is improved.

Description

Technical Field

[0001] The present invention belongs to the field of non-contact pulse wave and heart rate detection technology, and specifically provides a method and system for non-contact accurate measurement of pulse wave and heart rate. Background Art

[0002] The core of Chinese medicine is to observe, smell, ask and feel. Feeling refers to taking the pulse, which means pressing your fingers on the radial artery to observe the changes in the pulse. If you draw the changes in the pulse, it will form what we call the pulse wave. The pulse wave contains a lot of physiological parameter information, such as heart rate, blood oxygen, blood pressure, etc. Among them, heart rate is an extremely important physiological parameter in the human body.

[0003] Heart rate changes are closely related to heart disease, and heart rate is the most direct indicator of our heart health. The heart is the driving force for pumping blood and the guarantee for the normal functioning of various organ systems and the entire body.

[0004] When an adult's heart rate is over 100 beats / minute at rest, it is called sinus tachycardia. For example, when we are excited, agitated, smoke, drink, or drink coffee, we are likely to have a fast heart rate. There are also some common diseases that can cause the heart rate to increase, such as fever, shock, hyperthyroidism, heart failure, myocardial infarction, etc., or after taking certain drugs. When an adult's heart rate is less than 60 beats / minute at rest, it is called sinus bradycardia. This situation usually occurs in healthy people and athletes who have been engaged in physical labor for a long time. If you do not belong to these occupations and have a bradycardia, you need to consider whether you are taking certain drugs or have certain diseases, such as brain diseases such as cerebral hemorrhage and meningitis, or diseases such as peptic ulcers, intestinal infarction, gallstones, and even malnutrition, infectious diseases, etc., which may cause bradycardia. If the heart rate is less than 40 beats / minute, it may cause insufficient blood oxygen supply, fatigue, dizziness, and difficulty breathing. If it is not handled in time, you may suddenly faint and other emergency situations.

[0005] Therefore, in daily life, it is very important to monitor your heart rate.

[0006] Imaging Photoplethysmography (iPPG) is a non-contact physiological parameter detection technology that has developed rapidly in recent years. It is developed based on the traditional single-point contact photoplethysmography. The basic principle of iPPG is to record the changes in light intensity on the skin surface through a photoelectric sensor (usually a camera). These changes are related to the pulsation of blood in the blood vessels. When the heart beats, the amount of blood changes in the blood vessels, which affects the absorption and scattering of light irradiated by the skin. By analyzing these light intensity changes in the image, the signals related to the heart beat can be extracted to obtain information such as pulse waveform and heart rate. iPPG does not require contact with the skin and is suitable for non-contact physiological parameter detection and long-distance physiological parameter monitoring.

[0007] Currently in the field of iPPG research, the existing methods still have a lot of room for improvement in the accuracy of pulse wave and heart rate measurement. Therefore, the present invention proposes a method and system for non-contact accurate measurement of pulse wave and heart rate to improve the accuracy of non-contact pulse wave and heart rate measurement. Summary of the invention

[0008] The main purpose of the present invention is to provide a method and system for non-contact accurate measurement of pulse wave and heart rate, so as to improve the accuracy of non-contact pulse wave and heart rate measurement.

[0009] The technical solution adopted by the present invention is as follows: a non-contact pulse wave and heart rate detection method combining sliding window filtering with a multi-head attention mechanism, the detection method comprising the following steps:

[0010] S1: Use a high frame rate and high bit depth camera to shoot the human wrist and obtain a raw image sequence;

[0011] S2: Select the region of interest and obtain the G channel data in the region of interest;

[0012] S3: Calculate the mean of the G channel of each frame to obtain a mean sequence; pass the mean sequence through a bandpass filter to obtain a pulse wave;

[0013] S4: Pass the pulse wave through the trained multi-layer perceptron to restore more pulse wave details;

[0014] S5: Use Fourier transform to obtain the main frequency of the pulse wave, thereby obtaining the heart rate;

[0015] S6: Display the measured pulse wave and heart rate to the user through a display UI interface.

[0016] A further improvement of the present invention is that a camera with a high frame rate and a high bit depth is used as a shooting device in S1. The high frame rate characteristic enables the camera to better capture the temporal details of the pulse wave, and the high bit depth enables the camera to better capture the amplitude details of the pulse wave.

[0017] A further improvement of the present invention is that in S2 and S3, according to the Bayer pattern (RGGB, BGGR, GRBG or GBRG) of the captured raw image, that is, the arrangement of the color filter matrix on the digital image sensor, the G channel value of the region of interest is taken out, and the G channel mean in the region of interest of each frame is calculated, thereby obtaining a mean sequence, which is the most original unprocessed mean pixel change.

[0018] A further improvement of the present invention is that in S3, a digital Butterworth bandpass filter between 0.5 Hz and 5 Hz is used to remove other noises in the sequence. The formula of the digital Butterworth bandpass filter is as follows:

[0019] y[n]=b0x[n]+b1x[n-1]+b2x[n-2]-a1y[n-1]-a2y[n-2]

[0020] Among them, y[n] is the filtered signal, x[n] is the input signal, and b and a are the filter coefficients.

[0021] A further improvement of the present invention is that in S4, a trained multi-layer perceptron (MLP) is used to restore more pulse wave details. The formula of the MLP is as follows:

[0022] a (l) =σ(W (l) x (l-1) +b (l) )

[0023] Among them, W (l) is the weight matrix of layer l, x (l-1) is the output of the l-1th layer, which is also the input of the lth layer, b (l) is the bias vector of the lth layer, and σ(·) is the activation function.

[0024] A further improvement of the present invention is that the MLP in S4 has a total of 7 layers, and the number of nodes thereof are 1000, 3000, 5000, 7000, 5000, 3000, and 1000 respectively.

[0025] A further improvement of the present invention is that in S5, Fourier transform is used to obtain the main frequency of the pulse wave, thereby calculating the heart rate.

[0026] A further improvement of the present invention is that pyqt is used in S6 to draw a UI interface, thereby facilitating the user to perform non-contact pulse wave and heart rate measurement.

[0027] In summary, due to the adoption of the above technical scheme, the beneficial effects of the present invention are: using a high frame rate and high bit depth camera to shoot a human wrist to obtain a raw picture sequence; selecting an area of ​​interest and obtaining the G channel data in the area of ​​interest; calculating the G channel mean of each frame to obtain a mean sequence; passing the mean sequence through a bandpass filter to obtain a pulse wave; passing the pulse wave through a trained multi-layer perceptron to restore more pulse wave details; using Fourier transform to obtain the main frequency of the pulse wave to obtain the heart rate; and displaying the measured pulse wave and heart rate to the user through a display screen UI interface. The method and system of the present invention improve the accuracy of non-contact measurement of pulse waves and heart rates. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a physical diagram of the system.

[0029] Figure 2 The figure is a flow chart of the method of the present invention.

[0030] Figure 3 It is a user-oriented software algorithm flow chart.

[0031] Figure 4 For the UI interface. Specific implementation methods

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail in combination with the following specific embodiments and with reference to the accompanying drawings, but it is not intended to limit the scope of the present invention.

[0034] In this embodiment, the non-contact accurate measurement system of pulse wave and heart rate of the present invention is as follows: Figure 1 As shown, the user places his wrist squarely below the camera and simultaneously controls the software of the system for non-contact accurate measurement of pulse wave and heart rate of the present invention to complete the non-contact measurement of pulse wave and heart rate.

[0035] like Figure 2 As shown, the method and system for non-contact accurate measurement of pulse wave and heart rate of the present invention mainly include the following steps:

[0036] Step 1: Use a high frame rate and high bit depth camera to shoot the human wrist and obtain a raw image sequence.

[0037] The high frame rate feature enables the camera to better capture the temporal details of the pulse wave, and the high bit depth enables the camera to better capture the amplitude details of the pulse wave. The more details captured, the more accurate the measured pulse wave and the more accurate the calculated heart rate. The camera frame rate used in this embodiment is 100 frames / second and the bit depth is 12 bits.

[0038] Step 2: Select the region of interest and obtain the G channel data in the region of interest;

[0039] According to the Bayer pattern (RGGB, BGGR, GRBG or GBRG) of the raw image taken, that is, the arrangement of the color filter matrix on the digital image sensor, the G channel value of the region of interest is extracted. The raw image Bayer pattern used in this embodiment is GBRG, and the pseudo code for extracting the G channel value from it is as follows

[0040]

[0041]

[0042] Step 3, calculate the mean of the G channel of each frame to obtain a mean sequence; pass the mean sequence through a bandpass filter to obtain a pulse wave;

[0043] Calculate the mean of the G channel in the region of interest of each frame to obtain the mean sequence, which is the original unprocessed mean pixel change. Use a digital Butterworth bandpass filter between 0.5Hz and 5Hz to remove other noise in the sequence. The formula of the digital Butterworth bandpass filter is as follows:

[0044] y[n]=b0x[n]+b1x[n-1]+b2x[n-2]-a1y[n-1]-a2y[n-2]

[0045] Among them, y[n] is the filtered signal, x[n] is the input signal, and b and a are the filter coefficients.

[0046] Step 4: Pass the pulse wave through the trained multi-layer perceptron to restore more pulse wave details;

[0047] A dataset of 30 people was collected. The data collected for each volunteer included: a 30-second video, a 30-second PPG sensor data, and heart rate. The frame rate of the collected video was 100, and the sampling rate of the PPG sensor was 60 Hz. In order to keep the data aligned, the time series of the collected PPG sensor was interpolated to make its sampling rate 100 Hz.

[0048] The data set was further processed to divide each volunteer's 30-second video and 30-second PPG sensor time series into three equal parts, namely, three 10-second videos and three groups of 10-second PPG sensor time series, that is, three data pieces. Therefore, the resulting data set contains 90 data pieces, each of which contains one 10-second video, one group of 10-second PPG sensor sequence and the corresponding heart rate.

[0049] The data set is divided into a training set, a validation set, and a test set. The training set has 50 data, the validation set has 20 data, and the test set has 20 data. The data set is used to fully train the built multi-layer perceptron to enable it to acquire the ability to restore pulse wave details.

[0050] The entire training process uses Adam as the optimizer, uses mean square error as the loss function, and the initial learning rate is 0.001. The learning rate scheduler and early stopping method are used to optimize the training process. When the validation set loss does not decrease for 6 consecutive rounds, the learning rate automatically decreases to one tenth of the original. When the validation set loss does not decrease for 10 consecutive rounds, the training is terminated.

[0051] The training set loss function converged to 11.56, and the test set loss function converged to 14.71.

[0052] Step 5: Use Fourier transform to obtain the main frequency of the pulse wave, thereby obtaining the heart rate;

[0053] Convert the time domain signal x[n] to the frequency domain signal X[k] through Fast Fourier Transform (FFT)

[0054]

[0055] Calculate the magnitude spectrum of the spectrum |X[k]|

[0056]

[0057] Identify the maximum position k in the magnitude spectrum peak

[0058]

[0059] The index k peak Convert to actual frequency f peak

[0060]

[0061] Calculate heart rate HR

[0062] HR = f peak ×60

[0063] Step 6: Display the measured pulse wave and heart rate to the user through the display UI interface.

[0064] Use pyqt to draw the UI interface, so that users can easily measure the non-contact pulse wave and heart rate. The algorithm flow of the UI interface is as follows Figure 3 As shown, the UI interface is as follows Figure 4 Shown

[0065] In summary, the present invention uses a high-frame-rate and high-bit-depth camera to shoot a human wrist to obtain a raw picture sequence; selects an area of ​​interest and obtains the G channel data in the area of ​​interest; calculates the G channel mean of each frame to obtain a mean sequence; passes the mean sequence through a bandpass filter to obtain a pulse wave; passes the pulse wave through a trained multi-layer perceptron to restore more pulse wave details; uses Fourier transform to obtain the main frequency of the pulse wave to obtain the heart rate; and displays the measured pulse wave and heart rate to the user through a display screen UI interface. The method and system of the present invention improve the accuracy of non-contact measurement of pulse waves and heart rates.

[0066] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method and system for non-contact accurate measurement of pulse wave and heart rate, characterized in that: The following steps are involved: S1: Use a high frame rate and high bit depth camera to shoot the human wrist and obtain a raw image sequence; S2: Select the region of interest and obtain the G channel data in the region of interest; S3: Calculate the mean of the G channel of each frame to obtain a mean sequence; pass the mean sequence through a bandpass filter to obtain a pulse wave; S4: Pass the pulse wave through the trained multi-layer perceptron to restore more pulse wave details; S5: Use Fourier transform to obtain the main frequency of the pulse wave, thereby obtaining the heart rate; S6: Display the measured pulse wave and heart rate to the user through a display UI interface.

2. A method and system for non-contact accurate measurement of pulse wave and heart rate as claimed in claim 1, characterized in that: In S1, a camera with a high frame rate and a high bit depth is used as a shooting device. The high frame rate characteristic enables the camera to better capture the temporal details of the pulse wave, and the high bit depth enables the camera to better capture the amplitude details of the pulse wave.

3. A method and system for non-contact accurate measurement of pulse wave and heart rate as claimed in claim 1, characterized in that: In S2 and S3, according to the Bayer pattern (RGGB, BGGR, GRBG or GBRG) of the captured raw image, that is, the arrangement of the color filter matrix on the digital image sensor, the G channel value of the region of interest is taken out, and the G channel mean in the region of interest of each frame is calculated, so as to obtain the mean sequence, which is the most original unprocessed mean pixel change.

4. A method and system for non-contact accurate measurement of pulse wave and heart rate as claimed in claim 1, characterized in that: In S3, a digital Butterworth bandpass filter between 0.5 Hz and 5 Hz is used to remove other noises in the sequence. The formula of the digital Butterworth bandpass filter is as follows: y[n]=b0x[n]+b1x[n-1]+b2x[n-2]-a1y[n-1]-a2y[n-2] Among them, y[n] is the filtered signal, x[n] is the input signal, and b and a are the filter coefficients.

5. A method and system for non-contact accurate measurement of pulse wave and heart rate as claimed in claim 1, characterized in that: In S4, a trained multi-layer perceptron (MLP) is used to restore more pulse wave details. The formula of the MLP is as follows: a (l) =σ(W (l) x (l-1) +b (l) ) Among them, W (l) is the weight matrix of layer l, x (l-1) is the output of the l-1th layer, which is also the input of the lth layer, b (l) is the bias vector of the lth layer, and σ(·) is the activation function.

6. A method and system for non-contact accurate measurement of pulse wave and heart rate as claimed in claim 1, characterized in that: The MLP in S4 has a total of 7 layers, and the number of nodes are 1000, 3000, 5000, 7000, 5000, 3000, and 1000 respectively.

7. A method and system for non-contact accurate measurement of pulse wave and heart rate as claimed in claim 1, characterized in that: In S5, Fourier transform is used to obtain the main frequency of the pulse wave, thereby calculating the heart rate.

8. A method and system for non-contact accurate measurement of pulse wave and heart rate as claimed in claim 1, characterized in that: The S6 uses pyqt to draw the UI interface, so as to facilitate users to measure the non-contact pulse wave and heart rate.

Images