Non-contact pulse wave and heart rate detection method combining sliding window filtering and multi-head attention mechanism

Abstract

The invention discloses a non-contact pulse wave and heart rate detection method combining sliding window filtering and a multi-head attention mechanism, and the method comprises the following steps: selecting a radial artery region from a human wrist video as a region of interest, obtaining a mean time sequence of the region of interest in the video, and taking the sequence as an initial pulse wave signal; the signal is preprocessed through three times of sliding window filtering to achieve the effects of detrending and denoising; dividing the preprocessed sequence signal into a training set, a verification set and a test set according to a corresponding proportion; constructing a trainable deep learning network model combining a multi-head attention mechanism and a multi-layer perceptron; sending the divided training set into a deep learning neural network for training, wherein the output of the model is more accurate pulse waves with more details recovered; the pulse waves are subjected to Fourier transform, the main frequency of the pulse waves is obtained, and therefore the heart rate is obtained; and connecting a preprocessing algorithm, the trained deep learning model and a heart rate calculation algorithm to obtain an end-to-end non-contact pulse wave and heart rate detection network. According to the method, the accuracy of non-contact measurement of the pulse wave and the heart rate is improved.

Description

Technical Field

[0001] The present invention belongs to the field of non-contact pulse wave and heart rate detection technology, and specifically provides a non-contact pulse wave and heart rate detection method combining sliding window filtering with a multi-head attention mechanism Background Art

[0002] The core of Chinese medicine is to observe, smell, ask and feel. Feeling refers to taking the pulse, which means pressing your fingers on the radial artery to observe the changes in the pulse. If you draw the changes in the pulse, it will form what we call the pulse wave. The pulse wave contains a lot of physiological parameter information, such as heart rate, blood oxygen, blood pressure, etc. Among them, heart rate is an extremely important physiological parameter in the human body.

[0003] Heart rate changes are closely related to heart disease, and heart rate is the most direct indicator of our heart health. The heart is the driving force for pumping blood and the guarantee for the normal functioning of various organ systems and the entire body.

[0004] When an adult's heart rate is over 100 beats / minute at rest, it is called sinus tachycardia. For example, when we are excited, agitated, smoke, drink, or drink coffee, we are likely to have a fast heart rate. There are also some common diseases that can cause the heart rate to increase, such as fever, shock, hyperthyroidism, heart failure, myocardial infarction, etc., or after taking certain drugs. When an adult's heart rate is less than 60 beats / minute at rest, it is called sinus bradycardia. This situation usually occurs in healthy people and athletes who have been engaged in physical labor for a long time. If you do not belong to these occupations and have a bradycardia, you need to consider whether you are taking certain drugs or have certain diseases, such as brain diseases such as cerebral hemorrhage and meningitis, or diseases such as peptic ulcers, intestinal infarction, gallstones, and even malnutrition, infectious diseases, etc., which may cause bradycardia. If the heart rate is less than 40 beats / minute, it may cause insufficient blood oxygen supply, fatigue, dizziness, and difficulty breathing. If it is not handled in time, you may suddenly faint and other emergency situations.

[0005] Therefore, in daily life, it is very important to monitor your heart rate.

[0006] Imaging Photoplethysmography (iPPG) is a non-contact physiological parameter detection technology that has developed rapidly in recent years. It is developed based on the traditional single-point contact photoplethysmography. The basic principle of iPPG is to record the changes in light intensity on the skin surface through a photoelectric sensor (usually a camera). These changes are related to the pulsation of blood in the blood vessels. When the heart beats, the amount of blood changes in the blood vessels, which affects the absorption and scattering of light irradiated by the skin. By analyzing these light intensity changes in the image, the signals related to the heart beat can be extracted to obtain information such as pulse waveform and heart rate. iPPG does not require contact with the skin and is suitable for non-contact physiological parameter detection and long-distance physiological parameter monitoring.

[0007] Currently in the field of iPPG research, the existing methods still have a lot of room for improvement in the accuracy of pulse wave and heart rate measurement. Therefore, the present invention proposes a non-contact pulse wave and heart rate detection method that combines sliding window filtering with a multi-head attention mechanism to improve the accuracy of non-contact pulse wave and heart rate measurement. Summary of the invention

[0008] The main purpose of the present invention is to propose a non-contact pulse wave and heart rate detection method which combines sliding window filtering with a multi-head attention mechanism to improve the accuracy of non-contact pulse wave and heart rate measurement.

[0009] The technical solution adopted by the present invention is as follows: a non-contact pulse wave and heart rate detection method combining sliding window filtering with a multi-head attention mechanism, the detection method comprising the following steps:

[0010] S1: Select the region of interest in the wrist video, the region of interest is the radial artery of the human wrist;

[0011] S2: taking the mean sequence of the region of interest as the initial pulse wave, using sliding window filtering to remove the trend and denoise the initial pulse wave to obtain the preprocessed pulse wave;

[0012] S3: Use the multi-head attention mechanism to extract the features of the pulse wave, and map the features through a multi-layer perceptron to obtain a more accurate pulse wave;

[0013] S4: Use Fourier transform to obtain the main frequency of the pulse wave and thus obtain the heart rate.

[0014] S5: Connect the preprocessing algorithm, the trained deep learning model and the heart rate calculation algorithm to obtain an end-to-end contactless pulse wave and heart rate detection network.

[0015] A further improvement of the present invention is to combine traditional sliding window filtering, Fourier transform and deep learning methods to form an end-to-end non-contact pulse wave and heart rate accurate measurement network.

[0016] A further improvement of the present invention is that in step S2, a sliding window with a length of 100 is first used to extract the trend in the sequence and remove the trend, and then a sliding window with a length of 50 is used to remove the high-frequency noise in the sequence, and then a sliding window with a length of 3 is used to further remove the high-frequency noise in the sequence.

[0017] The formula for sliding window filtering is as follows:

[0018]

[0019] Among them, x[n] is the original data, y[n] is the output of sliding window filtering, M is the window size, and n is the current sample position.

[0020] A further improvement of the present invention is that the embedding dimension of the multi-head attention mechanism in step S3 is 500, the number of layers of the multi-head attention mechanism is 2, and the number of heads is 4. The attention weight calculation formula is as follows:

[0021]

[0022] Where Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the dimension of the key.

[0023] The calculation formula for each head is as follows:

[0024] head i =Attention(QW i Q , KW i K , VW i V )

[0025] Where W i Q , W i Q , W i V is the weight matrix used for the linear transformation.

[0026] The calculation formula of multi-head attention is as follows:

[0027] MultiHead(Q,K,V)=Concat(head1,...,head h )W o

[0028] Where h is the number of heads, W ois the output weight matrix.

[0029] A further improvement of the present invention is that in step S3, the multilayer perceptron has a total of 5 layers, and the number of nodes is 1000, 2000, 3000, 2000, and 1000 respectively. The activation function used is the hyperbolic tangent function (tanh). The calculation formula between layers is as follows:

[0030] y=tanh(Wx+b)

[0031] Where y is the layer output, x is the layer input, W is the weight matrix, and b is the bias vector.

[0032] A further improvement of the present invention is that Adam is used as the optimizer in the entire training process, the initial learning rate is 0.001, and the learning rate scheduler and early stopping method are used to optimize the training process. When the validation set loss has not decreased for 6 consecutive rounds, the learning rate is automatically reduced to one tenth of the original value. When the validation set loss has not decreased for 10 consecutive rounds, the training is terminated.

[0033] A further improvement of the present invention is that in step S4, Fourier transform is used to calculate the heart rate, and the formula is as follows:

[0034]

[0035] Where x[n] is the pulse wave, f s is the frame rate of the video, and N is the pulse wave length.

[0036] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: firstly, the pulse wave contained in the wrist video is preprocessed using sliding window filtering, and then the extracted pulse wave is feature extracted using a multi-head attention mechanism, and the extracted features are passed to a multi-layer perceptron, which maps the features and restores the pulse wave details lost in the preprocessing, thereby obtaining a more accurate pulse wave. The main frequency in the pulse wave is extracted using Fourier transform to obtain the heart rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is the algorithm flow chart of the present invention.

[0038] Specific implementation method In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail in combination with the following specific embodiments and with reference to the accompanying drawings, but it is not intended to limit the scope of the present invention.

[0039] like Figure 1 As shown, the non-contact pulse wave and heart rate detection method combining sliding window filtering with a multi-head attention mechanism of the present invention mainly includes the following steps:

[0040] Step 1: Collect a data set of 30 people. The data collected for each volunteer include: 30 seconds of video, 30 seconds of PPG sensor data and heart rate. The frame rate of the collected video is 100, and the sampling rate of the PPG sensor is 60hz. In order to keep the data aligned, the time series of the collected PPG sensor is interpolated to make its sampling rate 100hz.

[0041] Step 2: Select the region of interest in the wrist video. The region of interest is the radial artery of the human wrist. Calculate the mean of the region of interest in each frame of the video. The obtained mean sequence is the initial pulse wave, which contains a lot of noise in addition to the pulse wave. Use sliding window filtering to remove the trend and noise of the initial pulse wave. The formula is as follows:

[0042]

[0043] Among them, x[n] is the original data, y[n] is the output of sliding window filtering, M is the window size, and n is the current sample position.

[0044] First, a sliding window with a length of 100 is used to extract the trend in the sequence and remove the trend. Then, a sliding window with a length of 50 is used to remove the high-frequency noise in the sequence. Then, a sliding window with a length of 3 is used to further remove the high-frequency noise in the sequence.

[0045] The data set was further processed to divide each volunteer's 30-second video and 30-second PPG sensor time series into three equal parts, namely, three 10-second videos and three groups of 10-second PPG sensor time series, that is, three data pieces. Therefore, the resulting data set contains 90 data pieces, each of which contains one 10-second video, one group of 10-second PPG sensor sequence and the corresponding heart rate.

[0046] The data set is divided into training set, validation set and test set. The training set has 50 data, the validation set has 20 data, and the test set has 20 data.

[0047] Step 3: Use the multi-head attention mechanism to extract the features of the pulse wave, and map the features through a multi-layer perceptron to obtain a more accurate pulse wave; the embedding dimension of the multi-head attention mechanism is 500, the number of layers of the multi-head attention mechanism is 2, and the number of heads is 4. The attention weight calculation formula is as follows:

[0048]

[0049] Where Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the dimension of the key.

[0050] The calculation formula for each head is as follows:

[0051] headi =Attention(QW i Q ,KWK i R ,VW i V )

[0052] Where W i Q , W i Q , W i V is the weight matrix used for the linear transformation.

[0053] The calculation formula of multi-head attention is as follows:

[0054] MultiHead(Q,K,V)=Concat(head1,…,head h )W O

[0055] Where h is the number of heads, W O is the output weight matrix.

[0056] The multilayer perceptron has 5 layers in total, with the number of nodes being 1000, 2000, 3000, 2000, and 1000 respectively. The activation function used is the hyperbolic tangent function (tanh). The calculation formula between layers is as follows:

[0057] y=tanh(Wx+b)

[0058] Where y is the layer output, x is the layer input, W is the weight matrix, and b is the bias vector.

[0059] The entire training process uses Adam as the optimizer, uses mean square error as the loss function, and the initial learning rate is 0.001. The learning rate scheduler and early stopping method are used to optimize the training process. When the validation set loss does not decrease for 6 consecutive rounds, the learning rate automatically decreases to one tenth of the original. When the validation set loss does not decrease for 10 consecutive rounds, the training is terminated.

[0060] The training set loss function converged to 11.56, and the test set loss function converged to 14.71.

[0061] Step 4: Use Fourier transform to obtain the main frequency of the pulse wave and thus obtain the heart rate.

[0062] Convert the time domain signal x[n] to the frequency domain signal X[k] through Fast Fourier Transform (FFT)

[0063]

[0064] Calculate the magnitude spectrum of the spectrum |X[k]|

[0065]

[0066] Identify the maximum position k in the magnitude spectrum peak

[0067]

[0068] The index k peak Convert to actual frequency f peak

[0069]

[0070] Calculate heart rate HR

[0071] HR = f peak ×60

[0072] Step 5: Connect the preprocessing algorithm, the trained deep learning model and the heart rate calculation algorithm to obtain an end-to-end contactless pulse wave and heart rate detection network.

[0073] In summary, the present invention is a non-contact pulse wave and heart rate detection method combining sliding window filtering with multi-head attention mechanism. The method selects the radial artery area as the region of interest from the human wrist video, obtains the mean time series of the region of interest in the video, and uses the sequence as the initial pulse wave signal. The signal is pre-processed by sliding window filtering three times to achieve the effect of detrending and denoising; the pre-processed sequence signal is divided into a training set, a validation set and a test set according to the corresponding proportion; a trainable deep learning network model combining the multi-head attention mechanism and the multi-layer perceptron is constructed; the divided training set is sent to the deep learning neural network for training, and the output of the model is a more accurate pulse wave with more details restored; the pulse wave is Fourier transformed to obtain its main frequency, thereby obtaining the heart rate; the pre-processing algorithm, the trained deep learning model and the heart rate calculation algorithm are connected to obtain an end-to-end non-contact pulse wave and heart rate detection network. The method of the present invention improves the accuracy of non-contact measurement of pulse wave and heart rate.

[0074] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A non-contact pulse wave and heart rate detection method combining sliding window filtering with a multi-head attention mechanism, characterized in that: The detection method comprises the following steps: S1: Select the region of interest in the wrist video, the region of interest is the radial artery of the human wrist; S2: taking the mean sequence of the region of interest as the initial pulse wave, using sliding window filtering to remove the trend and denoise the initial pulse wave to obtain the preprocessed pulse wave; S3: Use the multi-head attention mechanism to extract the features of the pulse wave, and map the features through a multi-layer perceptron to obtain a more accurate pulse wave; S4: Use Fourier transform to obtain the main frequency of the pulse wave and thus obtain the heart rate. S5: Connect the preprocessing algorithm, the trained deep learning model and the heart rate calculation algorithm to obtain an end-to-end contactless pulse wave and heart rate detection network.

2. A non-contact pulse wave and heart rate detection method combining sliding window filtering with a multi-head attention mechanism as claimed in claim 1, characterized in that: The traditional sliding window filtering, Fourier transform and deep learning methods are combined to form an end-to-end non-contact pulse wave and heart rate accurate measurement network.

3. A non-contact pulse wave and heart rate detection method combining sliding window filtering with a multi-head attention mechanism as claimed in claim 1, characterized in that: In step S2, a sliding window with a length of 100 is first used to extract the trend in the sequence and remove the trend, and then a sliding window with a length of 50 is used to remove the high-frequency noise in the sequence, and then a sliding window with a length of 3 is used to further remove the high-frequency noise in the sequence. The formula of sliding window filtering is as follows: Among them, x[n] is the original data, y[n] is the output of sliding window filtering, M is the window size, and n is the current sample position.

4. A non-contact pulse wave and heart rate detection method combining sliding window filtering with a multi-head attention mechanism as claimed in claim 1, characterized in that: The embedding dimension of the multi-head attention mechanism in step S3 is 500, the number of layers of the multi-head attention mechanism is 2, and the number of heads is 4. The attention weight calculation formula is as follows: Where Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the dimension of the key. The calculation formula for each head is as follows: head i =Attention(QW i Q ,KW i K ,VW i V ) Where W i Q , W i Q , W i V is the weight matrix used for the linear transformation. The calculation formula of multi-head attention is as follows: MultiHead(Q,K,V)=Concat(head1,…,head h )W O Where h is the number of heads, W O is the output weight matrix.

5. A non-contact pulse wave and heart rate detection method combining sliding window filtering with a multi-head attention mechanism as claimed in claim 1, characterized in that: In step S3, the multilayer perceptron has a total of 5 layers, and the number of nodes is 1000, 2000, 3000, 2000, and 1000 respectively. The activation function used is the hyperbolic tangent function (tanh). The calculation formula between layers is as follows: y=tanh(Wx+b) Where y is the layer output, x is the layer input, W is the weight matrix, and b is the bias vector.

6. A non-contact pulse wave and heart rate detection method combining sliding window filtering with a multi-head attention mechanism as claimed in claim 1, characterized in that: The entire training process uses Adam as the optimizer, with an initial learning rate of 0.

001. The learning rate scheduler and early stopping method are used to optimize the training process. When the validation set loss does not decrease for 6 consecutive rounds, the learning rate automatically decreases to one tenth of the original value. When the validation set loss does not decrease for 10 consecutive rounds, the training is terminated.

7. A non-contact pulse wave and heart rate detection method combining sliding window filtering with a multi-head attention mechanism as claimed in claim 1, characterized in that: In step S4, the heart rate is calculated using Fourier transform, and the formula is as follows: Where x[n] is the pulse wave, f s is the frame rate of the video, and N is the pulse wave length.

Images