Remote heart rate detection method and system based on heartbeat probability

Abstract

The invention discloses a remote heart rate detection method and system based on the heartbeat probability, and belongs to the technical field of heart rate detection.The remote heart rate detection method and system based on the heartbeat probability comprise the following specific steps that firstly, a video of a person to be detected is collected through a camera, and PPG signals are collected through an instrument; step 2, preprocessing the video data, specifically, dividing a video into video frames with the length being equal to the specification, representing each frame in an RGB format, and recording a video sequence as the color channel, the width of the frame and the height of the frame; according to the method, the heart rate is detected by predicting the heartbeat occurrence probability, the signal processing process is simplified, and the stability under the condition of large signal fluctuation is effectively improved. The HBP-Net introduces a signal enhancement mechanism in model design to amplify the BVP signal, so that the adaptability of the model under different illumination conditions is improved, and the face BVP signal can be more clearly visualized.

Description

Technical Field

[0001] The present invention belongs to the technical field of heart rate detection, and in particular relates to a remote heart rate detection method and system based on heartbeat probability. Background Art

[0002] Heart rate (HR) is an important physiological indicator for assessing cardiovascular health, emotional state and physical load. Heart rate detection has received continuous attention due to its wide application in medical health, sports management and emotional detection. Traditional heart rate detection methods include photoplethysmography (PPG) and electrocardiography (ECG). PPG measures the changes in light reflection or transmission on the skin surface to detect the pulse signal of blood flow, and its equipment is commonly found in smart wearable devices. ECG is based on detecting the potential changes generated by myocardial electrical activity and is a widely used heart rate monitoring method in traditional medicine. However, these methods have limitations in practical applications. First, these methods mostly rely on contact sensors, which require direct contact between the device and the skin, causing discomfort when worn for a long time, and the signal is easily interfered during exercise. In addition, the fragility of the device and user compliance are also reasons that limit its widespread application. To solve these problems, academia and industry have gradually turned to exploring non-contact heart rate monitoring technology.

[0003] In recent years, remote photoplethysmography (rPPG) has gradually attracted attention as a non-contact heart rate monitoring technology. rPPG uses a camera to capture the tiny fluctuations of skin surface color as the heartbeat changes, thereby extracting the blood volume pulse (BVP) signal to achieve remote heart rate estimation. Compared with contact methods, rPPG has the advantages of non-invasiveness, convenience, and adaptability to a variety of scenarios. It is especially suitable for intelligent monitoring, remote health management, and physiological monitoring in special environments.

[0004] Traditional rPPG methods mainly rely on signal processing algorithms to estimate heart rate by extracting and filtering RGB signals. These methods are usually based on feature extraction and time-domain and frequency-domain analysis, and have poor robustness to noise, especially under the interference of lighting changes or individual movement. In order to improve the robustness of rPPG, researchers have gradually adopted deep learning methods to automatically learn features related to heart rate from video data. In recent years, methods such as Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) have been introduced into rPPG, which can more accurately extract heart rate under unstable lighting and complex backgrounds. In addition, the application of spatiotemporal convolutional networks in this field is also increasing. Through three-dimensional convolution, spatiotemporal information is effectively captured, which improves the accuracy of heart rate estimation.

[0005] At present, many studies have extracted BVP signals from rPPG videos to estimate heart rate. However, from the perspective of calculating heart rate, it is not necessary to directly extract BVP signals. Directly predicting the probability of heartbeat points can more effectively obtain heart rate or heart rhythm variability, and is more robust to noise interference. Especially in motion scenes, current rPPG methods are susceptible to artifacts introduced by head and facial movements, resulting in reduced accuracy of heart rate estimation. Therefore, how to stably extract heart rate signals in motion scenes is still a key challenge in the rPPG field. Summary of the invention

[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a remote heart rate detection method and system based on heartbeat probability.

[0007] The technical solution adopted to solve the above technical problem is: a remote heart rate detection method based on heartbeat probability, including the following specific steps:

[0008] Step 1: Capture the video of the person to be tested through the camera, and collect the PPG signal through the instrument;

[0009] Step 2: Preprocess the video data. Specifically, divide the video into video frames of length T. Each frame is represented in RGB format. The video sequence is recorded as X=(C, T, W, H), where C is the color channel, W is the width of the frame, and H is the height of the frame.

[0010] Step 3: Take the image at T=0, use MTCNN to perform face detection, crop the facial area of ​​each frame in the video and normalize it;

[0011] Step 4: Linearly interpolate the PPG sequence and perform mean normalization to obtain a PPG sequence of length T, and then perform peak detection (considering that PPG measurement may be disturbed by movement, if the heart rate detected by the peak value is significantly different from the heart rate obtained by the PSD method, this feature will be excluded). Let the BVP signal peak index sequence be P, D max =MAX(Diff(P)) is the maximum value of the peak interval, PB(x), PF(x) are the previous peak index and the next peak index of x respectively, calculate HBP;

[0012] Step 5: Consider each frame of the RGB video as a sampling, and then guess whether a certain frame is a heartbeat point. These images constitute a random process of whether it is a heartbeat point. The probability of a certain time point in this random process is related to the previous and next time points. When multiple values ​​of the signal before and after a certain time point are known, the probability of guessing the signal value at this time point will be greater, and the HBP(t) probability distribution is fitted;

[0013] Step 6: The HBP before and after associations obtained through spatiotemporal convolution learning are used to construct a hybrid attention model (HBP-Net network structure) using image full convolution and skip connections to obtain the HBP-Net model;

[0014] Step 7: Remote heart rate monitoring is performed by inputting the video into the HBP-Net model to recognize the facial BVP signal.

[0015] Furthermore, the HBP calculation formula in step 4 is as follows:

[0016]

[0017] The input feature of the HBP-Net network structure in step 6 is X∈R B×C×T×H×W , whose dimensions are batch, channel, time, width, and height respectively.

[0018] Furthermore, the spatiotemporal channel attention module in step 6 can calculate the weights through the following steps:

[0019] Pooling to get global information in time

[0020] X pool =Pool(X)∈R B×C×T×H×W

[0021] The number of channels is compressed through convolution or fully connected layers

[0022] W time =σ(Conv(X pool ))∈R B×C×T×1×1

[0023] Here σ is the Sigmoid activation function, and the generated W time is the attention weight at each time step, and W time Applied to the original input X

[0024] X out =X·W time

[0025] By weighting each frame, more important features in the time dimension are finally obtained.

[0026] Furthermore, the fusion loss is calculated by BCE, and BCE is adjusted to make the minimum loss value 0. The calculation method of Loss is as follows:

[0027]

[0028] Where y is the true probability distribution, is the predicted probability distribution, T is the number of samples, and the latter term of the subtraction does not participate in the derivation and back propagation if and only if When the same, Loss=0, Loss=0,when When , Loss→∞, and 10 is taken as the maximum value during the calculation process.

[0029] Furthermore, it includes a signal acquisition module, a data transmission module, a back-end processing center and a user terminal. The signal acquisition module is responsible for collecting data of the person to be detected through a camera sensor; the data transmission module is responsible for transmitting the collected signal data that has been preliminarily processed to the back-end processing device in real time.

[0030] Furthermore, the back-end processing center is responsible for receiving data from various acquisition terminals and storing them in the database, identifying the data in the video through the HBP-Net model to obtain the BVP.

[0031] Furthermore, the user terminal includes mobile applications and medical workstations. Mobile applications: For ordinary users, mobile applications adapted to iOS and Android systems are developed. Users can view their heart rate data and heart rate trend charts in real time on their mobile phones, and set personalized heart rate warning thresholds. When the heart rate rises abnormally or drops beyond the set range, the mobile phone can push notifications in time to remind users to pay attention to their health conditions. It is also convenient for users to share data.

[0032] The medical workstation is designed for professional medical personnel. Doctors access server data through computer workstations and view patients' detailed heart rate monitoring reports, which assists doctors in disease diagnosis, especially for remote tracking treatment of patients in the recovery period of arrhythmia and cardiovascular diseases.

[0033] The beneficial effects of the present invention are as follows: the present invention detects heart rate by predicting the probability of heartbeat occurrence, which not only simplifies the signal processing process, but also effectively improves the stability under the condition of large signal fluctuations. HBP-Net introduces a signal enhancement mechanism in model design to amplify the BVP signal, thereby improving the adaptability of the model under different lighting conditions and being able to more clearly visualize the facial BVP signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a process flow chart of the present invention from video input to generating heart rate estimation and probability distribution;

[0035] Figure 2 It is the HBP-Net network structure diagram of the present invention;

[0036] Figure 3 It is a 3D trend diagram of the loss size of the present invention along with the actual value and the predicted value;

[0037] Figure 4 It is a regression effect diagram of the real HR and predicted HR of UBFC-rPPG of the present invention;

[0038] Figure 5 It is a regression effect diagram of the COHFACE real HR and predicted HR of the present invention;

[0039] Figure 6 It is a network attention visualization diagram of the spatiotemporal dimension of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0041] like Figure 1-Figure 6 As shown, a remote heart rate detection method and system based on heartbeat probability of this embodiment includes the following specific steps:

[0042] Step 1: Capture the video of the person to be tested through the camera, and collect the PPG signal through the instrument;

[0043] Step 2: Preprocess the video data. Specifically, divide the video into video frames of length T. Each frame is represented in RGB format. The video sequence is recorded as X=(C, T, W, H), where C is the color channel, W is the width of the frame, and H is the height of the frame.

[0044] Step 3: Take the image at T=0, use MTCNN to perform face detection, crop the facial area of ​​each frame in the video and normalize it;

[0045] Step 4: Linearly interpolate the PPG sequence and perform mean normalization to obtain a PPG sequence of length T, and then perform peak detection (considering that PPG measurement may be disturbed by movement, if the heart rate detected by the peak value is significantly different from the heart rate obtained by the PSD method, this feature will be excluded). Let the BVP signal peak index sequence be P, D max =MAX(Diff(P)) is the maximum value of the peak interval, PB(x), PF(x) are the previous peak index and the next peak index of x respectively, calculate HBP;

[0046] Step 5: Consider each frame of the RGB video as a sampling, and then guess whether a certain frame is a heartbeat point. These images constitute a random process of whether it is a heartbeat point. The probability of a certain time point in this random process is related to the previous and next time points. When multiple values ​​of the signal before and after a certain time point are known, the probability of guessing the signal value at this time point will be greater, and the HBP(t) probability distribution is fitted;

[0047] Step 6: The HBP before and after associations obtained through spatiotemporal convolution learning are used to construct a hybrid attention model (HBP-Net network structure) using image full convolution and skip connections to obtain the HBP-Net model;

[0048] Step 7: Remote heart rate monitoring is performed by inputting the video into the HBP-Net model to recognize the facial BVP signal.

[0049] The HBP calculation formula in step 4 is as follows:

[0050]

[0051] The input feature of the HBP-Net network structure in step 6 is X∈R B×C×T×H×W , whose dimensions are batch, channel, time, width, and height respectively.

[0052] The spatiotemporal channel attention module in step 6 can calculate the weights by the following steps:

[0053] Pooling to get global information in time

[0054] X pool =Pool(X)∈R B×C×T×H×W

[0055] The number of channels is compressed through convolution or fully connected layers

[0056] W time =σ(Conv(X pool ))∈R B×C×T×1×1

[0057] Here σ is the Sigmoid activation function, and the generated W time is the attention weight at each time step, and W time Applied to the original input X

[0058] X out =X·W time

[0059] By weighting each frame, more important features in the time dimension are finally obtained.

[0060] The fusion loss is calculated through BCE, and BCE is adjusted to make the minimum loss value 0. The calculation method of Loss is as follows:

[0061]

[0062] Where y is the true probability distribution, is the predicted probability distribution, T is the number of samples, and the latter term of the subtraction does not participate in the derivation and back propagation if and only if When the same, Loss=0, Loss=0,when When , Loss→∞, and 10 is taken as the maximum value during the calculation process.

[0063] It includes a signal acquisition module, a data transmission module, a back-end processing center and a user terminal. The signal acquisition module is responsible for collecting data of the person to be detected through the camera sensor; the data transmission module is responsible for transmitting the collected signal data after preliminary processing to the back-end processing equipment in real time.

[0064] The back-end processing center is responsible for receiving data from various acquisition terminals and storing them in the database. It recognizes the data in the video through the HBP-Net model to obtain the BVP.

[0065] User terminals include mobile applications and medical workstations. Mobile applications: For ordinary users, we develop mobile applications that are compatible with iOS and Android systems. Users can view their heart rate data and heart rate trend charts in real time on their mobile phones, and set personalized heart rate warning thresholds. When the heart rate rises abnormally or drops beyond the set range, the mobile phone can push notifications in time to remind users to pay attention to their health conditions. It is also convenient for users to share data.

[0066] The medical workstation is designed for professional medical personnel. Doctors access server data through computer workstations and view patients' detailed heart rate monitoring reports, which assists doctors in disease diagnosis, especially for remote tracking treatment of patients in the recovery period of arrhythmia and cardiovascular diseases.

[0067] The present invention carries out the following experiment:

[0068] The experimental analysis includes a total of 260 minutes of video recordings from the COHFACE and UBFC-rPPG datasets, and the performance of other methods is compared, and then the lighting and motion stability are tested in our ZJXU-MOTION dataset. The reference heart rate (HR) values ​​of the training and test sets are calculated from the real PPG signals using the PSD method.

[0069] The COHFACE dataset contains 160 facial video recordings from 40 healthy subjects (28 males and 12 females) with an average age of 35±11 years old, covering different demographic characteristics (including age, gender, skin color, etc.). Each video is 60 seconds long, with a spatial resolution of 640×480 pixels and a low frame rate of 20fps. In order to enhance the diversity of lighting conditions, each subject recorded four videos under different lighting conditions, two of which were shot under studio lighting and the other two under natural light conditions with the curtains open. The experiment used a chest strap to record real breathing signals, and a BVP sensor (model SA9308M) was used to record real PPG signals at a sampling frequency of 256Hz.

[0070] The UBFC-rPPG dataset contains 42 video records that capture the facial area of ​​the subject. Each video is 60 seconds long, with a frame rate of 30fps, a resolution of 640×480 pixels, and the recording files are provided in an uncompressed format. All data collection was carried out indoors with varying lighting conditions. In the experiment, the subjects were required to play a time-sensitive math game facing the camera to increase their heart rate (HR). At the same time, the experiment used a transmissive fingertip pulse oximeter to record real PPG signals as a reference.

[0071] The results of training and testing in UBFC-rPPG and COHFACE are shown in Table 1 R is the Pearson correlation coefficient

[0072]

[0073]

[0074] Table 1

[0075] ZJXU-MOTION is a dataset constructed by the present invention, which contains 280 uncompressed RGB videos and 60 near-infrared videos of people aged between 20 and 45 years old. The spatial resolution is 30fps, and the recording time of each video is 1 minute. It provides 50Hz PPG signals, and the scenes are shown in Table 2:

[0076]

[0077] Table 2

[0078] The data of subjects 1 to 15 were selected to train HBP-Net, and then tested on the data of subjects 16 to 20. Table 3 shows the performance comparison of the HBP-Net method with the previous methods and multiple other methods inputting one batch. Compared with other deep learning-based methods in the table, the 3DCNN-based PhyNet shows higher stability by presenting comparable MAE and lower RMSE. LSTC-rPPG uses RhythmFormer and has achieved significant improvements in performance relative to other methods. Compared with all other methods, our proposed method shows excellent performance in all performance indicators, with MAE of 0.70, RMSE of 1.00, and r of 0.99.

[0079]

[0080]

[0081] Table 3

[0082] like Figure 6 As shown in Figure 6, the periodic attention map is visualized. The markers located to the left of the time point and space are given as queries. Attention score maps and key responses are obtained, where the greener areas indicate stronger attention, which is the area with high correlation provided by the attention phase to delve into more fine-grained rPPG features. Attention score maps and key responses are also obtained. As shown in Figure 6, the pre-attention scores are appropriately concentrated on skin areas with similar temporal phases, accurately expressing the periodic nature of rPPG. However, there are still many non-zero attention scores assigned to irrelevant areas. Fine attention is focused on the top-k areas, aiming to minimize attention to irrelevant areas. This focus of attention can extract rPPG features in more detail, reduce noise interference, and enhance the robustness of the model.

[0083] The effectiveness of HBP and the convergence effect of loss are verified by ablation tests. The experimental results are shown in Table 4, where Siloss represents the frequency domain scale-invariant loss, TARM represents the temporal attention refinement module, and the symbol √ indicates whether Siloss and TARM are applied. Compared with the baseline architecture, the use of Siloss and TARM alone leads to a slight improvement in MAE. In contrast, the joint use of Siloss and TARM shows a significant increase in MAE. Using TARM to optimize long-term and short-term features combined with the frequency scale-invariant hybrid loss to guide learning will produce a synergistic effect, thereby improving performance.

[0084]

[0085] Table 4

[0086] In order to analyze the computational efficiency of the proposed method, the parameters and multiplication-accumulation (MAC) performance of some implemented methods are calculated. The results are shown in Table 5:

[0087]

[0088] Table 5

[0089] The present invention achieves excellent performance with a moderate MAC with fewer parameters. Compared with other 3DCNNs models, HBP-Net uses fewer parameters, which shows that HBP-Net is easier to train to obtain a high-performance model.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A remote heart rate detection method based on heartbeat probability, characterized in that: The specific steps include: Step 1: Capture the video of the person to be tested through the camera, and collect the PPG signal through the instrument; Step 2: Preprocess the video data. Specifically, divide the video into video frames of length T. Each frame is represented in RGB format. The video sequence is recorded as X = (C, T, W, H), where C is the color channel, W is the width of the frame, and H is the height of the frame. Step 3: Take the image at T=0, use MTCNN to perform face detection, crop the facial area of ​​each frame in the video and normalize it; Step 4: Linearly interpolate the PPG sequence and perform mean normalization to obtain a PPG sequence of length T, and then perform peak detection (considering that PPG measurement may be disturbed by movement, if the heart rate detected by the peak value is significantly different from the heart rate obtained by the PSD method, this feature will be excluded). Let the BVP signal peak index sequence be P, D max =MAX(Diff(P)) is the maximum value of the peak interval, PB(x), PF(x) are the previous peak index and the next peak index of x respectively, calculate HBP; Step 5: Consider each frame of the RGB video as a sampling, and then guess whether a certain frame is a heartbeat point. These images constitute a random process of whether it is a heartbeat point. The probability of a certain time point in this random process is related to the previous and next time points. When multiple values ​​of the signal before and after a certain time point are known, the probability of guessing the signal value at this time point will be greater, and the HBP(t) probability distribution is fitted; Step 6: The HBP before and after associations obtained through spatiotemporal convolution learning are used to construct a hybrid attention model (HBP-Net network structure) using image full convolution and skip connections to obtain the HBP-Net model; Step 7: Remote heart rate monitoring is performed by inputting the video into the HBP-Net model to recognize the facial BVP signal.

2. A remote heart rate detection method based on heartbeat probability according to claim 1, characterized in that: The HBP calculation formula in step 4 is as follows: The input feature of the HBP-Net network structure in step 6 is X∈R B×C×T×H×W , whose dimensions are batch, channel, time, width and height respectively.

3. A remote heart rate detection method and system based on heartbeat probability according to claim 2, characterized in that: The spatiotemporal channel attention module in step 6 can calculate the weights by the following steps: Pooling to get global information in time X pool =Pool(X)∈R B×C×T×H×W The number of channels is compressed through convolution or fully connected layers W time =σ(Conv(X pool ))∈R B×C×T×1×1 Here σ is the Sigmoid activation function, and the generated W time is the attention weight at each time step, and W time Applied to the original input X X out =X·W time By weighting each frame, more important features in the time dimension are finally obtained.

4. A remote heart rate detection method based on heartbeat probability according to claim 3, characterized in that: The fusion loss is calculated through BCE, and BCE is adjusted to make the minimum loss value 0. The calculation method of Loss is as follows: Where y is the true probability distribution, is the predicted probability distribution, T is the number of samples, and the latter term of the subtraction does not participate in the derivation and back propagation if and only if When the same, Loss=0, Loss=0,when When , Loss→∞, and 10 is taken as the maximum value during the calculation process.

5. A remote heart rate detection system based on heartbeat probability according to claim 4, characterized in that: It includes a signal acquisition module, a data transmission module, a back-end processing center and a user terminal. The signal acquisition module is responsible for collecting data of the person to be detected through the camera sensor; the data transmission module is responsible for transmitting the collected signal data after preliminary processing to the back-end processing equipment in real time.

6. A remote heart rate detection system based on heartbeat probability according to claim 5, characterized in that: The back-end processing center is responsible for receiving data from various acquisition terminals and storing them in the database. It recognizes the data in the video through the HBP-Net model to obtain the BVP.

7. A remote heart rate detection system based on heartbeat probability according to claim 6, characterized in that: User terminals include mobile applications and medical workstations. Mobile applications: For ordinary users, we develop mobile applications that are compatible with iOS and Android systems. Users can view their heart rate data and heart rate trend charts in real time on their mobile phones, and set personalized heart rate warning thresholds. When the heart rate rises abnormally or drops beyond the set range, the mobile phone can push notifications in time to remind users to pay attention to their health conditions. It is also convenient for users to share data. The medical workstation is designed for professional medical personnel. Doctors access server data through computer workstations and view patients' detailed heart rate monitoring reports, which assists doctors in disease diagnosis, especially for remote tracking treatment of patients in the recovery period of arrhythmia and cardiovascular diseases.

Images