Video stream continuous blood pressure measurement method and device based on deep learning

Abstract

The invention discloses a video stream continuous blood pressure measurement method and device based on deep learning. According to the method, a video human face region of interest is optimized and selected to obtain a video pulse wave (rPPG) and rPPG reconstruction based on a CNN-BiLSTM network, so that a high-quality rPPG signal is obtained; a CLC-BP deep learning model based on a public data set and an experimental data set is constructed for human body blood pressure prediction, and a mixed rPPG blood pressure prediction method based on pulse waveform and characteristics is provided. On the basis, the video blood pressure measuring device based on the ARM Cortex-A72 embedded system is designed and realized, the software and hardware system design, the man-machine interaction system design and the data browsing interface design of the whole device are completed, then the embedded deep learning model deployment based on Arm NN is completed, the whole system operation time is greatly shortened, and the system reliability is improved. And the practical feasibility of the equipment is improved.

Description

Technical Field

[0001] This invention belongs to the field of biotechnology and new medicine, specifically relating to a method and device for continuous blood pressure measurement via video stream based on deep learning. Background Technology

[0002] Blood pressure (BP), as one of the four vital signs of the human body, is an extremely important control factor in the prevention and treatment of cardiovascular and cerebrovascular diseases. Elevated blood pressure is one of the important factors in the development of cardiovascular and cerebrovascular diseases. Globally, about 1.4 billion people suffer from hypertension, but more than 600 million hypertensive patients are unaware of their condition, resulting in poor control of hypertension. As a result, the demand for self-health monitoring is constantly expanding, and various new health monitoring devices have emerged accordingly.

[0003] Compared to other physiological parameters, blood pressure measurement is more difficult. Currently, blood pressure is generally measured using traditional cuff or wristband methods, which require the purchase of specialized blood pressure measuring equipment. This method cannot measure continuous blood pressure signals and no longer meets people's requirements in terms of comfort and portability.

[0004] Continuous blood pressure monitoring helps to understand the diurnal rhythm of blood pressure changes, thereby achieving the goal of preventing sudden cardiovascular diseases. Studies have shown that diseases such as stroke and sudden cardiac death have a higher incidence rate in the morning. Continuous blood pressure measurement can detect abnormal fluctuations in blood pressure in a timely manner, achieving the goal of early prevention and control of the disease. On the other hand, daily continuous blood pressure monitoring helps people taking antihypertensive drugs understand the effect of their medication on blood pressure, preventing situations such as overdose leading to excessively low blood pressure or insufficient dosage failing to achieve the desired blood pressure-lowering effect.

[0005] Extensive research has shown that blood pressure variability (BPV), the variation in blood pressure over a period of time, is closely related to the occurrence of cardiovascular and cerebrovascular diseases. The calculation of BPV relies on obtaining continuous blood pressure data. Currently, most studies on BPV use blood pressure data obtained through ambulatory blood pressure monitoring (ABPM), which measures blood pressure every 15-30 minutes. This method cannot achieve continuous, beat-by-beat blood pressure measurement over 24 hours, and therefore cannot provide more detailed information on BPV variability. Therefore, developing a continuous blood pressure monitoring algorithm suitable for daily patient monitoring would contribute to further research and analysis of BPV. Summary of the Invention

[0006] This invention provides a method and device for continuous blood pressure measurement via video stream based on deep learning. The invention focuses on the development and research of video signal processing, pulse wave signal feature analysis, and the correlation between pulse wave and blood pressure signals. It designs and proposes a portable blood pressure monitor based on remote photoplethosmograph (rPPG). By solving key problems such as rPPG reconstruction, blood pressure detection model construction, and the deployment of embedded deep learning algorithms, a novel method and device for continuous blood pressure measurement via video stream based on deep learning are designed and implemented.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A deep learning-based video stream continuous blood pressure measurement device includes: The camera video stream acquisition module is used to record continuous video signals of regions of interest of human faces and to extract facial pulse wave signals using a parallel processing method. The video pulse wave signal processing and reconstruction module is used to perform signal preprocessing on the obtained face pulse wave signal and pulse wave reconstruction based on a deep learning model to obtain a complete face pulse wave signal. The blood pressure modeling module is used to filter and optimize the pulse wave features in the time and frequency domains of facial pulse wave signals, and to complete the construction and training of a deep learning model for pulse wave blood pressure estimation based on public databases and experimental datasets. The embedded blood pressure calculation module is used to deploy and implement the embedded microcontroller of the trained pulse wave blood pressure estimation deep learning model, and to calculate systolic blood pressure, diastolic blood pressure, mean blood pressure and heart rate. The overall system design module is used for human-machine interaction button management, user and parameter display management, and Bluetooth remote transmission of measurement data.

[0008] A further improvement of the present invention is that the camera video stream acquisition module includes: The camera selected is a narrow-angle, high-resolution, distortion-free camera, which captures 30 seconds of continuous video signal from a 100*200 pixel area below the eyelids of the face; A CHROM extraction algorithm based on chromaticity analysis was used to extract a 30-second original facial pulse wave signal simultaneously with video acquisition.

[0009] A further improvement of the present invention is that the video pulse wave signal processing and reconstruction module includes: The original facial pulse wave signal was segmented and augmented to construct a dataset for deep learning models; A CNN-BiLSTM deep learning model based on the combination of convolutional neural network and bidirectional long short-term memory network is constructed to reconstruct the quality of the original rPPG signal, restore the details of the rPPG signal, and improve the quality of the rPPG signal.

[0010] A further improvement of this invention is that the blood pressure modeling module is constructed using the following method: A CLC-BP blood pressure prediction model combining CNN and BiLSTM was constructed to achieve feature extraction and time series processing, and to complete video blood pressure estimation based on rPPG+ feature hybrid input. The experimental datasets required for model building include 34,875 sets of data from 432 people in the public dataset MIMIC III, and data from 15 people in experimental dataset 1 and 20 people in experimental dataset 2, which are used as the model training set, validation set and test set respectively, thus constructing a complete experimental dataset scheme. A hybrid input calibration model for blood pressure prediction is developed based on a combination of intrinsic signal PPG and PPG signal characteristics, thereby improving the accuracy of blood pressure prediction.

[0011] A further improvement of the present invention is that the embedded blood pressure calculation module includes: The deployment and algorithm implementation of an embedded blood pressure calculation method based on an ARM Cortex-A72 CPU Raspberry Pi microcontroller system were carried out, and the continuous calculation and output of blood pressure parameters were completed. Complete the rPPG segmented reconstruction calculation of 30 seconds of continuous video data and output 15 points of blood pressure calculation data.

[0012] A further improvement of this invention lies in the blood pressure monitor system design module, which includes: Bluetooth remote transmission of measurement data integrates signal acquisition, data processing, feature calculation, and parameter calculation into four main tasks. It manages eight display interfaces for user selection, parameter measurement, curve display, and historical data retrieval by periodically scanning four buttons: scroll down, scroll up, confirm, and send. It completes the functions of signal acquisition, measurement, display, and communication. The human-computer interaction button management adopts a duplex button design to realize the functions of screen interface page turning, user selection, measurement start, and communication data transmission. The user and parameter display management provides multi-user management and multi-page data display management functions.

[0013] A deep learning-based method for continuous blood pressure measurement via video stream, comprising: The camera video stream acquisition module records continuous video signals of regions of interest on faces and uses a parallel processing method to extract facial pulse wave signals; The video pulse wave signal processing and reconstruction module performs signal preprocessing and pulse wave reconstruction based on a deep learning model on the obtained face pulse wave signal to obtain a complete face pulse wave signal. The blood pressure modeling module filters and optimizes the pulse wave features in the time and frequency domains of facial pulse wave signals, and completes the construction and training of a deep learning model for pulse wave blood pressure estimation based on public databases and experimental datasets. The embedded blood pressure calculation module completes the deployment and algorithm implementation of the embedded microcontroller for the trained pulse wave blood pressure estimation deep learning model, and calculates systolic blood pressure, diastolic blood pressure, mean blood pressure and heart rate. The overall system design module manages human-machine interaction buttons, user and parameter display, and Bluetooth remote transmission of measurement data.

[0014] Compared with the prior art, the present invention has at least the following beneficial technical effects: This invention includes the CHROM method for extracting facial video pulse waves and a pulse wave reconstruction method based on a CNN-BiLSTM deep learning model. It proposes a hybrid video pulse wave blood pressure prediction method based on pulse wave waveform and waveform features, constructs a CNN-BiLSTM-CNN-based video blood pressure prediction model (CLC-BP (CNN-BiLSTM-CNN Bloodpressure Prediction)), and based on this, completes an embedded video blood pressure detection device based on an ARM Cortex-A72 CPU microcontroller. It also completes key tasks such as optimizing the deployment scheme of the embedded deep learning model based on Arm NN, designing the overall hardware and software system, designing the human-computer interaction system, and designing the data interface. Attached Figure Description

[0015] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the design of a video blood pressure monitor.

[0017] Figure 2 This is the RGB waveform signal extracted from the face video signal.

[0018] Figure 3 These are the fingertip PPG signal and the rPPG signal extracted by the CHROM method, respectively.

[0019] Figure 4This is a flowchart of the reconstruction process of rPPG signals based on a CNN-BiLSTM network.

[0020] Figure 5 This is a schematic diagram of a video blood pressure prediction model based on a CLC-BP network.

[0021] Figure 6 This diagram illustrates the combination of signal and feature inputs in the CLC-BP model.

[0022] Figure 7 Pearson correlation plot and Bland-Altman plot for SBP and DBP under mixed input mode.

[0023] Figure 8 Correlation analysis and Bland-Altman plot for SBP blood pressure estimation before and after rPPG signal reconstruction.

[0024] Figure 9 Correlation analysis and Bland-Altman plot for DBP blood pressure estimation before and after rPPG signal reconstruction.

[0025] Figure 10 Flowchart for deploying the hardware environment for the video blood pressure monitor algorithm.

[0026] Figure 11 This diagram illustrates the comparison of single-repetition prediction time and overall runtime under three software framework models.

[0027] Figure 12 This diagram illustrates a comparison of the overall one-time prediction time and overall runtime under four software framework models.

[0028] Figure 13 This is a schematic diagram of the hardware configuration of a video blood pressure monitor based on a quad-core ARM Cortex-A72 microcontroller.

[0029] Figure 14 A schematic diagram of the software design framework for a video blood pressure monitor.

[0030] Figure 15 A schematic diagram of the user interface design for the detector. Detailed Implementation

[0031] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0032] In the description of this invention, it should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0033] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0034] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0035] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0036] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0037] Example 1 This invention provides a deep learning-based video stream continuous blood pressure measurement device, comprising: The camera video stream acquisition module is used to record continuous video signals of regions of interest of human faces and to extract facial pulse wave signals using a parallel processing method. The video pulse wave signal processing and reconstruction module is used to perform signal preprocessing on the obtained face pulse wave signal and pulse wave reconstruction based on a deep learning model to obtain a complete face pulse wave signal. The blood pressure modeling module is used to filter and optimize the pulse wave features in the time and frequency domains of facial pulse wave signals, and to complete the construction and training of a deep learning model for pulse wave blood pressure estimation based on public databases and experimental datasets. The embedded blood pressure calculation module is used to deploy and implement the embedded microcontroller of the trained pulse wave blood pressure estimation deep learning model, and to calculate systolic blood pressure, diastolic blood pressure, mean blood pressure and heart rate. The overall system design module is used for human-machine interaction button management, user and parameter display management, and Bluetooth remote transmission of measurement data.

[0038] The camera video stream acquisition module in this embodiment includes: The camera selected is a narrow-angle, high-resolution, distortion-free camera, which captures 30 seconds of continuous video signal from a 100*200 pixel area below the eyelids of the face; A CHROM extraction algorithm based on chromaticity analysis was used to extract a 30-second original facial pulse wave signal simultaneously with video acquisition.

[0039] The video pulse wave signal processing and reconstruction module in this embodiment includes: The original facial pulse wave signal was segmented and augmented to construct a dataset for deep learning models; A CNN-BiLSTM deep learning model based on the combination of convolutional neural network and bidirectional long short-term memory network is constructed to reconstruct the quality of the original rPPG signal, restore the details of the rPPG signal, and improve the quality of the rPPG signal.

[0040] The blood pressure modeling module in this embodiment is constructed using the following method: A CLC-BP blood pressure prediction model combining CNN and BiLSTM was constructed to achieve feature extraction and time series processing, and to complete video blood pressure estimation based on rPPG+ feature hybrid input. The experimental datasets required for model building include 34,875 sets of data from 432 people in the public dataset MIMIC III, and data from 15 people in experimental dataset 1 and 20 people in experimental dataset 2, which are used as the model training set, validation set and test set respectively, thus constructing a complete experimental dataset scheme. A hybrid input calibration model for blood pressure prediction is developed based on a combination of intrinsic signal PPG and PPG signal characteristics, thereby improving the accuracy of blood pressure prediction.

[0041] The embedded blood pressure calculation module in this embodiment includes: The deployment and algorithm implementation of an embedded blood pressure calculation method based on an ARM Cortex-A72 CPU Raspberry Pi microcontroller system were carried out, and the continuous calculation and output of blood pressure parameters were completed. Complete the rPPG segmented reconstruction calculation of 30 seconds of continuous video data and output 15 points of blood pressure calculation data.

[0042] The blood pressure monitor system design module in this embodiment includes: Bluetooth remote transmission of measurement data integrates signal acquisition, data processing, feature calculation, and parameter calculation into four main tasks. It manages eight display interfaces for user selection, parameter measurement, curve display, and historical data retrieval by periodically scanning four buttons: scroll down, scroll up, confirm, and send. It completes the functions of signal acquisition, measurement, display, and communication. The human-computer interaction button management adopts a duplex button design to realize the functions of screen interface page turning, user selection, measurement start, and communication data transmission. The user and parameter display management provides multi-user management and multi-page data display management functions.

[0043] Example 2 Figure 1 This is a design flowchart for a video blood pressure monitor, mainly consisting of three parts. First, the video blood pressure detection method and modeling: rPPG signals from facial videos are extracted using databases and experimental data collection. An rPPG reconstruction model is constructed using deep learning. Based on the reconstructed rPPG signals, a blood pressure prediction model is studied, a video blood pressure detection method is designed, and the accuracy of the video blood pressure detection method is verified. Second, the model porting and algorithm design for an ARM Cortex-A72-based embedded system are completed. First, rPPG signals are extracted from facial videos. Then, the original rPPG signals are input into the reconstruction model to obtain high-quality reconstructed rPPG signals. These reconstructed rPPG signals are then input into the blood pressure prediction model to obtain blood pressure prediction results. Basic performance testing and algorithm optimization, including facial video rPPG synchronous processing, rPPG signal amplification, and model optimization, are completed. Finally, the modular encapsulation of the non-contact blood pressure detection algorithm based on the embedded system is completed, along with the design of the monitor's user interface, multi-user data storage and display, and other device functions. Overall system testing and accuracy verification are also completed.

[0044] The data sources required for designing the deep learning model and algorithm in this invention mainly include public datasets and laboratory-collected datasets. Public dataset 1 is the UBFC-RPPG database containing facial videos and fingertip PPG signals of 42 subjects, with a camera frame rate of 30fps, a resolution of 640*480, and a video length of 1-2 minutes. This data is mainly used for building the rPPG reconstruction model. Public dataset 2 contains the MIMIC Ⅲ database containing users' continuous blood pressure and PPG signals, with signal sampling at 125 Hz. The physiological data of 432 subjects selected are mainly used to build a blood pressure prediction model. Experimental dataset 1 contains color video signals, continuous blood pressure, and PPG signals simultaneously collected from 5 subjects, mainly used to optimize rPPG reconstruction and validate the blood pressure model. Experimental dataset 2 contains black and white video signals and continuous blood pressure data simultaneously collected from 20 subjects, mainly used to validate the blood pressure prediction based on the reconstructed rPPG signal.

[0045] This invention primarily draws upon the standards of the Association for the Advancement of Medical Instrumentation (AAMI) and the British Hypertension Society (BHS) for the predicted blood pressure measurement. The AAMI standard requires that the mean absolute error and standard deviation between the blood pressure values ​​obtained by the target device or method and the actual diastolic and systolic blood pressure be less than 5 mmHg and 8 mmHg, respectively. The BHS uses a tiered standard for blood pressure measurement error, with Grade A requiring that the percentage of predicted blood pressure values ​​with errors less than 5 mmHg is greater than 60%, less than 10 mmHg is greater than 85%, and less than 15 mmHg is greater than 95%.

[0046] This invention extracts video pulse waves by recording a 30-second video signal of the region of interest (ROI) on a face, thereby completing subsequent blood pressure estimation. Therefore, the first step is to acquire the video signal of the face. To achieve rapid blood pressure estimation, a human-computer interaction method is adopted, in which the position of the nose is clearly marked on the device screen. The nose can be centered within a distance of 0.5 meters. The device automatically identifies and selects a continuous video signal of a 100*100 pixel area at the left eyelid of the face as the ROI, thereby greatly avoiding the time consumption caused by face recognition and ensuring that the required rPPG signal is extracted.

[0047] Figure 2 This is the RGB three-color signal of the video after processing by a Butterworth fourth-order bandpass filter, where each point is a weighted average of the pixels in each video frame. This invention selects the CHROM method based on chrominance extraction to extract the rPPG signal from the RGB signal. It has advantages such as strong anti-interference ability, fast processing speed, and certain robustness to non-white light illumination. The calculation formula is as follows:

[0048]

[0049] ,

[0050] In the above formula It is a video RGB signal. yes Standard deviation; yes The standard deviation; S is the video pulse wave rPPG signal to be extracted, such as... Figure 3As shown, compared to the standard PPG signal at the fingertip above, Figure 3 The lower middle part shows the rPPG signal extracted from the poor-quality facial video. It can be seen that the rPPG signal extracted using the CHROM method is relatively close to the standard fingertip PPG signal, but it still has the disadvantage of not being obvious in the diphthora wave, and the period consistency is also poor.

[0051] To address the issue of low-quality rPPG signals extracted from videos, this invention designs a deep learning model, CNN-BiLSTM, which combines a convolutional neural network (CNN) and a bidirectional long short-term memory (BiLSTM) network to reconstruct the rPPG signal. The CNN-BiLSTM model is as follows: Figure 4 As shown, this network structure combines the characteristics of CNN in accurate mapping between output and input with the characteristics of BiLSTM in bidirectional computation of long-term sequences, effectively reconstructing the rPPG signal. The reconstruction model was pre-trained using rPPG calculated from face video data in public dataset 1 and measured PPG data. Retraining and performance testing were performed using 1500 sets of data from 5 individuals collected in the laboratory 2. Table 1 shows the calculation results of the pulse wave feature values ​​of interest in blood pressure detection in this invention before and after reconstruction. Percentage error analysis shows that the reconstructed rPPG signal is improved in all features compared to the original rPPG signal. Subsequent blood pressure detection results also indicate that the reconstructed rPPG has better detection results.

[0052]

[0053] Based on the superior performance of CNN neural networks in feature extraction and BiLSTM networks in time series processing, this invention proposes a CLC-BP blood pressure prediction model that combines the advantages of CNN and BiLSTM. The model structure is as follows: Figure 5 As shown.

[0054] The blood pressure prediction model is mainly divided into three parts. The first part consists of a CNN network and pooling layers. This is used to initially extract features from the input signal. The ReLU activation function is chosen to enhance nonlinearity, and pooling layers are used to reduce feature dimensionality and computational load while preserving key features. The second part consists of a three-layer BiLSTM and Dropout layer. This mainly performs temporal analysis on the signal processed in the first part. BiLSTM can perform forward and inverse time-series analysis, improving the ability to analyze time correlations. The pulse wave signal is time-correlated and is a pseudo-periodic signal. A Dropout layer is connected after each BiLSTM layer to prevent overfitting. The third part consists of a two-layer CNN, pooling layer, Dropout layer, and two fully connected layers. This is mainly for feature re-extraction and blood pressure prediction, addressing the nonlinear correlation between pulse wave signals and blood pressure. The two CNN layers extract features for blood pressure prediction, while the pooling and Dropout layers reduce computation and prevent overfitting. Finally, two fully connected layers are connected to perform feature regression to obtain the predicted blood pressure value.

[0055] The CLC-BP prediction model was pre-trained and validated using a public dataset 2. According to the training requirements, 34,875 sets of PPG and blood pressure signal data from 432 subjects were selected, with 22,320 sets for the training set and 5,580 sets for the test set. At the same time, 6,975 sets of data were selected as the validation set.

[0056] In Experimental Dataset 1, 1200 data points from 5 participants were divided into training and testing sets. After processing their synchronized blood pressure signals, the data included PPG, rPPG, and the signals and features of reconstructed rPPG. Experimental Dataset 2, 400 data points from 20 participants, was used as the final validation data for the blood pressure prediction model.

[0057] To improve the accuracy of blood pressure prediction using pulse wave signals, this invention proposes a calibrated blood pressure prediction model based on a combination of the original signal and signal features. Six universal features most relevant to blood pressure prediction are selected and combined with the reconstructed waveform as model input to remove interfering factors and improve the accuracy of blood pressure prediction. Figure 6 The diagram shows a hybrid input consisting of the original signal and features. The original signal has a length of 72 points. The extracted 6 features are combined into a 1x6 feature vector. The feature vector is then augmented and concatenated with the original signal to obtain a 72x7 feature vector as the final model input data format. Finally, the training, testing and validation of the model are completed.

[0058] Table 2 illustrates the effectiveness of the CLC-BP blood pressure prediction model under mixed input mode. In the table, both systolic blood pressure (SBP) and diastolic blood pressure (DBP) meet the BHS A and C standards. Compared with PPG input, the PPG+ feature method improves the prediction results under the BHS standard except for DBP data ≤15mmHg, and the improvement is more significant in SBP.

[0059]

[0060] Figure 7 To employ statistical methods, Pearson correlation plots and Bland-Altman plots for SBP and DBP under the mixed input method were obtained. The above shows the SBP results; the correlation coefficient for SBP is 0.97, with a mean absolute error and standard deviation of 3.45 mmHg and 5.82 mmHg, respectively. Most data lie within the 95% confidence interval, however... The SD range is relatively large, around 10 mmHg; the results are converted to DBP, with a correlation coefficient of 0.95, a mean absolute error of 1.62 mmHg, and a standard deviation of 2.83 mmHg. Most data points fall within the 95% confidence interval. The SD of 1.96 mmHg is also around 5 mmHg, and both SBP and DBP meet the AAMI standard. This indicates that the hybrid input method can improve the accuracy of blood pressure prediction.

[0061] Considering the differences between different datasets, the data from Experimental Data Set 1 that was not used in model training was selected for retraining the blood pressure prediction model. The data from Experimental Data Set 2 was used to verify the accuracy of the model. rPPG was extracted and reconstructed, and the rPPG and corresponding features were used as input to the CLC-BP model for blood pressure prediction, thereby verifying the advantages of reconstructing the rPPG signal for blood pressure prediction. The results are shown in Table 3.

[0062]

[0063] Table 3 shows that the SBP and DBP prediction results of the reconstructed rPPG signal both meet the BHS standard (Grade A), while the SBP prediction result of the rPPG signal only meets the Grade B standard, and the DBP prediction result meets the Grade A standard. The prediction results indicate that the reconstructed rPPG signal accounts for a higher proportion in each interval than the original rPPG signal, suggesting that using the reconstructed rPPG signal is more advantageous and improves the accuracy of blood pressure prediction.

[0064] Meanwhile, statistical analysis was performed on the blood pressure prediction results of experimental dataset 2, namely rPPG signal and reconstructed rPPG signal, and Pearson correlation analysis plots and Bland-Altman plots of SBP and DBP for the two inputs were obtained.

[0065] Figure 8 Analysis of SBP prediction results for the original and reconstructed rPPG signals shows that the Pearson correlation coefficient for SBP of the original rPPG signal is 0.46, while the correlation coefficient of the reconstructed rPPG signal is 0.69, with a MAE of 4.51 mmHg and a STD of 7.21 mmHg, both showing good consistency with the true values. Therefore, using the reconstructed rPPG signal can improve the accuracy of SBP prediction.

[0066] Figure 9 Analysis of the DBP prediction results for the original and reconstructed rPPG signals shows that the Pearson correlation coefficient for DBP of the original rPPG signal is 0.42, while that of the reconstructed rPPG signal is 0.63. The MAE is 3.00 mmHg, and the STD is 4.95 mmHg. Although most data from the original rPPG signal fall within the 95% confidence interval, the interval range is relatively large. In contrast, the reconstructed rPPG signal not only has a higher correlation coefficient but also a smaller 95% confidence interval. Therefore, using the reconstructed rPPG signal can improve the prediction accuracy for DBP.

[0067] Figure 8 , Figure 9 Statistical analysis of the blood pressure prediction model results showed that the reconstructed signal had higher prediction accuracy and better consistency.

[0068] The hardware environment of the video blood pressure monitor proposed in this invention is a Raspberry Pi 4B microcontroller based on an ARM Cortex-A72 CPU. The deep learning prediction model, after algorithm verification, still needs to be deployed and further optimized on this microcontroller to achieve accurate and fast operation of the deep learning algorithm and convenient user operation. The deployment and optimization process is as follows: Figure 10 As shown.

[0069] When deploying deep learning models to embedded systems, there are many software frameworks that can accelerate deep learning model inference. This invention compares the impact of three model deployment methods—TensorFlow, TensorFlow Lite, and Arm NN—on parameters such as speed and accuracy, and ultimately selects the Arm NN model deployment method to solve the problem of implementing deep learning algorithms on a microcontroller.

[0070] Figure 11The graph compares the computation time of a single repeated calculation model and the overall running time under different CPU utilization rates. As the CPU utilization rate increases, the inference time of all three models tends to decrease. The overall running time, including model loading and weight loading, becomes the key factor affecting the speed.

[0071] Figure 12 To compare the overall one-time prediction time and overall runtime of four software framework models, including TensorFlow (saved_model), TensorFlow (Keras), TensorFlow Lite (tflite), and Arm NN, the runtime of all four models decreased as CPU utilization increased when predicting 1000 data points at a time. The Arm NN-based model consistently had the fastest inference speed. The runtime of the four models accounted for 33.90%, 66.92%, 98.66%, and 83.58% of the total, respectively. From an overall perspective, the TensorFlow Lite format model and the Arm NN-based model performed the best.

[0072] Table 4 shows the computation time of each unit module of the main algorithm of this invention. The Arm NN-based model deployment has no additional model loading steps, and the total time is shortened by 28.11 seconds, which is a significant performance improvement.

[0073]

[0074] Table 5 compares the accuracy of blood pressure calculation using the TensorFlow Lite model, the Arm NN model, and a general-purpose computer. The Arm NN-based model and the inference results on the computer have the same accuracy, both meeting the standards of BHS and AAMI. However, the TensorFlow Lite model's prediction of SBP did not meet the AAMI Grade A standard.

[0075]

[0076] This invention determines that the Arm NN model is used as the deployment scheme for the deep learning model of the detector. The 30-second data is repeatedly divided into 15 segments of 5 seconds each, and the inference calculation is performed in one go. This effectively reduces the communication time between the CPU and memory during the inference process, thereby achieving the goal of optimal performance.

[0077] The video blood pressure monitor proposed in this invention is a complex embedded application system that combines hardware and software. It not only needs to provide accurate blood pressure detection, but also needs to have an intuitive and user-friendly interface and a stable and reliable system program design.

[0078] Figure 13For the hardware system composition of the detector, the microcontroller hardware system selects a quad-core ARM Cortex-A72 as the CPU. The main parameters are: Raspberry Pi 4B, main frequency 1.5GHz, memory 8G+32GB, 7-inch touch screen, resolution 1920*1080, operating system is Linux system, and compilation environment is Python 3.7.

[0079] Figure 14 The software framework for the detection instrument system was designed. In order to meet the needs of human-computer interaction and design a simple and easy-to-use graphical user interface, the video blood pressure detection software based on the Python language PyQt5 framework was used to complete the system software design.

[0080] To fulfill the system software functional requirements, a user-friendly multi-user, multi-interface interactive system was designed. This system allows users to navigate through five categories of nine different interfaces via touch-sensitive up and down buttons on the display screen: user selection interface, parameter measurement interface, parameter display interface, health tips interface, and historical data interface for five users. Figure 15 As shown, first, select 5 different users using the touch keys; all subsequent measurement data will be saved using these users. Then, enter the blood pressure measurement interface, point your nose at the prompt box, record a 30-second video, and the analysis and calculation will begin. After about 6 seconds of calculation, you can turn to the parameter display function. The display interface provides parameters such as systolic blood pressure, diastolic blood pressure, mean blood pressure, and pulse rate, as well as parameter fluctuation curves within 30 seconds. To popularize hypertension knowledge, an interface is provided displaying national standards for hypertension and related knowledge on hypertension prevention. Finally, there is a review interface for 20 historical data points for the 5 users, which is updated in real time after each measurement and displayed in order from newest to oldest.

[0081] Example 3 This invention provides a deep learning-based method for continuous blood pressure measurement via video streams, comprising: The camera video stream acquisition module records continuous video signals of regions of interest on faces and uses a parallel processing method to extract facial pulse wave signals; The video pulse wave signal processing and reconstruction module performs signal preprocessing and pulse wave reconstruction based on a deep learning model on the obtained face pulse wave signal to obtain a complete face pulse wave signal. The blood pressure modeling module filters and optimizes the pulse wave features in the time and frequency domains of facial pulse wave signals, and completes the construction and training of a deep learning model for pulse wave blood pressure estimation based on public databases and experimental datasets. The embedded blood pressure calculation module completes the deployment and algorithm implementation of the embedded microcontroller for the trained pulse wave blood pressure estimation deep learning model, and calculates systolic blood pressure, diastolic blood pressure, mean blood pressure and heart rate. The overall system design module manages human-machine interaction buttons, user and parameter display, and Bluetooth remote transmission of measurement data.

[0082] In this embodiment, the camera video stream acquisition module includes: The camera selected is a narrow-angle, high-resolution, distortion-free camera, which captures 30 seconds of continuous video signal from a 100*200 pixel area below the eyelids of the face; A CHROM extraction algorithm based on chromaticity analysis was used to extract a 30-second original facial pulse wave signal simultaneously with video acquisition.

[0083] In this embodiment, the video pulse wave signal processing and reconstruction module includes: The original facial pulse wave signal was segmented and augmented to construct a dataset for deep learning models; A CNN-BiLSTM deep learning model based on the combination of convolutional neural network and bidirectional long short-term memory network is constructed to reconstruct the quality of the original rPPG signal, restore the details of the rPPG signal, and improve the quality of the rPPG signal.

[0084] In this embodiment, the blood pressure modeling module is constructed using the following method: A CLC-BP blood pressure prediction model combining CNN and BiLSTM was constructed to achieve feature extraction and time series processing, and to complete video blood pressure estimation based on rPPG+ feature hybrid input. The experimental datasets required for model building include 34,875 sets of data from 432 people in the public dataset MIMIC III, and data from 15 people in experimental dataset 1 and 20 people in experimental dataset 2, which are used as the model training set, validation set and test set respectively, thus constructing a complete experimental dataset scheme. A hybrid input calibration model for blood pressure prediction is developed based on a combination of intrinsic signal PPG and PPG signal characteristics, thereby improving the accuracy of blood pressure prediction.

[0085] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the scope of the invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0086] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be appropriately combined to form other embodiments that can be understood by those skilled in the art. The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A deep learning based video stream continuous blood pressure measurement device, characterized in that, Comprise: Camera video stream acquisition module, for taking continuous face region of interest video signal, and using parallel processing method, extracting face pulse wave signal; Video pulse wave signal processing and reconstruction module, for completing signal preprocessing of obtained face pulse wave signal, and pulse wave reconstruction based on deep learning model, to obtain complete face pulse wave signal; Blood pressure modeling module, for screening and optimizing pulse wave features in time domain and frequency domain of face pulse wave signal, completing construction and training of pulse wave blood pressure estimation deep learning model based on public database and experimental data set; Embedded blood pressure calculation module, for completing deployment and algorithm implementation of trained pulse wave blood pressure estimation deep learning model in embedded single-chip microcomputer, and completing calculation of systolic pressure, diastolic pressure, mean pressure and heart rate; Whole machine system design module, for human-computer interaction key management, user and parameter display management, and Bluetooth remote transmission of measurement data.

2. The deep learning based video stream continuous blood pressure measurement device according to claim 1, wherein, Camera video stream acquisition module, comprising: Selecting a narrow-view high-resolution non-distortion camera, collecting 30-second continuous video signal of 100*200 pixel area under the eyelid of the face; Using CHROM extraction algorithm based on chroma analysis, synchronously completing extraction of 30-second length original face pulse wave signal with video acquisition.

3. The deep learning based video stream continuous blood pressure measurement device according to claim 2, wherein, Video pulse wave signal processing and reconstruction module, comprising: Segmenting and data augmenting the original face pulse wave signal, constructing data set for deep learning model; Constructing CNN-BiLSTM deep learning model based on combination of convolutional neural network and bidirectional long short-term memory network, completing quality reconstruction of original rPPG signal, restoring rPPG signal details, and improving rPPG signal quality.

4. The deep learning based video stream continuous blood pressure measurement device according to claim 1, wherein, The blood pressure modeling module is constructed by the following method: Constructing CLC-BP blood pressure prediction model combining CNN and BiLSTM, realizing feature extraction and time series comprehensive processing, and completing video blood pressure estimation based on rPPG+ feature mixed input; The experimental data set required for model construction includes 34875 groups of data of 432 people of public data set MIMIC III, and 15 people of experimental data set 1 and 20 people of experimental data set 2, which are respectively used as model training set, validation set and test set, and a complete experimental data set scheme is constructed; The mixed input calibration blood pressure prediction model based on intrinsic signal PPG and PPG signal features improves blood pressure prediction accuracy.

5. The deep learning based video stream continuous blood pressure measurement device according to claim 1, wherein, Embedded blood pressure calculation module, comprising: Deployment and algorithm implementation of embedded blood pressure calculation method based on ARM Cortex-A72 CPU Raspberry Pi single-chip microcomputer system, completing continuous calculation and output of blood pressure parameters; Completing rPPG segmentation reconstruction calculation of 30-second continuous video data and 15-point blood pressure calculation data output.

6. The deep learning based video stream continuous blood pressure measurement device according to claim 1, wherein, Blood pressure meter whole machine system design module, comprising: The Bluetooth remote transmission of measurement data is to take signal acquisition, data processing, feature calculation and parameter calculation as four tasks, to manage the user selection, parameter measurement, curve display and historical data review of the measurement device through the four keys of down flip, up flip, confirmation and transmission, to complete the functions of signal acquisition, measurement, display and communication; The human-computer interaction key management adopts a compound key design to realize the functions of display screen interface page turning, user selection, measurement start and communication data transmission; The user and parameter display management provides multi-user management and multi-page data display management functions.

7. A deep learning based video stream continuous blood pressure measurement method, characterized in that, It includes: The camera video stream acquisition module records continuous face region of interest video signals and uses a parallel processing method to extract face pulse wave signals; The video pulse wave signal processing and reconstruction module completes signal preprocessing and pulse wave reconstruction based on a deep learning model to obtain complete face pulse wave signals; The blood pressure modeling module filters and optimizes pulse wave features in the time domain and frequency domain of the face pulse wave signals, constructs and trains a pulse wave blood pressure estimation deep learning model based on public databases and experimental data sets; The embedded blood pressure calculation module deploys and implements the trained pulse wave blood pressure estimation deep learning model on an embedded single-chip microcomputer, and calculates systolic pressure, diastolic pressure, mean pressure and heart rate; The whole system design module manages human-computer interaction keys, user and parameter display, and Bluetooth remote transmission of measurement data.

8. The deep learning based video stream continuous blood pressure measurement method according to claim 7, characterized in that, The camera video stream acquisition module includes: A camera selects a narrow-angle high-resolution non-distortion camera to collect 30-second continuous video signals in a 100*200 pixel area under the face eyelid; The CHROM extraction algorithm based on chroma analysis is used to extract 30-second length original face pulse wave signals synchronously with video acquisition.

9. The deep learning based video stream continuous blood pressure measurement method according to claim 7, wherein, The video pulse wave signal processing and reconstruction module includes: The original face pulse wave signals are segmented and data augmented to construct a data set for deep learning model; A CNN-BiLSTM deep learning model based on the combination of convolutional neural network and bidirectional long short-term memory network is constructed to complete quality reconstruction of the original rPPG signal, restore rPPG signal details and improve rPPG signal quality.

10. The deep learning based video stream continuous blood pressure measurement method according to claim 7, wherein, The blood pressure modeling module is constructed by the following methods: A CLC-BP blood pressure prediction model combining CNN and BiLSTM is constructed to realize feature extraction and time series comprehensive processing and complete video blood pressure estimation based on rPPG+ feature mixed input; The experimental data set required for model construction includes 34875 groups of data of 432 people in the public data set MIMIC III and 15 people in experimental data set 1 and 20 people in experimental data set 2, which are used as model training set, validation set and test set respectively, and a complete experimental data set scheme is constructed; The mixed input calibration blood pressure prediction model based on the combination of intrinsic signal PPG and PPG signal features improves blood pressure prediction accuracy.

Images