Non-contact blood pressure detection system based on computer vision and deep learning
Through a non-contact blood pressure detection system based on computer vision and deep learning, the convolutional neural network model is used to extract the reflected light signals of facial blood flow, which solves the problem of large measurement error of oscillometric blood pressure, and achieves high-precision and convenient blood pressure Monitoring and management are suitable for different scenarios and age groups, reducing hardware dependence and costs.
Patent Information
- Application Number
- CN202211374210.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-04
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-11-04
AI Technical Summary
The existing oscilloscope blood sphygmomanometer measurement methods have large errors, especially inadequate accuracy for unhealthy people, which cannot truly reflect blood pressure values. It relies on hypotheses and statistical methods, resulting in large measurement errors among some people.
Using a contactless blood pressure detection system based on computer vision and deep learning, facial video is collected through smartphone cameras, and the convolutional neural network model is used to extract facial blood flow reflected light signals, calculate systolic blood pressure and diastolic pressure, and output it to APP or WeChat. Program to provide high-precision blood pressure measurement.
Get rid of the dependence on hardware equipment, improve the convenience and accuracy of blood pressure monitoring, reduce user thresholds and costs, provide convenient recording, transmission and management of blood pressure data, and can model different scenarios and age groups in a timely manner. Remind blood pressure outliers.
Smart Images

Figure CN115736863B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of blood pressure measurement, and in particular to a non-contact blood pressure detection system based on computer vision and deep learning. Background Art
[0002] Traditional blood pressure measurement methods are divided into invasive blood pressure measurement and non-invasive blood pressure measurement. Invasive blood pressure measurement requires a catheter to pierce the blood vessels and connect the pressure sensor directly to the blood vessels. It is considered the most accurate blood pressure measurement method. This method of measuring blood pressure is generally only used on critically ill patients. In addition to invasive measurement, most other measurements are non-invasive.
[0003] Non-invasive measurement can be divided into two types according to whether a cuff is used: cuff measurement and cuffless measurement. Generally speaking, cuff measurement is more accurate than cuffless measurement. General sphygmomanometers use cuff measurement, while some smart watches and bracelets use cuffless measurement. Cuff measurement can be roughly divided into auscultation measurement and oscillometric measurement according to the measurement method. The oscillometric method is used by general electronic (automatic) sphygmomanometers, also known as the oscillation method. Each manufacturer uses its own oscillometric algorithm to infer the blood pressure value, and the accuracy is related to the subject. The principle is briefly described as follows: First, tie the cuff to the arm, automatically inflate the cuff, stop pressurizing when a certain pressure (generally 30 to 50 mmHg higher than the systolic pressure), and start to deflate. When the air pressure reaches a certain level, blood flow can pass through the blood vessels, and there is a certain oscillation wave. The oscillation wave is transmitted through the trachea to the pressure sensor, and the pressure sensor can detect the pressure and fluctuations in the measured cuff in real time. Gradually deflate, the oscillation wave becomes larger and larger. As the cuff and arm are looser, the pressure and fluctuation detected by the pressure sensor become smaller and smaller. Oscillometric blood pressure monitors are available in upper arm and wrist types. Generally, upper arm sphygmomanometers with oscillometric blood pressure monitors are more accurate than wrist sphygmomanometers with oscillometric blood pressure monitors. Oscillometric blood pressure monitors are a very popular type of blood pressure monitor, and 80% of families with blood pressure monitors use oscillometric blood pressure monitors. It is advanced only in terms of electronic blood pressure measurement, but the measurement technology is relatively poor.
[0004] Since the values measured by the oscillometric method are floating and changing, there is no regularity, and there may be several maximum values, the data obtained by calculation cannot truly reflect the value of blood pressure. Therefore, the oscillometric method of measuring blood pressure is also based on the premise of assumptions; first, the waveform of the subject is standardized; second, the formula and coefficient (0.45 and 0.75) used to calculate the blood pressure value are the same as the subject. Healthy people can generally meet the coefficient, and unhealthy people often do not meet it, but only unhealthy people need a sphygmomanometer more. The judgment basis of the oscillometric method is given by statistical methods through a large number of population experiments. Therefore, this measurement method will inevitably cause measurement errors in some people, and sometimes the error can reach tens of millimeters of mercury. According to statistics, the oscillometric method is only applicable to 70% of the population, and the error is large for the other 30% of the population; for the same person, the oscillometric method is sometimes accurate and sometimes has a large error; therefore, it is necessary to design a non-contact blood pressure detection system based on computer vision and deep learning to solve the above problems. Summary of the invention
[0005] The present invention provides a non-contact blood pressure detection system based on computer vision and deep learning, which solves the above-mentioned technical problems.
[0006] In order to solve the above technical problems, the present invention provides a non-contact blood pressure detection system based on computer vision and deep learning, comprising the following steps:
[0007] S1, collect 20 seconds facial video from the user through a smartphone or other terminal camera, and perform face recognition, environment detection, facial analysis and signal processing through C++ scripts;
[0008] S2, select part of the face area in each video frame as Region of Interest, and upload the analyzed point information to the cloud;
[0009] S3. Using a convolutional neural network model, the facial blood flow reflection light signal extracted from the video is simulated into a physiological characteristic signal of a medical hardware detection device approved by a medical device regulatory agency;
[0010] S4. Use convolutional neural network models to calculate key vital sign results such as systolic and diastolic blood pressure;
[0011] S5. The calculation results are output to APP, WeChat applet or other front-end software for doctors or ordinary users to view, use and share.
[0012] Furthermore, the video acquisition in S1 includes the steps of:
[0013] S101, smartphone or other terminal camera recording video;
[0014] S102, analyzing the video through a C++ script;
[0015] S103, uploading pixel information of key points to the cloud.
[0016] Furthermore, the partial face region selected in S2 is a plurality of facial parts, including but not limited to a forehead, a cheek and a chin.
[0017] Furthermore, the training of the convolutional neural network model in S3 is achieved based on a large amount of facial video collection and actual blood pressure detection results collected synchronously with the video.
[0018] Furthermore, the data used in the modeling are extracted by the testers in a general living environment through cameras of smartphones with different operating systems such as iOS and Android and different brands. At the same time, the testers collect and record key vital signs data such as systolic and diastolic blood pressure through blood pressure monitoring equipment approved by medical device regulatory agencies.
[0019] Furthermore, S4 is modeled using a 3D convolutional neural network model.
[0020] Furthermore, the calculation steps of the 3D convolutional neural network model adopted in S4 are as follows:
[0021] S401, first of all, k d Continuous frames form a 3D image sequence, and then convolution calculation is performed in the image sequence;
[0022] S402, downsampling is achieved by reducing the feature map output by the convolution layer, and at the same time, the features in the receptive field are screened to extract the most representative features in the region and retain the most important information in the feature map;
[0023] S403, then nonlinear factors are introduced into neurons, and nonlinear transformation is performed on input information, so that the neural network can arbitrarily approximate any nonlinear function, and then the transformed output information is transmitted as input information to the next layer of neurons;
[0024] S404, summarizing the features extracted by the convolutional neural network, and mapping the multi-dimensional features into the output of the regression result.
[0025] Furthermore, the loss function used in this step includes the Entropy equation, a confusion matrix to determine the applicability of the model, and an L1 Norm equation.
[0026] Compared with related technologies, the non-contact blood pressure detection system based on computer vision and deep learning provided by the present invention has the following beneficial effects:
[0027] Compared with traditional sphygmomanometer technology, the advantage of the system provided by the present invention is that it gets rid of the traditional blood pressure monitoring's dependence on hardware equipment, greatly facilitates blood pressure monitoring and the recording, transmission and management of blood pressure data, improves the user experience of blood pressure measurement, and reduces the user's usage threshold and usage cost.
[0028] The present invention provides users with high-precision blood pressure measurement values through artificial intelligence algorithms, adopts cloud storage, provides users with convenient data recording, transmission and management, users can view the changing trend of blood pressure, and provides convenient and accurate records for doctors, medical institutions or other institutions that need user blood pressure data, thereby reducing the cost of data collection and application.
[0029] The present invention provides that the system can model different scenes, including resting scenes, post-exercise scenes, etc.; and can model different age groups specifically, including the elderly, teenagers, etc.
[0030] The present invention provides that, while meeting the existing demand for blood pressure detection, the present invention can help users get rid of their dependence on hardware devices and help users record blood pressure historical data. When abnormal blood pressure values appear, users can be reminded in time through APP, mini-programs or other front-end software to decide whether to seek medical help.
[0031] The present invention provides that with the rapid iteration and upgrade of smartphone camera hardware and artificial intelligence algorithms, the accuracy of the blood pressure detection method based on computer vision and deep learning will continue to be optimized. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 A schematic diagram of a non-contact blood pressure detection system based on computer vision and deep learning according to the present invention;
[0033] Figure 2 This is a schematic diagram of the CNN structure of the present invention.
[0034] Figure 3 Add data transmission diagram DETAILED DESCRIPTION
[0035] Embodiment 1, by Figure 1-3 A non-contact blood pressure detection system based on computer vision and deep learning is given, including the following steps:
[0036] S1, collect 20 seconds facial video from the user through a smartphone or other terminal camera, and perform face recognition, environment detection, facial analysis and signal processing through C++ scripts;
[0037] S2, select part of the face area in each video frame as Region of Interest, and upload the analyzed point information to the cloud;
[0038] S3. Using a convolutional neural network model, the facial blood flow reflection light signal extracted from the video is simulated into a physiological characteristic signal of a medical hardware detection device approved by a medical device regulatory agency;
[0039] S4. Use convolutional neural network models to calculate key vital sign results such as systolic and diastolic blood pressure;
[0040] S5. The calculation results are output to APP, WeChat applet or other front-end software for doctors or ordinary users to view, use and share.
[0041] The specific implementation is as follows:
[0042] The first step is to collect a 20-second facial video from the user through a smartphone or other terminal camera, and perform face recognition, environment detection, facial analysis and signal processing through a C++ script.
[0043] Among them, S101, a smartphone or other terminal camera records the video; S102, the video is analyzed by a C++ script; S103, the pixel information of key points is uploaded to the cloud.
[0044] Among them, the C++ script includes denoising and signal processing such as face recognition, environment detection, facial analysis, and brightness analysis. Face recognition can select multiple face parts for vital sign measurement, and detect faces and facial feature analysis in real time through the OpenCV deep neural network (DNN) face detector. Based on the Single Shot Detector (SSD), face detection can locate the bounding box of the entire face in the image, which is used to detect faces frame by frame from the video;
[0045] In the second step, after the face is detected, part of the face area is selected as the RoI in each video frame. Each RoI has three RGB channels: red, green, and blue. The raw BVP signal of each frame is represented by the average pixel value of the green channel because it contains the strongest plethysmographic signal. Therefore, the RoI generates a raw BVP signal, which indicates the frame number.
[0046] Among them, the partial face area is selected as multiple facial parts, including but not limited to the forehead, cheeks and chin. The established algorithm is applied to identify the best ROI from the user's face to maximize the real signal; De-noising filter: After obtaining the original BVP signal, a denoising filter is applied to remove large jumps and steps caused by motion noise such as head movement and handshake. We calculate the absolute difference between consecutive signals and remove the signals with differences above a predefined threshold to reduce noise.
[0047] In the third step, a convolutional neural network model is used to simulate the facial blood flow reflection light signals extracted from the video into physiological characteristic signals of medical hardware detection equipment approved by medical device regulatory agencies.
[0048] The training of the convolutional neural network model in this simulation step is based on a large amount of facial video collection and actual blood pressure test results collected synchronously with the video. The data used in the modeling is extracted by the tester in a general living environment through the camera of a smartphone with different operating systems such as iOS and Android and different brands. At the same time, the tester collects and records key vital signs such as systolic and diastolic blood pressure through a blood pressure monitoring device approved by the medical device regulatory agency;
[0049] The smartphones used include those with different operating systems such as iOS and Android and different brands. The living environment is extracted without restrictions on light (natural light, artificial light), weather, and recent behavior (after exercise, after sitting for a long time, after work, etc.); the loss function of this step is cosine similarity. The purpose of the model is to maximize the similarity of facial blood flow reflection light signals and fit the signals that can only be obtained by contact method in a non-contact way.
[0050] In the fourth step, a convolutional neural network model is used to calculate key vital sign results such as systolic and diastolic blood pressure.
[0051] Different from the common 2D convolution method, we creatively use 3D convolution to model;
[0052] When processing 3D images or videos, the network input has an additional dimension, and the input changes from (c, height, width) to (c, depth, height, width), where c is the number of channels and depth is the width of the input data. Therefore, when processing this data, the convolution needs to be transformed accordingly, from 2D convolution to 3D convolution.
[0053] 3D convolution has one more dimension in structure than 2D convolution. The size of 2D convolution can be expressed as k h ×k w , and the size of the 3D convolution can be expressed as k h ×k w ×k d The specific calculation method of 3D convolution is similar to that of 2D convolution, that is, each time a slide is performed, a multiplication and addition operation is performed with an image of c channels and size (depth, height, width) to obtain a value in the output feature map. In the video understanding task, k d It represents the time dimension, that is, the number of consecutive frames processed by each 3D convolution kernel.
[0054] In the 3D convolution calculation in the field of video understanding, k d k consecutive frames form a 3D image sequence, and then convolution calculation is performed on the image sequence. The 3D convolution kernel will be d Slide on consecutive frames, and slide k each time d The elements in the corresponding positions in the consecutive frames are multiplied and added with the parameters in the convolution kernel, and finally a value in the output feature map is obtained. Our video data is 3D, for example, the face we want to analyze is 3D.
[0055] The model uses convolution to extract features from the input image. The convolution calculation range is performed within the spatial neighborhood of the pixel point, so the spatial information of the input image can be used. The convolution kernel itself is independent of the size of the input image, and it represents the extraction of a certain feature pattern within the spatial neighborhood. Then, downsampling is achieved by reducing the feature map output by the convolution layer. At the same time, the features in the receptive field are screened, the most representative features in the region are extracted, and the most important information in the feature map is retained. Subsequently, nonlinear factors are introduced to the neurons, and nonlinear transformation is performed on the input information, so that the neural network can arbitrarily approximate any nonlinear function, and then the transformed output information is passed as input information to the next layer of neurons. Finally, the features extracted by the convolutional neural network are summarized, and the multi-dimensional features are mapped to the output of the regression result.
[0056] The loss function of this step includes the Entropy equation, the Confusion Matrix to determine the applicability of the model, and the L1 Norm equation.
[0057] Loss Function
[0058] Cosine Similarity
[0059] Assume x1=(x 1,1 , x 1,2 , …, x 1,n ), x2=(x 2,1 , x 2,2 , …, x 2,n ), na‰¥1 is an integer,
[0060]
[0061] The dot product and L2 norm are defined as follows:
[0062]
[0063] Entropy
[0064] C is the number of categories, x is the predicted column value, y is the predicted category accuracy value, and N is the number of samples
[0065] l(x, y) = L = {l1, ..., l N} T ,
[0066] When the reduction method is "mean".
[0067] Confusion Matrix
[0068]
[0069]
[0070] True Positive (TP): True class, the actual class of the sample is positive, and the model predicts the class is also positive
[0071] False Negative (FN): False negative class, the true category of the sample is positive, and the model predicts the category is also negative
[0072] False Positive (FP): False positive class, the true class of the sample is negative, and the model predicts the class is also positive
[0073] True Negative (TN): True negative class, the actual class of the sample is negative, and the model predicts the class is also negative
[0074] L1 Norm
[0075] Assume x = (x1, x2, ..., x n ), y=(y1,y2,…,y n ), na‰¥1 is an integer
[0076] l(x, y) = L = {l1, ..., l N} T , l n =|x n -y n |
[0077] l(x, y) = mean(L), when the reduction method is "mean".
[0078] The fifth step is to output the calculation results to the APP, WeChat applet or other front-end software for doctors or ordinary users to view, use and share.
[0079] The measurement method is based on the fluid model, that is, it is assumed that the blood flowing in the blood vessels flows like water in the water pipes. When the limiting conditions such as the friction of the pipe wall are known or relatively fixed, as long as the blood flow rate is known, the pressure on the side wall can be calculated. In general, the absorption of light by the human body consists of three main parts: arterial blood, venous blood, bones and other tissues. The changes in the absorption of light by venous blood, bones and other tissues can be ignored at a fixed time point. As the heart contracts and relaxes, the flow rate of arterial blood changes, so the changes in the absorption of light by the skin are directly related to the changes in the flow rate of arterial blood. Based on the fluid model and the characteristics of the skin's absorption of light, we conclude that the changes in the absorption of light by the skin are directly related to the changes in the flow rate of arterial blood. Therefore, as long as we measure the changes in the absorption of light by the skin, we can deduce the changes in the flow rate of arterial blood, and then we can get the blood pressure value at a certain time point. So how do we monitor the changes in the absorption of light by the skin? We use the built-in optical sensor of the bracelet to continuously collect signals from the user's skin in a certain period of time, and process the signals of continuous images in a certain period of time to derive the changes in the absorption of light by the skin at that time point. Therefore, through the changes in the skin's absorption of light, we can obtain the changes in the flow rate (ν) of arterial blood, and then calculate the changes in blood pressure (P) through the fluid model.
[0080] In summary, compared with the existing traditional blood pressure detection technology, the advantage of this system is that it gets rid of the dependence of traditional blood pressure monitoring on hardware equipment, greatly facilitates blood pressure monitoring and the recording, transmission and management of blood pressure data, improves the user experience of blood pressure measurement, and reduces the user's usage threshold and usage cost;
[0081] Through artificial intelligence algorithms, it provides users with high-precision blood pressure measurements and adopts cloud storage to facilitate data recording, transmission and management. Users can view the changing trends of blood pressure and provide convenient and accurate records for doctors, medical institutions or other institutions that need user blood pressure data, reducing the cost of data collection and application.
[0082] This system can model different scenarios, including resting scenarios, post-exercise scenarios, etc.; it can model different age groups, including the elderly, teenagers, etc.
[0083] While meeting the existing demand for blood pressure detection, the present invention can help users get rid of their dependence on hardware devices and help users record blood pressure historical data. When abnormal blood pressure values appear, users can be reminded in time through APP, mini-programs or other front-end software to decide whether to seek medical help.
[0084] The present invention provides that with the rapid iteration and upgrade of smartphone camera hardware and artificial intelligence algorithms, the accuracy of the blood pressure detection method based on computer vision and deep learning will continue to be optimized.
Claims
1. Non-contact blood pressure detection system based on computer vision and deep learning, It is characterized in that The following steps are involved: S1, collect 20 seconds facial video from the user through a smartphone or other terminal camera, and perform face recognition, environment detection, facial analysis and signal processing through C++ scripts; Among them, face recognition can select multiple face parts for vital sign measurement, and detect faces and facial feature analysis in real time through OpenCV deep neural network (DNN) face detector. Based on Single Shot Detector (SSD), face detection can locate the bounding box of the entire face in the image, which is used to detect faces frame by frame from the video; S2, select part of the face area in each video frame as the Region of Interest, and upload the analyzed point information to the cloud; Among them, each RoI has three RGB channels: red, green, and blue; the raw BVP signal of each frame is represented by the average pixel value of the green channel, which contains the strongest plethysmographic signal; therefore, the RoI generates a raw BVP signal, which indicates the frame number; Among them, the partial face area is selected as multiple facial parts, including but not limited to the forehead, cheeks and chin; the established algorithm is applied to identify the best return on investment from the user's face to maximize the real signal; denoising filter: after obtaining the original BVP signal, a denoising filter is applied to remove large jumps and steps caused by head movement and handshake movement noise; we calculate the absolute difference between consecutive signals and remove the signals with differences above a predefined threshold to reduce noise; S3, using a convolutional neural network model to simulate facial blood flow reflection light signals extracted from the video into physiological characteristic signals of medical hardware detection equipment; The training of the convolutional neural network model is based on a large amount of facial video collection and actual blood pressure detection results collected synchronously with the video; The data used for modeling was extracted by testers using smartphone cameras of different brands. At the same time, the testers collected and recorded systolic and diastolic blood pressure through blood pressure monitoring equipment. The smartphones used include iOS and Android operating systems. There are no restrictions on the light, weather, and recent behavior of the living environment. The loss function of this step is cosine similarity. The purpose of the model is to maximize the similarity of facial blood flow reflection light signals and fit the signals that can only be obtained by contact methods in a non-contact way. S4, using convolutional neural network model to calculate systolic and diastolic blood pressure; Among them, a 3D convolutional neural network model is adopted, and the calculation steps of the 3D convolutional neural network model are as follows: S401, first Continuous frames form a 3D image sequence, and then convolution calculation is performed in the image sequence; S402, downsampling is achieved by reducing the feature map output by the convolution layer, and at the same time, the features in the receptive field are screened to extract the most representative features in the region and retain the most important information in the feature map; S403, then nonlinear factors are introduced into neurons, and nonlinear transformation is performed on input information, so that the neural network can arbitrarily approximate any nonlinear function, and then the transformed output information is transmitted as input information to the next layer of neurons; S404, summarizing the features extracted by the convolutional neural network, and mapping the multi-dimensional features into the output of the regression result; The loss function used in this step includes the Entropy equation, the confusion matrix to judge the applicability of the model, and the L1 norm equation; S5. The calculation results are output to APP, WeChat applet or other front-end software for doctors or ordinary users to view, use and share; The measurement method is based on the fluid model, which assumes that the blood flowing in the blood vessels is like water flowing in the water pipes. When the limiting conditions are known or relatively fixed, the pressure on the side wall can be calculated as long as the blood flow rate is known. The absorption of light by the human body is mainly composed of three parts: arterial blood, venous blood, bones and other tissues. The changes in the absorption of light by venous blood, bones and other tissues can be ignored at a fixed time point. As the heart contracts and relaxes, the flow rate of arterial blood will change, so the changes in the absorption of light by the skin are directly related to the changes in the flow rate of arterial blood. Based on the fluid model and Based on the characteristics of skin's absorption of light, it is concluded that the change in skin's absorption of light is directly related to the change in arterial blood flow velocity. Therefore, as long as the change in skin's absorption of light is measured, the change in arterial blood flow velocity can be deduced, and then the blood pressure value at a certain point in time can be obtained; the built-in optical sensor of the bracelet collects continuous signals from the user's skin in a certain period of time, and by processing the continuous images of a certain period of time, the change in skin's absorption of light at that point in time can be obtained; therefore, the change in arterial blood flow velocity (ν) can be obtained through the change in skin's absorption of light, and then the change in blood pressure (P) can be calculated through the fluid model.
2. The non-contact blood pressure detection system based on computer vision and deep learning according to claim 1, It is characterized in that The video acquisition in S1 includes the following steps: S101, smartphone or other terminal camera recording video; S102, analyzing the video through C++ script; S103, uploading pixel information of key points to the cloud.
3. The non-contact blood pressure detection system based on computer vision and deep learning according to claim 1, It is characterized in that In S2, the partial face region is selected as a plurality of facial parts, including a forehead, a cheek and a chin.
