A cardiovascular disease prevention and diagnosis system based on visual intelligent recognition

Abstract

The present invention proposes a cardiovascular disease prevention and diagnosis system based on visual intelligent recognition. It automatically extracts a preset patient-specific observation region (ROI) by combining visible light images and infrared thermal images, and intelligently analyzes the distribution characteristics of the images in the region. It calculates the potential risk of cardiovascular disease based on the human body temperature distribution reflected by the infrared thermal image, thereby achieving the prevention and diagnosis of cardiovascular disease.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent medical devices, and in particular, relates to a cardiovascular disease prevention and diagnosis system based on visual intelligent recognition. Background Art

[0002] Hypertension is the primary risk factor for cardiovascular death such as stroke and myocardial infarction. The awareness and control rates of hypertension patients in my country are low, resulting in a high incidence of cardiovascular diseases and cardiovascular complications, which seriously threatens the health and life safety of residents. Data show that half of hypertensive patients do not know that they have hypertension, and only less than a quarter of patients can effectively control their blood pressure. It can be seen that blood pressure monitoring plays a key role in the prevention and diagnosis of cardiovascular diseases and urgently needs to be promoted and applied.

[0003] The poor awareness of patients to prevent abnormal blood pressure is a subjective factor for the high incidence of cardiovascular diseases in my country, while the complexity and specialization of blood pressure monitoring methods and equipment, combined with the weak overall medical knowledge of Chinese residents, are corresponding objective factors. Therefore, it is necessary to explore and develop a non-contact, easy-to-operate, intelligent blood pressure monitoring and cardiovascular disease prevention and diagnosis method. With the advancement of technology, the monitoring method based on the specific observation area (ROI) of infrared thermal images can effectively monitor the blood pressure levels of patients with cardiovascular diseases. By capturing and calculating the distribution characteristics of infrared thermal images of specific parts of the patient's face, an association with the potential risk of the disease can be established, thereby achieving prevention and diagnosis of cardiovascular diseases.

[0004] However, the existing technology relies more on experience to determine the ROI area that can indicate hypertension, that is, the doctor delineates it in the acquired image, which makes the entire system difficult to use at home and relies on experience, so the reliability is not high. There are also some technologies that use algorithms to delineate the ROI area, but these algorithms are not targeted at the ROI area indicating hypertension, and it is difficult to accurately delineate the area indicating hypertension, which makes further processing difficult.

[0005] In addition, for image areas indicating hypertension, the latest technology currently uses neural networks for processing, but there is no mature network structure, which leads to large errors in the final risk judgment and a heavy computational burden. It can only be performed using a server and cannot provide real-time feedback of prediction information on site. Summary of the invention

[0006] To solve one or more of the above problems, the present invention proposes a method for the prevention and diagnosis of cardiovascular diseases based on visual intelligent recognition. By combining visible light images and infrared thermal images, a preset patient-specific observation region (ROI) is automatically extracted, and the distribution characteristics of the image in the region are intelligently analyzed. The potential risk of cardiovascular disease is estimated based on the human body temperature distribution reflected by the infrared thermal image, thereby achieving the prevention and diagnosis of cardiovascular diseases.

[0007] A cardiovascular disease prevention and diagnosis system based on visual intelligent recognition includes an infrared camera, a visible light camera and a processor; the processor performs the following operations:

[0008] Step 1:

[0009]

[0010]

[0011] Among them, A α (u, v) is the visible light image of the patient’s face; and are the filter windows along the row and column directions of the image, respectively, i, j represent the position coordinates of any pixel relative to the center of the window; 2 k is the filter window size, 2≤k≤6; according to equation (7), the value Θ(u, v) is mapped to In each mapping set, we select the first κ pixels with the largest Γ(u, v), where κ = 16. α Select κ*8 pixels from (u, v) as the feature points of the visible light image, and the composition set is recorded as For any point in the above set According to the linear mapping model, the corresponding coordinates in the infrared thermal image are calculated from the coordinates in the visible light image. The corresponding pixel in the infrared image is denoted as f′. The neighborhood of a fixed size around f′ is selected and denoted as N. f′ , whereby the set of all neighborhoods is marked as ROI in the infrared image;

[0012] Step 2: Use 4 sets of independent channel convolution kernels to model the local features of the infrared thermal image ROI; when modeling, the excitation function used is:

[0013]

[0014] Perform global feature modeling, as shown below:

[0015]

[0016] Where Δ(θ, n, u′, v′) is the weighted value of the modeling results based on the local features of the above four groups of independent channels; b1 represents the global linear offset;

[0017] The risk of cardiovascular disease is modeled based on the global characteristics, expressed as follows:

[0018]

[0019] Where b2 represents the risk linear shift.

[0020] The processor is an on-site processor.

[0021] The processor is a remote server.

[0022] The acquisition device composed of an infrared camera and a visible light camera sends the acquired images to a remote server.

[0023] The transmission is via a wireless network.

[0024] The transmission is via a wired network.

[0025] The collection device is used in the patient's home.

[0026] Calibrate infrared and visible light cameras.

[0027] The calibration refers to determining the intrinsic parameters and extrinsic parameters of the infrared camera and the visible light camera.

[0028] The spatial mapping relationship between the visible light image and the infrared image is established according to the above calibration method.

[0029] The invention and technical effects of the present invention are as follows:

[0030] 1. The present invention can quickly and accurately obtain the feature points of the visible light image through the optimization algorithm, and then map these feature points to the infrared image through a mapping relationship, and determine the area indicating hypertension by the neighboring pixels of these points. In this way, the infrared image sub-area that is most indicative of hypertension can be obtained quickly, accurately and automatically, thus laying the foundation for further accurate prediction.

[0031] 2. By constructing a neural network model (network connection structure, activation function, hierarchical modeling, etc.) specifically for the infrared image area indicating hypertension, the area obtained by the first step algorithm can be accurately and quickly predicted through the network. This model structure is specially designed for the area obtained by the first step algorithm, and the combination of the two can be more accurate and faster. DETAILED DESCRIPTION

[0032] A cardiovascular disease prevention and diagnosis system based on visual intelligent recognition includes an infrared camera, a visible light camera and a processor.

[0033] The infrared camera is used to collect infrared images of patients, the visible light camera is used to collect visible light images of patients, and the processor is used to process the collected images to predict the patient's risk of cardiovascular disease.

[0034] It can be understood that the processor can be an on-site processor, for example, integrated with an infrared camera and a visible light camera to form a device; it can also be a remote server, that is, only a collection device consisting of an infrared camera and a visible light camera is set up on site, and the collection device sends the collected images to the remote server via a wireless network or a wired network. The server processes the data and prompts risks.

[0035] In this way, the system can be used in hospitals, where infrared cameras, visible light cameras, and processors are integrated into one device to provide services to patients; it can also be used in patients' homes, where infrared cameras and visible light cameras form a relatively compact collection device, which patients can operate according to prompts and send the collected data to the hospital's server for processing and risk warning. For example, the server can send the prediction results to the patient's mobile terminal.

[0036] The data processing method performed in the processor is as follows:

[0037] Step 1: Combine visible light images and infrared thermal images to extract specific observation areas (ROI) in infrared thermal images

[0038] The front and side images of the patient are collected using calibrated coaxial visible light and infrared thermal cameras. Feature points are extracted from the visible light image according to the features of the visible light image, and the position of each feature point in the infrared thermal image is calculated according to the calibration parameters, thereby generating the ROI in the infrared thermal image.

[0039] The specific observation region (ROI) refers to the area where a specific part of a human body is located in the image, and in this case, specifically refers to the area where a specific part of a human face is located in the image.

[0040] The calibration is the process of determining the intrinsic parameters and extrinsic parameters of a camera. In this case, the camera refers to a visible light camera and an infrared thermal camera.

[0041] The images described in the present invention refer to digital images recorded in a discrete numerical expression manner.

[0042] In the camera model, the three-dimensional coordinates of a point in the physical world are (X, Y, Z), and it is mapped to a point in the digital image with coordinates (u, v). This mapping can be described as a linear mapping model:

[0043]

[0044] Among them, M int 、M ext They are the intrinsic parameter matrix and the extrinsic parameter matrix respectively:

[0045]

[0046]

[0047] In the extrinsic parameter matrix They represent rotation parameters and translation parameters respectively; in the intrinsic parameter matrix, f is the focal length of the camera, s is the distortion factor, r is the ratio of the horizontal to vertical directions of the two-dimensional image, and (u0, v0) is the projection coordinates of the optical center of the camera on the image plane.

[0048] The intrinsic parameter matrix and extrinsic parameter matrix of the above linear mapping model can be solved by calculation methods such as the least squares method to complete the calibration.

[0049] According to the above calibration method, the spatial mapping relationship between visible light images and infrared thermal images is established. By capturing feature points with significant features in the visible light camera, irrelevant image information is reduced, thereby improving computational efficiency and performance.

[0050] During measurement, the patient's face is completely brought into the field of view of the visible light camera and is made to fill the visible area as much as possible. As an optimization method, the intrinsic parameters and extrinsic parameters of the visible light camera and the infrared thermal camera are controlled to be similar or identical in the camera hardware design, so that when the visible light camera completely captures the patient's face, the infrared thermal camera can also completely capture the patient's face.

[0051] A visible light digital image of the patient's face is obtained from the visible light camera, denoted as A α (u, v), u, v represents the coordinates of a point in the digital image, i.e., the pixel coordinates. The corresponding infrared thermal camera simultaneously captures an infrared thermal image, denoted as B α (u, v). α represents the time of shooting.

[0052] The local Haar filter is defined as follows:

[0053]

[0054]

[0055] In the above two formulas, W represents the window size of the filter. Respectively represent the filter window along the row and column directions of the image, i and j represent the position coordinates of any pixel in the window relative to the center of the window. By setting different W values, multiple Haar filters are obtained to form a filter bank. The following set is used:

[0056]

[0057] in

[0058]

[0059]

[0060] The filter group defined by formula (4) is used to perform convolution filtering on the visible light image:

[0061]

[0062] make:

[0063]

[0064]

[0065] Where tan -1 Represents the inverse tangent trigonometric function and normalizes the range to π is the circumference of a circle. According to equation (7), the value Θ(u, v) is mapped to The eight equally divided intervals are:

[0066] According to the value of Θ(u, v), all pixels are mapped to the eight intervals divided above, and the first κ pixels with the largest Γ(u, v) are selected in each mapping set, and κ=16 is preferably selected. α Select κ*8 pixels from (u, v) as the feature points of the visible light image, and the composition set is recorded as Through the above selection, the number of feature points is greatly reduced while ensuring that almost no information is lost, thereby reducing the number of ROI areas in the next step, reducing the burden on the network in the second step, and improving computing efficiency.

[0067] The feature points in the visible light image are extracted by the method of formula (2-7). Since the facial texture in the visible light image is richer, its features have better distribution differences. Combined with the Haar filter group that is highly similar to the facial features, it can filter out the hair, background and other information in the image that are not related to the face, so that the specific observation area (ROI) is concentrated on the facial skin area with rich blood vessels, and it is possible to infer blood pressure and disease conditions based on image analysis.

[0068] For any point in the above set According to the linear mapping model (1), the corresponding coordinates in the infrared thermal image can be calculated according to its coordinates in the visible light image, and the corresponding pixel in the infrared thermal image is denoted as f′. For, select the surrounding 16*16 size neighborhood and denote it as N f′ , the set of all the above neighborhoods is marked as ROI in the infrared thermal image. It is expressed as follows:

[0069]

[0070] In the above formula, ∪{N f′} represents all neighborhoods N f′ The union of It means that the pixel f′ in the infrared thermal image corresponds to the pixel f in the visible light image. By locating several ROIs in the infrared thermal image through the spatial mapping relationship between the visible light image and the infrared thermal image, the facial area that reflects the blood pressure change can be effectively captured, thereby realizing image-based disease risk identification.

[0071] For the front and side images, the above method is used to obtain their respective ROIs.

[0072] Step 2: Automatically assess the potential risk of cardiovascular disease based on the image features of a specific region of interest (ROI) in the infrared thermal image

[0073] A mapping model between image features and disease risks is established, the mapping model is learned based on training data, and the learned model is used to automatically assess risks based on input image features to achieve preventive diagnosis of cardiovascular diseases.

[0074] According to step 1, the ROI includes 2*κ*8 16*16 neighborhoods, which can be expressed as:

[0075] ROI(n,u′,v′)

[0076] Where n represents the neighborhood, and its value range is 1≤n≤2*κ*8, u′, v′ are the pixel coordinates in the neighborhood, 1≤u′, v′≤16.

[0077] Four groups of independent channel convolution kernels are proposed to model the local features of infrared thermal image ROI, which are expressed as follows:

[0078]

[0079]

[0080]

[0081]

[0082] Where ω1, ω2, ω3, and ω4 represent four independent channel convolution kernels, 1≤i′, j′≤5 represent the relative position coordinates of the convolution kernel window, and 1≤η≤4 represent the relative order coordinates of the convolution kernel window. b0 represents the linear offset of the convolution window. The σ function is defined as follows:

[0083]

[0084] The above function σ is a nonlinear function, and its purpose is to model the local nonlinear characteristics. x Represents the natural exponential function, lnx represents the natural logarithmic function. Since the grayscale perception of the image by the human eye conforms to the exponential distribution characteristics, this method helps to improve the performance of local feature extraction and makes it closer to the effect of manual diagnosis.

[0085] Furthermore, according to the results of local feature modeling, global feature modeling is performed, which is expressed as follows:

[0086]

[0087] in,

[0088] Δ(θ,n,u′,v′)=δ(θ,n,u′,v′)*V1(n,u′,v′)+δ(θ,n,u′,v′)*V2(n,u′,v′)+δ(θ,n,u′,v′)*V3(n,u′,v′)+δ(θ,n,u′,v′)*V4(n,u′,v′)…(15)

[0089] In equations (14) and (15), δ represents the global weight, δ(θ, n, u′, v′) represents the weight component between the pixel located by n, u′, v′ and the θth component of the global feature in the global weight, b1 represents the global linear offset, and σ is the function defined in (13).

[0090] Furthermore, the risk of cardiovascular disease is modeled based on the global characteristics, as follows:

[0091]

[0092] Where χ represents the risk weight, each of which corresponds to the corresponding component of the global feature. b2 represents the risk linear shift. σ is the function defined in (13).

[0093] Formulas (9)-(16) constitute the mapping model between image features and cardiovascular disease risk. The model needs to be learned first to determine the parameters ω1, ω2, ω3, ω4 of the convolution kernel window in local feature modeling, the parameter δ of global feature modeling, the parameter χ of risk modeling, and the linear offsets b0, b1, b2.

[0094] According to the method in step 1, several groups of facial images of patients with cardiovascular diseases and several groups of images of subjects without cardiovascular diseases are collected as training samples for learning. The two types of samples are marked as 1 and 0 respectively. The images of the training samples are used to generate ROIs according to the method described in step 1 and used as the input of the model to equations (9)-(12). The corresponding model outputs E risk ; The sample label 0 or 1 is used as the output of the model and as the reference value of the output of formula (16), denoted as

[0095] Define the cost function:

[0096]

[0097] Wherein, τ is a linear proportional control parameter, which is set according to the average brightness of the training data image and is in the range of 0.5<τ<1. If the average brightness of the training data image is brighter, τ can take a larger value. ξ is an offset control parameter, which can reduce the probability of the model entering overfitting, and ξ=0.03 ​​is preferred. The cost function (17) is iteratively calculated using the backpropagation method until convergence, and the above parameters are recorded at convergence to complete the learning.

[0098] The learned models (9)-(16) are used to automatically assess the risk based on the input image, thereby achieving preventive diagnosis of cardiovascular disease. When the output value of the model for the input image is close to 1, it is considered that the corresponding patient has a high risk of cardiovascular disease and should be further examined.

[0099] Table 1 shows the experimental data of the accuracy of cardiovascular disease risk prediction by the method of the present invention and the existing method. The prediction accuracy refers to the probability of further confirming the diagnosis of cardiovascular disease after the method of the present invention predicts the cardiovascular risk. The data show that the method of the present invention has a high accuracy in predicting cardiovascular disease risk in all age groups, and has a higher accuracy in predicting cardiovascular disease risk in elderly patients, which helps to help the elderly, a group with a high incidence of cardiovascular disease, to discover disease risks in advance and take countermeasures.

[0100] Table 1

[0101] Age of subjects The prediction accuracy of the present invention Prediction accuracy of existing algorithms 40-49 71.2％ 53.1％ 50-59 77.8％ 54.4％ 60-69 86.5％ 67.3％ 70-79 88.5％ 64.9％

[0102] Table 2 shows the experimental data of the speed of cardiovascular disease risk prediction by the method of the present invention and the existing method. As can be seen from the table, the method of the present invention has a significant advantage in speed.

[0103] Table 2

[0104] Age of subjects The present invention predicts time Existing algorithms predict time 40-49 5 seconds 25 seconds 50-59 5 seconds 23 seconds 60-69 4 seconds 23 seconds 70-79 6 seconds 24 seconds

[0105] The above embodiments are only limited examples. Due to the length of the article, they cannot be listed exhaustively. Therefore, they are not intended to limit the scope of protection of the rights. All technical solutions similar to the above products and methods are within the scope of protection of this application.

Claims

1. A cardiovascular disease prevention and diagnosis system based on visual intelligent recognition, characterized by: It includes an infrared camera, a visible light camera and a processor; the processor performs the following operations: Step 1: The local Haar filter is expressed as follows: ； ； W represents the window size of the filter, , Respectively represent the filter windows along the row and column directions of the image; by setting different W values, multiple Haar filters are obtained to form a filter bank; the following set is used: ； in ； ； The filter group defined by formula (4) is used to perform convolution filtering on the visible light image: ； ； ； in, A visible light image of the patient's face; and are the filter windows along the row and column directions of the image, respectively. Indicates the position coordinates of any pixel relative to the center of the window; is the filter window size, ; According to the value of formula (7) Map to and take the eight equally divided intervals in each mapping set. The largest front pixels, of which ; So far in the original image Select pixels, as the feature points of the visible light image, constitute a set denoted as ; For any point in the above set According to the linear mapping model, the corresponding coordinates in the infrared thermal image are calculated by the coordinates in the visible light image. The corresponding pixels in the infrared image are recorded as ; Select The surrounding fixed-size neighborhood is denoted by , whereby the set of all neighborhoods is marked as ROI in the infrared image; Step 2: Use 4 sets of independent channel convolution kernels to model the local features of the infrared thermal image ROI; when modeling, the excitation function used is: ； Perform global feature modeling, as shown below: ； in ； In formula (14) and (15), In the global weight, The localized pixels and the global features The weight component between the components; Represents the global linear offset; It represents the result of local feature modeling of infrared thermal image ROI by using 4 groups of independent channel convolution kernels; represents the neighborhood, is the pixel coordinate in the neighborhood; The risk of cardiovascular disease is modeled based on the global characteristics, expressed as follows: ； in, represents the risk weight, Represents a linear shift in risk.

2. A cardiovascular disease prevention and diagnosis system based on visual intelligent recognition as claimed in claim 1, characterized in that: The processor is an on-site processor.

3. A cardiovascular disease prevention and diagnosis system based on visual intelligent recognition as claimed in claim 1, characterized in that: The processor is a remote server.

4. A cardiovascular disease prevention and diagnosis system based on visual intelligent recognition as claimed in claim 3, characterized in that: The acquisition device composed of an infrared camera and a visible light camera sends the acquired images to a remote server.

5. A cardiovascular disease prevention and diagnosis system based on visual intelligent recognition as claimed in claim 4, characterized in that: The transmission is via a wireless network.

6. A cardiovascular disease prevention and diagnosis system based on visual intelligent recognition as claimed in claim 4, characterized in that: The transmission is via a wired network.

7. A cardiovascular disease prevention and diagnosis system based on visual intelligent recognition as claimed in any one of claims 4 to 6, characterized in that: The collection device is used in the patient's home.

8. A cardiovascular disease prevention and diagnosis system based on visual intelligent recognition as claimed in claim 1, characterized in that: Calibrate infrared and visible light cameras.

9. A cardiovascular disease prevention and diagnosis system based on visual intelligent recognition as claimed in claim 8, characterized in that: The calibration refers to determining the intrinsic parameters and extrinsic parameters of the infrared camera and the visible light camera.

10. A cardiovascular disease prevention and diagnosis system based on visual intelligent recognition as claimed in claim 9, characterized in that: The spatial mapping relationship between the visible light image and the infrared image is established according to the above calibration method.

Images