Coronary artery CT angiography image plaque identification method based on deep learning

Abstract

The invention provides a method for identifying plaques in a coronary artery CT angiography image based on deep learning. The method comprises the following steps: firstly, calculating a coronary artery center line of a coronary artery CT angiography image, performing multi-angle curved surface reconstruction of the image based on the center line, and then performing fusion processing to realize image quality improvement; and then, constructing a coronary heart disease plaque detection network based on an attention mechanism, training the network by using the fused image to obtain a more effective detection network, and detecting coronary artery plaques in the image by using the trained detection network. According to the method, the problem of low accuracy caused by angle limitation in a single curved surface reconstruction image can be effectively solved, and the accuracy of plaque recognition in the coronary artery CT angiography image is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing and computer-aided diagnosis, and in particular relates to a method for identifying plaques in coronary CT angiography images based on deep learning. Background Art

[0002] CT angiography (CTA) is a medical imaging technique that uses CT scanning and contrast agents to obtain three-dimensional images of blood vessels. CTA is widely used to detect and diagnose coronary artery disease. It can show the anatomical structure and stenosis of the coronary arteries and is an important non-invasive diagnostic tool. The original CTA image is usually presented in the form of a transverse profile of the human body, providing detailed cross-sectional images of the coronary arteries and surrounding tissues.

[0003] Due to the complex anatomical structure of the coronary arteries and the differences in angiography angles at different locations, it is difficult to directly observe and analyze the coronary artery branches, lesion severity, and morphological characteristics in the original CTA images. These original images are not post-processed and contain a large amount of anatomical noise and overlapping information. Direct analysis of these images is not only difficult but also prone to misdiagnosis or missed diagnosis. Therefore, how to extract lesion information with clinical diagnostic value from the original CTA images is an urgent problem to be solved. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a method for identifying plaques in coronary CT angiography images based on deep learning. First, the centerline of the coronary artery of the coronary CT angiography image is calculated, and multi-angle surface reconstruction of the image is performed based on the centerline, and then fusion processing is performed to improve the image quality; then, a coronary heart disease plaque detection network based on the attention mechanism is constructed, and the network is trained using the fused image to obtain a more effective detection network, and the trained detection network is used to detect coronary artery plaques in the image. The present invention can effectively solve the problem of low accuracy caused by angle limitation in single surface reconstruction images, and significantly improve the accuracy of plaque recognition in coronary CT angiography images.

[0005] A method for identifying plaques in coronary CT angiography images based on deep learning, characterized by the following steps:

[0006] Step 1, calculating the coronary artery centerline: first, the input coronary artery CT angiography image is enhanced using the coronary artery three-dimensional model enhancement algorithm, and then the coronary artery centerline is calculated using a three-dimensional thinning method;

[0007] Step 2, multi-angle surface reconstruction: First, calculate the coordinate value of each point on the center line of the coronary artery in the TNB coordinate system, and use the Frenet formula to calculate the curvature and torsion of the center line of the coronary artery in three-dimensional space. Then, perform super-resolution processing on the curvature and torsion, reconstruct the coronary artery from multiple angles, and generate surface reconstructed images at multiple angles;

[0008] Step 3, multi-angle surface reconstruction image fusion: the surface reconstruction images at various angles are fused using the previous fusion method to obtain a number of fused images;

[0009] Step 4: Construct a coronary heart disease plaque detection network based on the attention mechanism: Based on the Faster R-CNN network model, the ResNet50 convolutional neural network with the attention mechanism is used as the backbone network to extract the feature map of the image. Then, the feature map is subjected to target detection through the RPN network to obtain several target prediction frames. Then, the prediction frames are resized through the RoI pooling layer to obtain target prediction frames of the same size. Classification and prediction frame regression are then performed through the fully connected layer to obtain the final target detection result as the plaque detection result.

[0010] Step 5, network training: training the coronary heart disease plaque detection network based on the attention mechanism;

[0011] Step 6, plaque detection: The coronary artery CT angiography image to be detected is input into the trained coronary heart disease plaque detection network based on the attention mechanism, and the plaque prediction box is output.

[0012] Specifically, the specific processing process of the coronary artery three-dimensional model enhancement algorithm described in step 1 is: first, the coronary artery CT angiography image is processed by a three-dimensional cavity elimination algorithm to obtain an image after the cavity in the model is eliminated, then the image is three-dimensionally expanded twice, and then three-dimensionally eroded once, and finally, the three-dimensional cavity elimination algorithm is used to eliminate the cavity in the model to obtain an enhanced coronary artery CT angiography image.

[0013] Specifically, the specific processing process of calculating the coronary artery centerline using the three-dimensional thinning method described in step 1 is: first, the image is three-dimensionally thinned using a three-dimensional skeleton calculation method to obtain a number of discrete points; then, these discrete points are aggregated and connected using a three-dimensional scattered point clustering algorithm to obtain a directed line segment from the aorta to the end of the coronary artery; finally, the line segment is smoothed using a three-dimensional convolution smoothing algorithm to obtain the final coronary artery centerline.

[0014] Specifically, the super-resolution processing described in step 2 refers to recalculating the HU value of each point by increasing the sampling rate and combining the HU values ​​of surrounding voxels.

[0015] Specifically, the ResNet50 convolutional neural network with the introduction of the attention mechanism described in step 4 refers to adding an attention unit and a channel attention unit between every two Bottlenecks of the ResNet50 convolutional neural network. The attention unit uses a 1×1 convolution to fuse all feature maps into one, and then uses a Softmax operation to obtain the attention weight of the feature map; the channel attention unit uses an average pooling operation to pool the feature map of each channel into a 1×1 size, and then uses a 1×1 convolution to learn aggregate features from each channel, performs a Softmax operation on the learned features, and assigns different weights to different feature channels.

[0016] Specifically, the non-maximum suppression method is used to filter the prediction boxes output by the coronary heart disease plaque detection network based on the attention mechanism, and the filtered prediction boxes are used as the plaque detection results.

[0017] Specifically, the specific process of the network training described in step 5 is:

[0018] Step a: Use the pre-trained weights of ImageNet to initialize the shared conv layer of the RPN network, randomly initialize the unique layer of the RPN network, and then train the RPN network to update the weights of its shared conv layer and unique layer;

[0019] Step b: Get the target prediction box based on the trained RPN network;

[0020] Step c: Use the pre-trained weights of the image dataset obtained in step 3 to initialize the shared conv layer of the Faster RCNN network, randomly initialize the unique layer of the Faster RCNN network, and then train the Faster RCNN network to update its shared conv layer and unique layer weights;

[0021] Step d: Use the shared conv layer trained in step c and the RPN-specific layer trained in step a to initialize the RPN network, and train the RPN network for the second time. During training, the weights of the shared conv layer are fixed, and only the weights of the specific layer are updated;

[0022] Step e: Get the target prediction box based on the trained RPN network;

[0023] Step f: Use the shared conv layer and unique layer trained in step c to initialize the Faster RCNN network, and train the Faster RCNN network for the second time. During training, fix the shared conv layer weights and only update the unique layer weights.

[0024] The beneficial effects of the present invention are as follows: since the coronary artery three-dimensional model enhancement algorithm and the morphological processing of expansion corrosion are used to extract the coronary artery centerline, the cavity inside the coronary artery model and the noise on the model surface can be removed, the smoothness of the model surface can be improved, and the image quality can be effectively improved; since multi-angle surface reconstruction and image fusion are used, multi-angle image information can be integrated, and the problem of low accuracy caused by angle limitation in single surface reconstructed images can be effectively solved, and the accuracy of plaque recognition in coronary artery CT angiography images can be significantly improved; since the ResNet50 convolutional neural network with an attention mechanism is used as the backbone network, image features can be better extracted by assigning different weights, the target detection effect of the network can be improved, and the detection accuracy of plaques in images can be greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flow chart of a method for identifying plaques in coronary CT angiography images based on deep learning of the present invention;

[0026] Figure 2 It is a comparison of the centerlines calculated by the original coronary artery model and the enhanced coronary artery model;

[0027] In the figure, (a) - original coronary artery model, (b) - directly calculated centerline, (c) - calculated centerline after 3D enhancement;

[0028] Figure 3 This is the effect of the same coronary artery after curved surface reconstruction at different angles;

[0029] Figure 4 is a schematic diagram of a spatial attention unit;

[0030] Figure 5 is a schematic diagram of the channel attention unit;

[0031] Figure 6 It is a schematic diagram of the detection network training process of the present invention;

[0032] Figure 7 It is a comparison chart of the detection results of a single reconstructed image and the fused image. DETAILED DESCRIPTION

[0033] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The present invention includes but is not limited to the following embodiments.

[0034] The present invention provides a method for identifying plaques in coronary CT angiography images based on deep learning, such as Figure 1 As shown, the specific implementation process is as follows:

[0035] 1. Calculate the coronary artery centerline

[0036] The original coronary CT angiography image (CTA image) is input. In order to complete the plaque feature extraction from the coronary artery model, the centerline of the coronary artery must be calculated first. The specific process is: first, the coronary artery 3D model enhancement algorithm is used to enhance the coronary artery 3D model in the image, and the cavity inside the model and the noise on the model surface are removed. Then, the 3D refinement method is used to complete the calculation of the coronary artery centerline.

[0037] Specifically, the specific processing process of the coronary artery 3D model enhancement algorithm is as follows: First, the coronary artery CT angiography image is processed using a 3D cavity elimination algorithm to obtain an image after the cavity in the model is eliminated, thereby achieving coronary artery 3D model enhancement. Then, the image is further processed using a morphological processing method. First, the 3D expansion algorithm is used twice to remove tiny burrs and gaps on the model surface, and then the model surface is smoothed using a 3D corrosion algorithm to further improve the model quality. The 3D expansion algorithm will introduce new cavities in the model, and the cavities have an impact on both the corrosion and expansion algorithms. Therefore, a cavity elimination process is still performed at the end.

[0038] Specifically, a three-dimensional thinning method is used to calculate the coronary artery centerline, including a three-dimensional skeleton calculation algorithm, a three-dimensional scattered point clustering algorithm, and a three-dimensional convolution smoothing algorithm. The specific steps are: first, the three-dimensional skeleton calculation method is used to perform three-dimensional thinning on the image to obtain a number of discrete points; then, these discrete points are aggregated and connected using the three-dimensional scattered point clustering algorithm to obtain a directed line segment from the aorta to the end of the coronary artery; finally, the three-dimensional convolution smoothing algorithm is used to smooth the line segment to obtain the final coronary artery centerline. After the three-dimensional convolution smoothing algorithm, a smooth and continuous coronary artery centerline is output, which can effectively judge the intravascular plaque and stenosis. Figure 2 A comparison of the centerline calculated from the original coronary artery model and the enhanced coronary artery model is given, where (a) is the original coronary artery model, (b) is the directly calculated centerline, and (c) is the centerline calculated after three-dimensional enhancement. It can be seen that the surface smoothness of the coronary artery model has been significantly improved after enhancement, and cavities, burr noise, etc. have been eliminated. The topological structure of the coronary artery can be effectively obtained by using the centerline extraction algorithm based on the enhanced coronary artery model, and only 9% of the images have tiny and easily removable burrs.

[0039] 2. Multi-angle surface reconstruction

[0040] The input CTA image is marked with a centerline after step 1, and this step completes multi-angle surface reconstruction on this basis. First, the final coronary artery centerline is used to calculate the coordinate value of each point on the coronary artery centerline in the TNB coordinate system, and the Frenet formula is used to describe the curvature and torsion changes of the blood vessel in three-dimensional space. Then, super-resolution processing is performed on the curvature and torsion changes of the blood vessel in three-dimensional space described by the Frenet formula. By increasing the sampling rate and combining the HU values ​​of the surrounding voxels, the HU value of each point is recalculated to improve the image clarity. Finally, the obtained clear image is used to reconstruct the coronary artery from multiple angles to generate multi-angle surface reconstructed images. Figure 3 The effect pictures of the same coronary artery after surface reconstruction at different angles are given.

[0041] 3. Multi-angle surface reconstruction image fusion

[0042] The surface reconstructed images at various angles are fused by using a preliminary fusion method to obtain multiple images that integrate multi-angle information and enhance the display of plaque features.

[0043] 4. Constructing a coronary heart disease plaque detection network based on attention mechanism

[0044] Based on the image data set fused in step 3, the present invention constructs a coronary heart disease plaque detection network model based on the attention mechanism, and takes the deep learning network Faster R-CNN as the basic network model. The specific calculation process of Faster RCNN is as follows: (1) inputting the file to be processed into the neural network; (2) obtaining the target features of the image to be processed through the feature extraction network; (3) generating candidate domains through the region generation network; (4) using the classifier to distinguish the candidate regions; (5) adjusting the candidate region positions through the regressor.

[0045] The feature extraction network used in the present invention is based on the Faster R-CNN network model, and the ResNet50 convolutional neural network with the introduction of the attention mechanism is used as the backbone network for feature extraction. The overall process is divided into five stages, of which the first stage can be regarded as a preprocessing process for the input. The calculation process of the remaining four stages consists of four Bottlenecks with similar structures. The present invention adds an attention unit and a channel attention unit between every two Bottlenecks, and introduces an attention mechanism to give different weights to the reconstructed image, thereby improving the recognition effect of plaques in the surface reconstructed image of the CTA image sequence.

[0046] Among them, Figure 4 As shown in , the attention unit uses a 1×1 convolution to fuse all feature maps into one, and then uses the Softmax operation to obtain the attention weight of the feature map. Figure 5 As shown in the figure, the channel attention unit uses an average pooling operation to pool the feature map of each channel (size is W×H×N) to a size of 1×1, and then uses a 1×1 convolution to learn aggregate features from each channel, performs a Softmax operation on the learned features, and assigns different weights to different feature channels.

[0047] After extracting the feature map of the image through the ResNet50 network with the introduction of the attention mechanism, the feature map is subjected to target detection through the region proposal network (RPN network) to generate high-quality candidate regions and obtain several target prediction boxes that may have plaque features. Then, the prediction box is resized through the RoI pooling layer. The purpose of the RoI pooling layer is to convert candidate regions of different sizes from the RPN into feature representations of fixed size so that the subsequent fully connected layer can process them. Finally, classification and prediction box regression are performed through the fully connected layer to obtain the final target detection result, i.e., the plaque detection result. Furthermore, the prediction box can be filtered through non-maximum suppression, retaining the lesion prediction box with the highest confidence, and removing other lesion prediction boxes with lower confidence, and the filtered prediction box is used as the final plaque detection result.

[0048] 5. Network training

[0049] The coronary heart disease plaque detection network based on the attention mechanism constructed in step 4 is trained, such as Figure 6 As shown, the specific process is as follows:

[0050] Step a: Use the pre-trained weights of ImageNet to initialize the shared conv layer of the RPN network, randomly initialize the unique layer of the RPN network, and then train the RPN network to update the weights of its shared conv layer and unique layer; ImageNet models include ZF, VGG, etc.

[0051] Step b: Get proposals (target prediction boxes) based on the trained RPN network;

[0052] Step c: Use the pre-trained weights of the image dataset obtained in step 3 to initialize the shared conv layer of the Faster RCNN network, randomly initialize the unique layer of the Faster RCNN network, and then train the Faster RCNN network to update its shared conv layer and unique layer weights;

[0053] Step d: Use the shared conv layer trained in step c and the RPN-specific layer trained in step a to initialize the RPN network, and train the RPN network for the second time. During training, the weights of the shared conv layer are fixed, and only the weights of the specific layer are updated;

[0054] Step e: Get proposals based on the trained RPN network;

[0055] Step f: Use the shared conv layer and unique layer trained in step c to initialize the Faster RCNN network, and train the Faster RCNN network for the second time. During training, fix the shared conv layer weights and only update the unique layer weights.

[0056] After the above steps, a coronary heart disease plaque detection network based on the attention mechanism based on the Fast R-CNN network model can be obtained.

[0057] 6. Plaque detection

[0058] The coronary CT angiography image to be detected is input into the trained coronary heart disease plaque detection network based on the attention mechanism, and a multi-target plaque detection image with a prediction frame is output. The specific location of the plaque in the three-dimensional image can be displayed, which will help the next step of diagnosis and treatment.

[0059] In the experimental comparison, the recognition rate of coronary artery plaques in surface reconstructed images was increased to 95% by optimizing the post-processing process of CTA image sequences, improving the quality of surface reconstructed images, creating data sets, selecting more effective deep learning network models and feature extraction networks, fusing multi-angle surface reconstructed images to increase the amount of information, and combining attention units to reduce the impact of noise.

[0060] Multi-angle curved surface reconstruction image fusion can effectively improve the accuracy of plaque recognition in the intermediate image of the coronary artery curved surface, and solve the problem of low accuracy caused by angle limitations in single curved surface reconstruction images.

[0061] At the same time, the coronary heart disease plaque detection network based on the attention mechanism not only has improved accuracy compared to the single surface reconstruction image detection network, but also has a significantly faster computing speed than the single surface reconstruction image detection network, which detects three reconstructed images three times.

[0062] In the experiment carried out by the method of the present invention, Figure 7 As shown in the figure, the coronary heart disease plaque detection network based on the attention mechanism uses three fused surface reconstructed images as input, which can effectively identify the lesion site. In the figure, the numbers represent the confidence of the target prediction box to detect the plaque.

Claims

1. A method for identifying plaques in coronary CT angiography images based on deep learning, characterized in that Here are the steps: Step 1, calculating the coronary artery centerline: first, the input coronary artery CT angiography image is enhanced using the coronary artery three-dimensional model enhancement algorithm, and then the coronary artery centerline is calculated using a three-dimensional thinning method; Step 2, multi-angle surface reconstruction: First, calculate the coordinate value of each point on the center line of the coronary artery in the TNB coordinate system, and use the Frenet formula to calculate the curvature and torsion of the center line of the coronary artery in three-dimensional space. Then, perform super-resolution processing on the curvature and torsion, reconstruct the coronary artery from multiple angles, and generate surface reconstructed images at multiple angles; Step 3, multi-angle surface reconstruction image fusion: the surface reconstruction images at various angles are fused using the previous fusion method to obtain a number of fused images; Step 4: Construct a coronary heart disease plaque detection network based on the attention mechanism: Based on the Faster R-CNN network model, the ResNet50 convolutional neural network with the attention mechanism is used as the backbone network to extract the feature map of the image. Then, the feature map is subjected to target detection through the RPN network to obtain several target prediction frames. Then, the prediction frames are resized through the RoI pooling layer to obtain target prediction frames of the same size. Classification and prediction frame regression are then performed through the fully connected layer to obtain the final target detection result as the plaque detection result. Step 5, network training: training the coronary heart disease plaque detection network based on the attention mechanism; Step 6, plaque detection: The coronary artery CT angiography image to be detected is input into the trained coronary heart disease plaque detection network based on the attention mechanism, and the plaque prediction box is output.

2. The method for identifying plaques in coronary CT angiography images based on deep learning according to claim 1, characterized in that: The specific processing process of the coronary artery three-dimensional model enhancement algorithm described in step 1 is: first, the coronary artery CT angiography image is processed by a three-dimensional cavity elimination algorithm to obtain an image after the cavity in the model is eliminated, then the image is subjected to two three-dimensional expansion processes, and then to a three-dimensional corrosion process, and finally, the three-dimensional cavity elimination algorithm is used to eliminate the cavity in the model to obtain an enhanced coronary artery CT angiography image.

3. The method for identifying plaques in coronary CT angiography images based on deep learning according to claim 1, characterized in that: The specific processing process of calculating the coronary artery centerline using the three-dimensional thinning method described in step 1 is: first, the image is three-dimensionally thinned using the three-dimensional skeleton calculation method to obtain a number of discrete points; then, these discrete points are aggregated and connected using the three-dimensional scattered point clustering algorithm to obtain a directed line segment from the aorta to the end of the coronary artery; finally, the line segment is smoothed using the three-dimensional convolution smoothing algorithm to obtain the final coronary artery centerline.

4. The method for identifying plaques in coronary CT angiography images based on deep learning according to claim 1, characterized in that: The super-resolution process described in step 2 is to recalculate the HU value of each point by increasing the sampling rate and combining the HU values ​​of surrounding voxels.

5. The method for identifying plaques in coronary CT angiography images based on deep learning according to claim 1, characterized in that: The ResNet50 convolutional neural network with the introduction of the attention mechanism described in step 4 refers to adding an attention unit and a channel attention unit between every two Bottlenecks of the ResNet50 convolutional neural network. The attention unit uses a 1×1 convolution to fuse all feature maps into one, and then uses a Softmax operation to obtain the attention weight of the feature map. The channel attention unit uses an average pooling operation to pool the feature map of each channel into a 1×1 size, and then uses a 1×1 convolution to learn aggregate features from each channel, performs a Softmax operation on the learned features, and assigns different weights to different feature channels.

6. The method for identifying plaques in coronary CT angiography images based on deep learning according to claim 1, characterized in that: The non-maximum suppression method is used to filter the prediction boxes output by the coronary heart disease plaque detection network based on the attention mechanism, and the filtered prediction boxes are used as the plaque detection results.

7. The method for identifying plaques in coronary CT angiography images based on deep learning according to claim 1, characterized in that: The specific process of network training described in step 5 is: Step a: Use the pre-trained weights of ImageNet to initialize the shared conv layer of the RPN network, randomly initialize the unique layer of the RPN network, and then train the RPN network to update the weights of its shared conv layer and unique layer; Step b: Get the target prediction box based on the trained RPN network; Step c: Use the pre-trained weights of the image dataset obtained in step 3 to initialize the shared conv layer of the Faster RCNN network, randomly initialize the unique layer of the Faster RCNN network, and then train the Faster RCNN network to update its shared conv layer and unique layer weights; Step d: Use the shared conv layer trained in step c and the RPN-specific layer trained in step a to initialize the RPN network, and train the RPN network for the second time. During training, the weights of the shared conv layer are fixed, and only the weights of the specific layer are updated; Step e: Get the target prediction box based on the trained RPN network; Step f: Use the shared conv layer and unique layer trained in step c to initialize the Faster RCNN network, and train the Faster RCNN network for the second time. During training, fix the shared conv layer weights and only update the unique layer weights.

Images