A coronary artery image segmentation method based on multi-label and segmentation network

Abstract

The present invention provides a coronary artery image segmentation method based on multi-label and segmentation network, comprising the following steps: S1: acquiring a cardiovascular three-dimensional image; S2: using a two-dimensional segmentation network and a three-dimensional segmentation network to segment the cardiovascular three-dimensional image, respectively, to obtain a first segmentation feature map and a second segmentation feature map with labels; S3: fusing the first segmentation feature map and the second segmentation feature map to predict the coronary artery image, to obtain a predicted two-dimensional coronary artery image; S4: reconstructing a coronary artery three-dimensional image based on the predicted two-dimensional coronary artery image, that is, obtaining a segmented coronary artery image. The present invention provides a coronary artery image segmentation method based on multi-label and segmentation network, which solves the problem that the existing segmentation technology is difficult to accurately segment the coronary artery image.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and more specifically, to a coronary artery image segmentation method based on multi-label and segmentation network. Background Art

[0002] Anomalous origin of the left coronary artery is a congenital heart malformation with a very high mortality rate. Modern medical imaging modalities (such as CT and MRI) can generate a large amount of data in the form of three-dimensional image volumes. The current gold standard for traditional segmentation of cardiovascular images is done manually by clinical experts, which is a time-consuming task, and manual segmentation is also prone to bias and errors.

[0003] When using classic deep learning segmentation models for cardiovascular images of cases with anomalous origin of the left coronary artery, the segmentation is often inaccurate due to the thin diameter and position of the left and right coronary arteries and the unclear contrast with the surrounding background in the vascular CT images. Furthermore, due to different personal physiological conditions, disease progression, pathophysiology and classification, the morphology and size of the left and right coronary arteries of different patients vary greatly, which also makes it difficult for existing segmentation technologies to accurately segment coronary artery images to a certain extent. Summary of the invention

[0004] In order to overcome the technical defect that the existing segmentation technology is difficult to accurately segment the coronary artery image, the present invention provides a coronary artery image segmentation method based on multi-label and segmentation network.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] A coronary artery image segmentation method based on multi-label and segmentation network comprises the following steps:

[0007] S1: Acquire cardiovascular three-dimensional images;

[0008] S2: using a two-dimensional segmentation network and a three-dimensional segmentation network to segment the cardiovascular three-dimensional image, respectively, to obtain a first segmentation feature map and a second segmentation feature map with labels;

[0009] S3: fusing the first segmentation feature map and the second segmentation feature map to perform coronary artery image prediction to obtain a predicted coronary artery two-dimensional image;

[0010] S4: reconstructing a three-dimensional image of the coronary arteries according to the predicted two-dimensional image of the coronary arteries, that is, obtaining a segmented coronary artery image.

[0011] In the above scheme, the cardiovascular three-dimensional image is segmented by a two-dimensional segmentation network and a three-dimensional segmentation network to obtain a first segmentation feature map and a second segmentation feature map with labels, and then fusion, prediction and reconstruction are performed based on the first segmentation feature map, the second segmentation feature map and the label information, thereby improving the segmentation accuracy.

[0012] Preferably, the two-dimensional segmentation network is a 2D U-Net.

[0013] Preferably, the first segmentation feature map is obtained by the following steps:

[0014] First, slice the cardiovascular 3D image with a size of l*h*w along the Z axis to obtain w 2D slices, and the i-th 2D slice is denoted as I i , whose label is L i ; Then the 2D slice is input into the 2D U-Net to segment the coronary artery part, and the 2D feature map of the second to last layer of the 2D U-Net is taken as the first segmentation feature map M i , i=1,2,…,w, l, h, w are the length, height and width of the cardiovascular three-dimensional image respectively.

[0015] Preferably, i=3,4,…,w-2.

[0016] Preferably, the three-dimensional segmentation network includes 3D U-Net and 3D modified U-Net; wherein,

[0017] The 3D U-Net is used to pre-segment cardiovascular three-dimensional images;

[0018] The 3D modified U-Net is used to further segment the pre-segmented cardiovascular three-dimensional image.

[0019] Preferably, in the pre-segmentation process, the label data includes the myocardium, left ventricle, right ventricle, aorta, pulmonary artery, left coronary artery and right coronary artery.

[0020] Preferably, the second segmentation feature map is obtained by the following steps:

[0021] S2.1: Pre-segmentation of cardiovascular 3D images using 3D U-Net;

[0022] S2.2: performing downsampling and normalization preprocessing on the pre-segmented cardiovascular three-dimensional image in sequence;

[0023] S2.3: Input the preprocessed cardiovascular 3D image into 3D modified U-Net for further segmentation;

[0024] S2.4: Take the 3D feature map of the penultimate layer of the 3D modified U-Net and upsample the 3D feature map;

[0025] S2.5: Slice the upsampled 3D feature map along the Z axis to obtain the second segmentation feature map M i ′.

[0026] Preferably, in step S2.4, the size of the three-dimensional feature map is upsampled to the same as that of the cardiovascular three-dimensional image, while the number of channels of the three-dimensional feature map is kept unchanged.

[0027] Preferably, the first segmentation feature map and the second segmentation feature map are fused by channel splicing to obtain a fused feature map.

[0028] Preferably, the fused feature map is used to predict the coronary artery image through an integrated model, wherein the integrated model includes a 3*3 two-dimensional convolution layer, a ReLU function activation layer, a fully connected layer and a residual structure;

[0029] In the integrated model, the fused feature map is sequentially passed through a 3*3 two-dimensional convolutional layer and a ReLU function activation layer to extract a high-level semantic feature map, and the first segmentation feature map and the high-level semantic feature map are superimposed (element-wise sum, i.e., directly adding the corresponding positions of the array / matrix) through a residual structure, and finally the predicted two-dimensional image of the coronary artery is output through a fully connected layer.

[0030] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0031] The present invention provides a coronary artery image segmentation method based on multi-label and segmentation network. A cardiovascular three-dimensional image is segmented by a two-dimensional segmentation network and a three-dimensional segmentation network to obtain a first segmentation feature map and a second segmentation feature map with labels. Then, the first segmentation feature map, the second segmentation feature map and label information are fused, predicted and reconstructed, thereby improving the segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flowchart of the implementation steps of the technical solution of the present invention;

[0033] Figure 2 is an implementation flow chart of an embodiment of the present invention;

[0034] Figure 3 This is a flow chart of the present invention for predicting the coronary artery image after fusing the first segmentation feature map and the second segmentation feature map;

[0035] Figure 4 Schematic diagram of the comparison between the segmentation results and the true labels of the methods respectively using 2DU-Net, 3DU-Net, maximum integrated voting, and the described methods in the present invention. DETAILED DESCRIPTION

[0036] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0037] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0038] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0039] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0040] Example 1

[0041] like Figure 1 As shown, a coronary artery image segmentation method based on multi-label and segmentation network includes the following steps:

[0042] S1: Acquire cardiovascular three-dimensional images;

[0043] S2: using a two-dimensional segmentation network and a three-dimensional segmentation network to segment the cardiovascular three-dimensional image, respectively, to obtain a first segmentation feature map and a second segmentation feature map with labels;

[0044] S3: fusing the first segmentation feature map and the second segmentation feature map to perform coronary artery image prediction to obtain a predicted coronary artery two-dimensional image;

[0045] S4: reconstructing a three-dimensional image of the coronary arteries according to the predicted two-dimensional image of the coronary arteries, that is, obtaining a segmented coronary artery image.

[0046] In the specific implementation process, the cardiovascular three-dimensional image is segmented by a two-dimensional segmentation network and a three-dimensional segmentation network to obtain a first segmentation feature map and a second segmentation feature map with labels, and then the first segmentation feature map, the second segmentation feature map and the label information are fused, predicted and reconstructed to improve the segmentation accuracy.

[0047] Example 2

[0048] A coronary artery image segmentation method based on multi-label and segmentation network comprises the following steps:

[0049] S1: Acquire cardiovascular three-dimensional images;

[0050] S2: using a two-dimensional segmentation network and a three-dimensional segmentation network to segment the cardiovascular three-dimensional image, respectively, to obtain a first segmentation feature map and a second segmentation feature map with labels;

[0051] S3: fusing the first segmentation feature map and the second segmentation feature map to perform coronary artery image prediction to obtain a predicted coronary artery two-dimensional image;

[0052] S4: reconstructing a three-dimensional image of the coronary arteries according to the predicted two-dimensional image of the coronary arteries, that is, obtaining a segmented coronary artery image.

[0053] More specifically, the two-dimensional segmentation network is a classic 2D U-Net.

[0054] In the specific implementation process, a five-fold cross validation method is used to train the 2D U-Net. The training method of the 2D U-Net is existing and will not be described in detail here.

[0055] More specifically, the first segmentation feature map is obtained by the following steps:

[0056] First, slice the cardiovascular 3D image with a size of l*h*w along the Z axis to obtain w 2D slices, and the i-th 2D slice is denoted as I i , whose label is L i ; Then the 2D slice is input into the 2D U-Net to segment the coronary artery part, and the 2D feature map of the second to last layer of the 2D U-Net is taken as the first segmentation feature map M i , i=1,2,…,w, l, h, w are the length, height and width of the cardiovascular three-dimensional image respectively.

[0057] More specifically, i=3,4,…,w-2.

[0058] More specifically, the three-dimensional segmentation network includes a classic 3D U-Net and a 3D modified U-Net; wherein,

[0059] The 3D U-Net is used to pre-segment cardiovascular three-dimensional images;

[0060] The 3D modified U-Net is used to further segment the pre-segmented cardiovascular three-dimensional image.

[0061] In the specific implementation process, the classic 3D modified U-Net is improved based on U-Net, which mainly consists of a compression path that captures context information and an expansion path that is symmetrical to the compression path for precise positioning. 3Dmodified U-Net can make full use of a small number of images for end-to-end training, segment each pixel, obtain higher segmentation accuracy, and also facilitate better learning of deep image features.

[0062] In the specific implementation process, a five-fold cross validation method is used to train the 3D U-Net and 3D modified U-Net. The training methods of the 3D U-Net and 3D modified U-Net are existing and will not be described in detail here.

[0063] More specifically, in the pre-segmentation process, the label data includes myocardium, left ventricle, right ventricle, aorta, pulmonary artery, left coronary artery and right coronary artery.

[0064] More specifically, the second segmentation feature map is obtained by the following steps:

[0065] S2.1: Pre-segmentation of cardiovascular 3D images using 3D U-Net;

[0066] S2.2: performing downsampling and normalization preprocessing on the pre-segmented cardiovascular three-dimensional image in sequence;

[0067] S2.3: Input the preprocessed cardiovascular 3D image into 3D modified U-Net for further segmentation;

[0068] S2.4: Take the 3D feature map of the penultimate layer of the 3D modified U-Net and upsample the 3D feature map;

[0069] S2.5: Slice the upsampled 3D feature map along the Z axis to obtain the second segmentation feature map M i ′.

[0070] More specifically, in step S2.4, the size of the three-dimensional feature map is upsampled to the same as that of the cardiovascular three-dimensional image, while the number of channels of the three-dimensional feature map remains unchanged.

[0071] More specifically, the first segmentation feature map and the second segmentation feature map are fused through channel splicing to obtain a fused feature map.

[0072] More specifically, the fused feature map is used to predict the coronary artery image through an integrated model, wherein the integrated model includes a 3*3 two-dimensional convolution layer, a ReLU function activation layer, a fully connected layer and a residual structure;

[0073] In the integrated model, the fused feature map is sequentially passed through a 3*3 two-dimensional convolutional layer and a ReLU function activation layer to extract a high-level semantic feature map, and the first segmentation feature map and the high-level semantic feature map are superimposed (element-wise sum, i.e., directly adding the corresponding positions of the array / matrix) through a residual structure, and finally the predicted two-dimensional image of the coronary artery is output through a fully connected layer.

[0074] Example 3

[0075] like Figure 2 As shown, a coronary artery image segmentation method based on multi-label and segmentation network includes the following steps:

[0076] S1: Acquire cardiovascular three-dimensional images;

[0077] It is assumed that the cardiovascular three-dimensional image is a cardiovascular CT three-dimensional image with a size of 512*512*275.

[0078] S2: using a two-dimensional segmentation network and a three-dimensional segmentation network to segment the cardiovascular three-dimensional image, respectively, to obtain a first segmentation feature map and a second segmentation feature map with labels;

[0079] More specifically, the two-dimensional segmentation network is a 2D U-Net.

[0080] In the specific implementation process, the accurate identification of blood vessel edges through 2D U-Net helps to improve the accuracy of coronary artery image segmentation.

[0081] More specifically, the first segmentation feature map is obtained by the following steps:

[0082] First, slice the cardiovascular 3D image with a size of 512*512*275 along the Z axis to obtain 275 2D slices, and the i-th 2D slice is denoted as I i , whose label is L i ; Then the 2D slice is input into the 2D U-Net to segment the coronary artery part, and the 2D feature map of the second to last layer of the 2D U-Net is taken as the first segmentation feature map M i , i=1,2,…,275.

[0083] More specifically, for CT images around the heart, the first, second, and last two images usually do not include coronary arteries, so i=3, 4, ..., 273 can be taken here.

[0084] More specifically, the three-dimensional segmentation network includes 3D U-Net and 3D modified U-Net; wherein,

[0085] The 3D U-Net is used to pre-segment cardiovascular three-dimensional images;

[0086] The 3D modified U-Net is used to further segment the pre-segmented cardiovascular three-dimensional image.

[0087] More specifically, in the pre-segmentation process, the label data includes myocardium, left ventricle, right ventricle, aorta, pulmonary artery, left coronary artery and right coronary artery.

[0088] In the specific implementation process, the whole heart is segmented according to the label data to improve the pre-segmentation effect. Since the original size of cardiovascular CT three-dimensional images is large, the images need to be downsampled before being input into 3DU-Net. The branches and terminal parts of the coronary arteries are relatively small. During this downsampling process, the resolution drops significantly, which greatly affects the pre-segmentation effect. The information of the labels around the coronary arteries has greatly improved the pre-segmentation effect, ensuring that the region of interest after pre-segmentation can contain the complete coronary artery and greatly reduce the size of the image without affecting the image resolution.

[0089] More specifically, the second segmentation feature map is obtained by the following steps:

[0090] S2.1: Pre-segmentation of cardiovascular 3D images using 3D U-Net;

[0091] S2.2: performing downsampling and normalization preprocessing on the pre-segmented cardiovascular three-dimensional image in sequence;

[0092] S2.3: Input the preprocessed cardiovascular 3D image into 3D modified U-Net for further segmentation;

[0093] S2.4: Take the 3D feature map of the penultimate layer of the 3D modified U-Net and upsample the 3D feature map;

[0094] S2.5: Slice the upsampled 3D feature map along the Z axis to obtain the second segmentation feature map M i ′.

[0095] More specifically, in step S2.4, the size of the three-dimensional feature map is upsampled to the same as that of the cardiovascular three-dimensional image, while the number of channels of the three-dimensional feature map remains unchanged.

[0096] In the specific implementation process, the size of the 3D feature map obtained by the penultimate layer of the 3D modified U-Net is 256*256*128*8, where 8 is the number of channels of the 3D feature map. After the 3D feature map is upsampled and sliced, the size of the second segmentation feature map obtained is 512*512*8, that is, the number of channels of the second segmentation feature map is also 8.

[0097] S3: fusing the first segmentation feature map and the second segmentation feature map to perform coronary artery image prediction to obtain a predicted coronary artery two-dimensional image;

[0098] More specifically, Figure 3 As shown, the first segmentation feature map and the second segmentation feature map are fused by channel splicing to obtain a fused feature map.

[0099] More specifically, the fused feature map is used to predict the coronary artery image through an integrated model, wherein the integrated model includes a 3*3 two-dimensional convolution layer, a ReLU function activation layer, a fully connected layer and a residual structure;

[0100] In the integrated model, the fused feature map is sequentially passed through a 3*3 two-dimensional convolutional layer and a ReLU function activation layer to extract a high-level semantic feature map, and the first segmentation feature map and the high-level semantic feature map are superimposed (element-wise sum, i.e., directly adding the corresponding positions of the array / matrix) through a residual structure, and finally the predicted two-dimensional image of the coronary artery is output through a fully connected layer.

[0101] In the specific implementation process, the segmentation results of the 2DU-Net model show that its continuity between different slices is poor, while the segmentation of the blood vessel edge is smooth; the 3DU-Net model and the 3D modified U-Net model can extract information between slices, which helps to improve the continuity between slices, and the obtained feature map contains information provided by other labels; by channel splicing and fusing the different feature information and position information provided by the labels in the first segmentation feature map and the second segmentation feature map, and then using deep convolution to extract high-level semantic features from the fused feature map, the segmentation accuracy of the coronary artery image is effectively improved. In addition, the residual structure is used to reduce the cost of model learning, and only the first segmentation feature map obtained by 2DU-Net segmentation is superimposed with the high-level semantic feature map using a jump connection method, which helps to retain the information extracted by 2DU-Net and prevent the integrated model from paying too much attention to the information about other labels provided by 3DU-Net.

[0102] S4: reconstructing a three-dimensional image of the coronary arteries according to the predicted two-dimensional image of the coronary arteries, that is, obtaining a segmented coronary artery image.

[0103] Example 4

[0104] This embodiment uses the Dice coefficient as an indicator to quantitatively analyze and evaluate the segmentation effect of the coronary artery image segmentation method based on multi-label and segmentation network (hereinafter referred to as the method).

[0105] The Dice coefficient is a set similarity measurement function, usually used to calculate the similarity between two samples (value range is [0,1]). Its calculation formula is:

[0106]

[0107] In image segmentation, |X∩Y| represents the intersection between the predicted image X and the real image Y; |X| and |Y| represent the area size of the predicted image X and the real image Y respectively. The larger the s value, the more similar the predicted image X and the real image Y are.

[0108] The Dice index is used to evaluate the similarity between the results of the classic 2DU-Net, 3DU-Net, maximum integrated voting and the segmentation results of the described method and the true label. Table 1 shows the results of the Dice operation of the coronary artery segmentation experimental results obtained by 2DU-Net, 3DU-Net, maximum integrated voting and the segmentation of the described method and the true label.

[0109] Table 1

[0110] Split method Dice indicator 2DU-Net 0.5851 3DU-Net 0.5430 Maximum integrated voting 0.5936 The method described 0.6019

[0111] As can be seen from Table 1, the Dice index between the coronary artery segmentation experimental results obtained by the method and the true label is higher than that of the segmentation methods using 2DU-Net, 3DU-Net, and maximum integrated voting, that is, the segmentation accuracy of the method is higher than that of the segmentation methods using 2DU-Net, 3DU-Net, and maximum integrated voting. Figure 4 Comparison between the segmentation results and the true labels using 2DU-Net, 3DU-Net, maximum ensemble voting, and the described method.

[0112] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A coronary artery image segmentation method based on multi-label and segmentation network, characterized in that: The following steps are involved: S1: Acquire cardiovascular three-dimensional images; S2: using a two-dimensional segmentation network and a three-dimensional segmentation network to segment the cardiovascular three-dimensional image, respectively, to obtain a first segmentation feature map and a second segmentation feature map with labels; The first segmentation feature map is obtained by the following steps: First, slice the cardiovascular 3D image with a size of l*h*w along the Z axis to obtain w 2D slices, and the i-th 2D slice is denoted as I i , whose label is L i ; Then the 2D slice is input into the 2D U-Net to segment the coronary artery part, and the 2D feature map of the second to last layer of the 2D U-Net is taken as the first segmentation feature map M i , i=1,2,…,w, l, h, w are the length, height and width of cardiovascular three-dimensional image respectively; The second segmentation feature map is obtained by the following steps: S2.1: Pre-segmentation of cardiovascular 3D images using 3D U-Net; S2.2: performing downsampling and normalization preprocessing on the pre-segmented cardiovascular three-dimensional image in sequence; S2.3: Input the preprocessed cardiovascular 3D image into 3D modified U-Net for further segmentation; S2.4: Take the 3D feature map of the penultimate layer of the 3D modified U-Net and upsample the 3D feature map; S2.5: Slice the upsampled 3D feature map along the Z axis to obtain the second segmentation feature map M i ′ ; S3: fusing the first segmentation feature map and the second segmentation feature map to perform coronary artery image prediction to obtain a predicted coronary artery two-dimensional image; S4: reconstructing a three-dimensional image of the coronary arteries according to the predicted two-dimensional image of the coronary arteries, that is, obtaining a segmented coronary artery image.

2. The method for coronary artery image segmentation based on multi-label and segmentation network according to claim 1, characterized in that: i=3,4,…,w-2.

3. The method for coronary artery image segmentation based on multi-label and segmentation network according to claim 1, characterized in that: In the pre-segmentation process, the label data includes myocardium, left ventricle, right ventricle, aorta, pulmonary artery, left coronary artery, and right coronary artery.

4. The method for segmenting coronary artery images based on multi-label and segmentation network according to claim 3, characterized in that: In step S2.4, the size of the three-dimensional feature map is upsampled to the same as that of the cardiovascular three-dimensional image, while the number of channels of the three-dimensional feature map remains unchanged.

5. The method for coronary artery image segmentation based on multi-label and segmentation network according to claim 1, characterized in that: The first segmentation feature map and the second segmentation feature map are fused through channel splicing to obtain a fused feature map.

6. The method for coronary artery image segmentation based on multi-label and segmentation network according to claim 5, characterized in that: Predicting a coronary artery image on the fused feature map through an integrated model, wherein the integrated model includes a 3*3 two-dimensional convolutional layer, a ReLU function activation layer, a fully connected layer, and a residual structure; In the integrated model, the fused feature map is sequentially passed through a 3*3 two-dimensional convolutional layer and a ReLU function activation layer to extract a high-level semantic feature map, the first segmentation feature map and the high-level semantic feature map are superimposed through a residual structure, and finally the predicted coronary artery two-dimensional image is output through a fully connected layer.

Images