Abdominal CT image segmentation method based on branch growth neural network architecture search

Abstract

The invention discloses an abdominal CT image segmentation method based on branch growth neural network architecture search, and the method comprises the steps: constructing a search space network based on a local width branch and a global depth branch; alternately performing access of the local width branch and the global depth branch on a growth down-sampling layer of the encoder based on the search space network, and constructing a grown candidate network; training and testing the grown candidate network to obtain an image segmentation model; and acquiring to-be-detected abdominal CT image data, and inputting the data into the image segmentation model to complete abdominal CT image segmentation. According to the invention, the accuracy of abdomen CT image segmentation is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to an abdominal CT image segmentation method based on branch-growth neural network architecture search. Background Art

[0002] Medical image segmentation is a key step in medical image analysis. Convolutional neural networks (CNNs) have been widely used in medical image segmentation, with U-Net, an encoder-decoder structure, as the representative. There are many subsequent improved models based on U-Net. For example, Xiao et al. proposed Res-UNet by combining residual blocks with U-Net. By adding residual connections between convolutional blocks, satisfactory results were achieved in the retinal vascular segmentation task. To enhance the extraction of contextual information, Huang et al. proposed FECC-Net, which added ASPP modules and enhanced encoder branches to U-Net, improving the accuracy of brain MRI image segmentation. Oktay et al. combined the attention mechanism with U-Net and proposed Attention U-Net, which achieved good performance in the abdominal pancreas segmentation task. Although CNN has made significant progress in medical segmentation tasks, it still has some inherent defects in capturing global features. Its receptive field is limited, it cannot capture global information, and its global modeling ability is insufficient. To solve this problem, researchers have proposed a variety of improvement methods, such as dense connection, image pyramid, self-attention and other modules. At the same time, inspired by the success of Transformer in the field of natural language processing, a visual attention model (Vision Transformer, ViT) was proposed based on the Transformer structure, which uses the self-attention mechanism to capture the dependencies between each position in the input sequence and successfully solves the long-distance dependency problem. Learning all image blocks in the image through the self-attention mechanism to obtain global dependencies has also performed well in the field of medical image segmentation. For example, Chen et al. first used the global context modeling ability of Transformer and the high-resolution positioning ability of U-Net in the field of medical image segmentation, and achieved good performance in multi-organ segmentation tasks; Zhang et al. proposed a model with two encoders in parallel, Transformer branch and CNN branch, and achieved good segmentation results in segmentation tasks such as colon polyps, skin, and prostate. Although these methods have achieved good results in the field of medical segmentation, these networks are all manually designed and have poor adaptability. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes an abdominal CT image segmentation method based on branch growing neural network architecture search to improve the accuracy of abdominal CT image segmentation.

[0004] To achieve the above object, the present invention provides an abdominal CT image segmentation method based on branch growing neural network architecture search, comprising:

[0005] Based on local width branches and global depth branches, a search space network is constructed;

[0006] Alternately accessing the local width branch and the global depth branch to the growable downsampling layer of the encoder based on the search space network to construct a grown candidate network;

[0007] Training and testing the grown candidate network to obtain an image segmentation model;

[0008] Obtain the abdominal CT image data to be tested, input the image segmentation model, and complete the abdominal CT image segmentation.

[0009] Optionally, the local width branch Conv and Res2net are composed of a Conv+Res2net×N structure, wherein Res2net×N represents N Res2net modules.

[0010] Optionally, the global depth branch consists of a Transformer layer, which captures long-distance dependent global features globally through a self-attention mechanism.

[0011] Optionally, the growable downsampling layer consists of a pooling layer, a convolutional layer, and a branch access layer;

[0012] The pooling layer is used to reduce the scale of the feature map to facilitate the extraction of features of different scales;

[0013] The convolution layer is used to extract local features;

[0014] The branch access layer is used to access new branches to enable the network to continue to grow.

[0015] Optionally, the branch access layer sets a visited attribute to distinguish whether it is accessed by a branch, including a situation where there is no branch access and a situation where there is branch access;

[0016] In the case of no branch access, the branch access layer will be directly skipped, and the result processed by the convolution layer will be used as the final output;

[0017] In the case of branch access, the output processed by the convolution layer is fused with the access branch through the splicing layer, and the dimension is normalized through convolution.

[0018] Optionally, the implementation process of the grown candidate network includes:

[0019] The first generation growth initial encoder searches for the connected growable downsampling layer from the encoder and connects to the first local width branch, and after growth, the first generation growth of the encoder is obtained; the first generation growth of the encoder grows from the depth direction, and connects to the first global depth branch after the last growable downsampling layer to obtain the second generation growth of the encoder; the second generation growth of the encoder continues to search for the connected growable downsampling layer and connects to the new local width branch to obtain the third generation growth of the encoder; the third generation growth of the encoder connects to a new global depth branch after the last global depth branch, and repeats this process.

[0020] Technical effect of the present invention: The present invention discloses an abdominal CT image segmentation method based on branch growth neural network architecture search. In order to automatically construct a network model, the present invention proposes a local-global branch iterative growth algorithm, so that the network grows on the previous generation network model, realizing the dynamic expansion of the network architecture, and each generation of training is initialized based on the network parameters of the previous generation, thereby reducing the training time. In order to enable the network to effectively acquire multi-scale features and global context information, the present invention designs growth branches based on the Res2net module and the Transformer module, grows the network in both depth and width, and then obtains richer semantic information. The generated network was verified on the Synapse multi-organ CT image dataset. The experimental results show that the evaluation accuracy of the present invention is higher than that of the manually designed image segmentation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 It is a flowchart of an abdominal CT image segmentation method based on branch-growing neural network architecture search according to an embodiment of the present invention;

[0023] Figure 2 Schematic diagram of local width branch and global depth branch in an embodiment of the present invention, wherein (a) is a local width branch and (b) is a global depth branch;

[0024] Figure 3 A schematic diagram of a growable downsampling layer according to an embodiment of the present invention;

[0025] Figure 4 Schematic diagram of size change in an embodiment of the present invention, where (a) is access without branches and (b) is access with branches;

[0026] Figure 5 A schematic diagram of a local-global branch iterative growth strategy according to an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of the network architecture of LG-BG-NAS according to an embodiment of the present invention;

[0028] Figure 7 Schematic diagram of the local-global branch iterative growth process according to an embodiment of the present invention;

[0029] Figure 8 This is a schematic diagram for visually comparing the segmentation results of different methods on the Synapse dataset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0031] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] like Figure 1 As shown, this embodiment provides an abdominal CT image segmentation method based on branch-growing neural network architecture search, including:

[0033] Based on local width branches and global depth branches, a search space network is constructed;

[0034] Based on the search space network, the growable downsampling layer of the encoder is alternately connected with the local width branch and the global depth branch to construct a candidate network after growth;

[0035] Train and test the grown candidate network to obtain an image segmentation model;

[0036] Obtain the abdominal CT image data to be tested, input the image segmentation model, and complete the abdominal CT image segmentation.

[0037] Specifically, this embodiment proposes a local-global branch iterative growth algorithm, which enables the network to grow on the previous generation network model, realizes the dynamic expansion of the network architecture, and each generation of training is initialized based on the network parameters of the previous generation, thereby reducing the training time. In order to enable the network to effectively obtain multi-scale features and global context information, this embodiment designs growth branches based on the Res2net module and the Transformer module, grows the network in both depth and width, and obtains richer semantic information.

[0038] Further, such as Figure 2 The local width branch (LB) shown in (a) consists of Conv and Res2net, and adopts the Conv+Res2net×N structure, where Res2net×N represents N Res2net modules. The main purpose is to extract local features of the input data. In order to capture richer local features using multi-branch convolutional layers, LB generally grows in the width direction of the encoder.

[0039] Further, such as Figure 2 The global depth branch (GB) shown in (b) consists of a Transformer layer, which consists of layer normalization, multi-head attention, layer normalization and a multi-layer perceptron. The Transformer layer captures global features with long-distance dependencies globally through a self-attention mechanism.

[0040] Furthermore, in order to dynamically insert branches into the encoder to achieve branch growth, a growable downsampling layer (Grownable pooling, GPool) is designed. Figure 3 As shown in the figure, GPool consists of a pooling layer, a convolutional layer, and a branch access layer (Accesslayer), which allows branches to access the network through the Access layer, allowing the network to continue to grow. A visited attribute is set in the Access layer to distinguish whether it is accessed by a branch. When no branch is accessed (visited = false), the Access layer will be skipped directly, and the result processed by the Conv layer will be used as the final output. When a branch is accessed (visited = true), the output processed by the Conv layer will be concatenated (Concat) with the access branch, and the dimension will be normalized through 1×1 convolution. Assume that the input size is (3, H, W), and the size change of the feature map is as follows Figure 4 As shown. Figure 4 In (a), there is no branch access, the input passes through the convolution layer, the pooling layer in GPool and the convolution layer, and the final output size is 2C, H / 2, W / 2; when there is a branch access, as shown in Figure 4 As shown in (b), the input passes through the convolutional layer and the Res2net layer, so that the final output is also 2C, H / 2, W / 2, so as to facilitate concat with the original branch.

[0041] In order to improve the local and global feature extraction capabilities of candidate neural network encoders in the search, a local-global branch iterative growth strategy is adopted. First, the growth of LB is used to extract richer fine-grained local features at similar scales, and then GB is grown to better capture global feature relationships. The above branch growth process is repeated until the algorithm converges. The local-global branch iterative growth strategy is as follows Figure 5 As shown in Figure 1, LB growth is performed first, followed by GB growth, and LB is connected to the network from shallow to deep. m Indicates the encoder's mth generation growth, with the initial encoder being G0. First, the first generation of growth G0 searches for accessible GPools from the encoder and accesses the first local width branch (LB1), and then obtains G1; then G1 grows in the depth direction, that is, after the last GPool, it accesses the first global depth branch (GB1) to obtain G2; G2 continues to search for accessible GPool layers and accesses new LB m / 2+1 , get G3; then G3 accesses the new one after the last GB This process is repeated. To ensure that the two inputs of the Access layer (from LB and GPool) have the same shape, LB i There should be a Res2net module in it.

[0042] The training and testing of the candidate network after growth specifically includes: using the encoding-decoding structure U-Net as the initial network, after multiple generations of LG-BG-NAS growth, the trained encoding-decoding architecture neural network is directly used for testing, the structure is as follows Figure 6 As shown, the decoder part is consistent with U-Net, consisting of L upsampling and a 1×1 convolution. The network does not require retraining or parameter adjustment, and directly segments the images in the test dataset.

[0043] A specific experimental example of this embodiment is as follows:

[0044] The experiment was conducted using Pytorch on a computer equipped with an Intel i5 processor, DDR416G×2 memory and RTX 3060 GPU. The average similarity coefficient (Dice) and average Hausdorff distance (HD) were used to compare with SOTA methods such as Att-UNet, TransUnet, and SwinUnet to verify the effectiveness of the method. The network structure of U-Net is the same as the initial encoder in the method proposed in this embodiment; TransUnet is a hybrid model that combines CNN and Transformer, thus having the advantages of both CNN and Transformers; SwinUnet is a pure Transformer model. Therefore, these methods were selected for comparison with the proposed method. The experiment was trained and tested using the Synapse multi-organ dataset. The Synapse dataset includes 30 cases of 3779 abdominal axial clinical CT images, 18 samples are divided into training sets, and 12 samples are divided into test sets. All 3D volumes were segmented slice by slice to evaluate 9 abdominal organs (aorta, gallbladder, spleen, left kidney, right kidney, liver, pancreas, spleen, and stomach).

[0045] The experiment uses 224×224 images as input, batch size is 3, SGD optimization algorithm is used for training, initial learning rate is 0.01, momentum is set to 0.9, weight decay is 0.0001, and cosine annealing is used to update the learning rate. The epochs of training G0 is 200, and the epochs of G1-G5 is 60. The threshold T is set to 0.1 and L is set to 4.

[0046] The experiment uses the average Dice coefficient and average HD to evaluate the segmentation accuracy of the model. The Dice coefficient is a set similarity index used to calculate the similarity between two samples. The calculation formula is as follows:

[0047]

[0048] Among them, |X| and |Y| represent the true label and the predicted result respectively, and |X∩Y| represents their intersection.

[0049] HD is also a set similarity index. It defines the similarity between two sets of points by calculating the distance between them. The smaller the value, the higher the similarity between the two sets. The calculation formula of HD is as follows:

[0050] HD(A,B)=max(h(A,B),h(B,A)) (2)

[0051]

[0052] Among them, ‖·‖ is the distance form between point set A and point set B, h(A, B) and h(B, A) are the one-way Hausdorff distances from set A to set B and from set B to set A, respectively.

[0053] First, we search for candidate networks on the Synapse dataset. Since this dataset does not have an independent validation set, we directly validate it on its test set. The network growth process is as follows: Figure 7 As shown in the figure, first, the width direction is grown, and the accessible GPool layer is searched from GPool1 in the encoder. Initially, GPool1 has no access branch, so a corresponding LB is generated and connected to GPool1, so that G0 grows to G1 and is trained. The network parameters of G1 are initialized according to the trained G0, so there is no need to retrain the entire network, only the newly connected branches need to be retrained. Then, the trained G1 is grown in the depth direction, that is, 1 GB is grown after the last GPool layer, so that G1 grows to G2. Similarly, G2 is initialized and trained according to the network parameters of the trained G1, and then G2 continues to grow in the width direction to generate G3 and train it, and then grows in the depth direction on G3 to obtain G4. This growth process is iterated continuously until the average Dice improvement is less than the threshold T.

[0054] Table 1 shows the average Dice, training time, number of parameters, and amount of computation for G0-G5. First, since the training epochs of G0 is 200, the training time will be relatively long. From G1 to G5, the training time of each generation of networks increases with the access of branches. The training time will increase significantly after accessing LB, and the training time will be slightly affected after accessing GB. The number of parameters and the amount of computation both show a trend of gradual increase from G0 to G5. The increase in the number of parameters is more significant after accessing GB, which is caused by the multi-head self-attention mechanism of Transformer. In addition, the amount of computation increases more significantly after accessing LB, which is caused by the parallel convolution operation of the Res2Net module. Overall, the improvement in the number of parameters and the amount of computation of each generation of networks is quite obvious, while the training time will fluctuate greatly after accessing LB, which reflects the increase in computing resources and time consumption while improving network performance. From the average Dice in the table, we can see that with the increase of growth generations, the average Dice growth rate decreases. After 4 generations of growth, the average Dice is basically stable. When it grows to the G5 model, the average Dice value is less than the threshold T compared with G4, and the HD value is not as good as G4. Therefore, G4 is selected as the best model.

[0055] Table 1

[0056]

[0057] Table 2 compares the test results of LG-BG-NAS with traditional CNN segmentation models such as V-Net, DARR, R50 U-Net and Att-UNet on the Synapse dataset. LG-BG-NAS performs best, with G4 having an average Dice value and HD value of 83.07% and 18.87mm on the test set, respectively, while Att-UNet, which ranks second among these methods, adds the attention mechanism to the jump connection, with an average Dice value and HD value of 77.77% and 36.02mm, respectively.

[0058] In addition, Table 2 also compares the test results of the CNN+Transformer hybrid model on the Synapse dataset, including R50 ViT, TransUnet, Swin UNet and TransClaw U-Net. Similarly, LG-BG-NAS performed best. The second-ranked Swin Unet adopts the encoding-decoding architecture of pure Transformer, with an average Dice value and HD value of 79.13% and 21.55mm, respectively. The third-ranked TransClaw U-Net combines the Transformer with the Claw Unet architecture, with an average Dice value and HD value of 78.09% and 26.38mm. TransUnet is similar to the LG-BG-NAS architecture, but the CNN and Transformer modules are fixed structures, and their average Dice value and HD value are 77.48% and 31.69mm, respectively. The present invention also tested other networks in the growth process. When G0 grows to G4, the average Dice rises rapidly; when G4 grows to G5, the average Dice only increases by 0.05%, reaching the algorithm convergence condition and no longer growing.

[0059] Table 2

[0060]

[0061] Figure 8 Shows a visual comparison of the segmentation results of different methods on the Synapse dataset. Figure 8 It can be seen that the Att-Unet method is most prone to over-segmentation or under-segmentation, such as the over-segmentation of the gallbladder (green) in the first sample (row 1) and the under-segmentation of the stomach (light blue) in the second sample (row 2). TransUnet and Swin Unet also have under-segmentation, such as the pancreas (cyan) in the third sample (row 3). Compared with other models that incorporate Transformer, LG-BG-NAS uses LB to obtain multi-scale feature information and GB to better capture global feature relationships, achieving better segmentation results than other methods.

[0062] The present invention discloses an abdominal CT image segmentation method based on branch growth neural network architecture search. In order to automatically construct a network model, the present invention proposes a local-global branch iterative growth algorithm, which enables the network to grow on the previous generation network model, realizes the dynamic expansion of the network architecture, and each generation of training is initialized based on the network parameters of the previous generation, thereby reducing the training time. In order to enable the network to effectively obtain multi-scale features and global context information, the present invention designs growth branches based on the Res2net module and the Transformer module, grows the network in both depth and width, and then obtains richer semantic information. The generated network was verified on the Synapse multi-organ CT image dataset. The experimental results show that the evaluation accuracy of the present invention is higher than that of the manually designed image segmentation method.

[0063] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An abdominal CT image segmentation method based on branch-growing neural network architecture search, characterized in that: include: Based on the local width branch and the global depth branch, a search space network is constructed; Alternately accessing the local width branch and the global depth branch to the growable downsampling layer of the encoder based on the search space network to construct a grown candidate network; Training and testing the grown candidate network to obtain an image segmentation model; Obtain the abdominal CT image data to be tested, input the image segmentation model, and complete the abdominal CT image segmentation.

2. The abdominal CT image segmentation method based on branch-growing neural network architecture search according to claim 1, characterized in that: The local width branch Conv and Res2net are composed, and adopt a Conv+Res2net×N structure, where Res2net×N represents N Res2net modules.

3. The abdominal CT image segmentation method based on branch-growing neural network architecture search according to claim 1, characterized in that: The global deep branch consists of Transformer layers, which capture global features of long-range dependencies globally through a self-attention mechanism.

4. The abdominal CT image segmentation method based on branch-growing neural network architecture search according to claim 1, characterized in that: The growable downsampling layer consists of a pooling layer, a convolutional layer, and a branch access layer; The pooling layer is used to reduce the scale of the feature map to facilitate the extraction of features of different scales; The convolution layer is used to extract local features; The branch access layer is used to access new branches to enable the network to continue to grow.

5. The abdominal CT image segmentation method based on branch-growing neural network architecture search according to claim 4, characterized in that: The branch access layer sets a visited attribute to distinguish whether it is accessed by a branch, including a situation where there is no branch access and a situation where there is a branch access; In the case of no branch access, the branch access layer will be directly skipped, and the result processed by the convolution layer will be used as the final output; In the case of branch access, the output processed by the convolution layer is fused with the access branch through the splicing layer, and the dimension is normalized through convolution.

6. The abdominal CT image segmentation method based on branch-growing neural network architecture search according to claim 5, characterized in that: The implementation process of the grown candidate network includes: The first generation growth initial encoder searches for the connected growable downsampling layer from the encoder and connects to the first local width branch, and after growth, the first generation growth of the encoder is obtained; the first generation growth of the encoder grows from the depth direction, and connects to the first global depth branch after the last growable downsampling layer to obtain the second generation growth of the encoder; the second generation growth of the encoder continues to search for the connected growable downsampling layer and connects to the new local width branch to obtain the third generation growth of the encoder; the third generation growth of the encoder connects to a new global depth branch after the last global depth branch, and repeats this process.

Images