A medical image segmentation method and system based on feature reuse and attention gate

Abstract

The present invention discloses a medical image segmentation method and system based on feature reuse and attention gate. The method separates the parts of interest in the medical image by constructing an image segmentation model based on the Unet structure. The image segmentation model based on the Unet structure is based on the Unet network structure and includes an encoder, a decoder, an ASPP module and an attention gate. The encoder part introduces a densely connected convolutional network block to achieve feature reuse in the encoding process, expands the receptive field through the ASPP module, and performs multi-scale feature extraction on the output of the encoder. An attention gate is also set between the jump connection of the encoder and the decoder to assign different weights to different areas in the feature map, so that the model can focus more on the parts of interest. The method can improve the segmentation accuracy of medical images under the condition that the network operation speed remains unchanged.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, relates to medical image segmentation, and is a medical image segmentation method based on feature reuse and attention gate and a system thereof. Background Art

[0002] Medical image segmentation refers to the process of pixel-by-pixel segmentation of regions of interest in medical images. The Unet network is a classic neural network in this field, consisting of an encoder and a decoder. The encoder consists of a series of convolutional layers that extract semantic and detailed information at different levels. The decoder restores the image resolution through upsampling and outputs the segmentation result. Thanks to the skip connections between the encoder and decoder, high-level and low-level features can be combined in the Unet network, minimizing the loss of feature pixels. Therefore, the Unet network is well-suited for medical image segmentation tasks.

[0003] However, the encoding portion of the UNet network typically uses standard convolution with a regular kernel size and a fixed receptive field. This results in the feature pixels being extracted but unable to reflect the true state of the object. The extracted information is limited, restricting improvements in segmentation accuracy. Furthermore, while skip connections can reduce the loss of feature pixels, the lack of feature reuse in the encoding portion still results in information loss, which cannot be recovered during upsampling. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention proposes a medical image segmentation method and system based on feature reuse and attention gate. Based on the Unet network, the encoding part is modified and a feature reuse mechanism is introduced to improve the segmentation accuracy of medical images.

[0005] A medical image segmentation method based on feature reuse and attention gate includes the following steps:

[0006] Step 1: Construct an image segmentation model based on the Unet structure, including an encoder, a decoder, an ASPP module, and an attention gate. The encoder includes four coding layers, namely, a convolutional layer, DenseBlock1, DenseBlock2, and DenseBlock3 from top to bottom. The convolutional layer is connected to DenseBlock1 through a downsampling operation, and the DenseBlocks are connected through transitions. The number of layers of DenseBlock1 to DenseBlock3 is 6, 12, and 24 respectively.

[0007] The convolution layer performs a 3*3 convolution and ReLu activation operation on the input data, and the DenseBlock performs L nonlinear transformation operations on the input data, where L is the number of DenseBlock layers.

[0008] The Transition performs BN, ReLu activation, 1*1 convolution and average pooling operations in sequence;

[0009] Step 2: Input the image to be segmented into the image segmentation model constructed in step 1, and encode the image to be segmented through the encoder to obtain sub-feature maps E1 to E4 with channel numbers D, (D+32*6), (D+32*(6+12)), and (D+32*(6+12+24)), where D is the number of channels of the image to be segmented;

[0010] The sub-feature map E i =H([x0,x1,...,x l ,...,x L-1 ]), i=1,2,3,4. Where H() represents the nonlinear transformation operation of BN, ReLu activation, 1*1 convolution, BN, ReLu activation and 3*3 convolution on the input data. x0 represents the input data of DenseBlock, x l Represents the output of the lth layer in DenseBlock, x l =H([x0,x1,...,x l-1 ]), l=1,2,...,L. [x0,x1,...,x l-1 ] indicates connection in the channel dimension;

[0011] Step 3: Input the sub-feature map E4 obtained in step 2 into the ASPP module to obtain a multi-scale feature map M;

[0012] The ASPP module first reduces the dimension of the sub-feature map E4 through a 1*1 ordinary convolution operation; then extracts features of different scales from the sub-feature map E4 through three 3*3 dilated convolutions with dilation rates of 1, 2, and 3 respectively; then extracts features from the sub-feature map E4 through a pooling layer, a 1*1 ordinary convolution, and an upsampling operation; finally, the outputs of the above operations are superimposed and then subjected to a 1*1 ordinary convolution operation to obtain a multi-scale feature map M;

[0013] Step 4: Use the decoder to upsample and deconvolve the multi-scale feature map M and the sub-feature map after the attention gate, and output the image segmentation result G1; the decoder includes three decoding layers, and the decoding layer performs two 3*3 deconvolution and ReLu activation operations on the input, and outputs G3, G2, and G1 from bottom to top; the input of each decoding layer is composed of the output of the previous decoding layer after upsampling and the output of the attention gate; among them, the input of the lowest decoding layer is the multi-scale feature map M after upsampling, and the output of the multi-scale feature map M and the sub-feature map E3 after the attention gate is connected;

[0014] The attention gate receives the output E of the encoding layer and the output G of the decoding layer or the multi-scale feature map M, and outputs the weighted sub-feature map E i '=Rsampler(σ2(φ(σ1(W E E i +W G G i ))))E i , G4=M; where σ1 and σ2 represent ReLu activation and Sigmoid activation functions respectively, W E 、W G They represent convolution with a stride of 2 and ordinary convolution, φ represents ordinary convolution, and Rsampler represents a resampling operation.

[0015] A medical image segmentation system based on feature reuse and attention gate, which is used to implement the above-mentioned image segmentation method, including an image acquisition module, an image segmentation module and a result display module;

[0016] The image acquisition module is used to acquire the medical image to be segmented;

[0017] The image segmentation module includes a trained Unet-based image segmentation model, which receives the medical image to be segmented, extracts sub-feature maps at different levels through an encoder, further extracts multi-scale feature maps through an ASPP module, assigns weights to different regions in the feature map through an attention gate, and finally restores it to the same size as the input image through a decoder, outputting the segmentation result;

[0018] The result display module receives the segmentation result output by the image segmentation module and displays it on the screen.

[0019] The present invention has the following beneficial effects:

[0020] This method uses DenseBlock in DenseNet to replace the encoding layer in the Unet network, reduces the loss of feature pixels by reusing features, and designs an attention gate structure on the jump connection path between the encoder and decoder to highlight local features in specific areas, allowing the network to pay more attention to the image locations of interest. Before entering the decoder, the ASPP module is used to expand the receptive field and extract multi-scale features. This method can improve the segmentation accuracy of medical images while keeping the network running speed unchanged. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of image segmentation model based on Unet structure;

[0022] Figure 2 This is a schematic diagram of the DenseBlock structure;

[0023] Figure 3 Schematic diagram of a medical image to be segmented in an embodiment;

[0024] Figure 4 This is a schematic diagram of the ASPP module structure;

[0025] Figure 5 Schematic diagram of the attention gate structure;

[0026] Figure 6 is the image segmentation result in the embodiment. DETAILED DESCRIPTION

[0027] This example uses a publicly available liver tumor dataset, along with accompanying figures, to further illustrate the present invention. The segmentation results are merely indications of tumor location and cannot be relied upon as definitive diagnostic results. It should be noted that the following examples are intended only to illustrate specific implementations of the present invention and are not exhaustive. All other implementations derived by persons of ordinary skill in the art based on these examples without inventive effort are intended to fall within the scope of this invention.

[0028] A medical image segmentation method based on feature reuse and attention gate includes the following steps:

[0029] Step 1: Build Figure 1The image segmentation model based on the Unet structure shown in the figure includes an encoder, a decoder, an ASPP module and an attention gate. The encoder includes four encoding layers, which are the convolution layer, DenseBlock1, DenseBlock2 and DenseBlock3 from top to bottom. The convolution layer is connected to DenseBlock1 through a downsampling operation, and the DenseBlocks are connected through Transition. The number of layers of DenseBlock1 to 3 are 6, 12 and 24 respectively. The convolution layer performs a convolution of size 3*3 and a ReLu activation operation on the input data, as shown in FIG. Figure 2 As shown in the figure, DenseBlock performs L nonlinear transformation operations on the input data, where L is the number of layers of DenseBlock. The nonlinear transformation operations include BN, ReLu activation, 1*1 convolution, BN, ReLu activation and 3*3 convolution. The Transition performs BN, ReLu activation, 1*1 convolution and average pooling operations in sequence.

[0030] Step 2: Figure 3 The 64-channel image to be segmented with a size of 512*512 pixels is input into the image segmentation model constructed in step 1. The image to be segmented is encoded by the encoder to obtain sub-feature maps E1~E4 with 64, 256, 640 and 1408 channels respectively. After each encoding layer, the size of the image is reduced to 1 / 4 of the original size.

[0031] The sub-feature map E i =H([x0,x1,...,x l ,...,x L-1 ]), i=1,2,3,4. Where H() represents the nonlinear transformation operation on the input data, x0 represents the input data of DenseBlock, x l Represents the output of the lth layer in DenseBlock, x l =H([x0,x1,...,x l-1 ]), l=1,2,...,L. [x0,x1,...,x l-1 ] indicates connection in the channel dimension.

[0032] Step 3: Input the sub-feature graph E4 obtained in step 2 into Figure 4In the ASPP module shown, the ASPP module first reduces the dimension of the sub-feature map E4 through a normal convolution operation of size 1*1, and then extracts features of different scales from the sub-feature map E4 through three hole convolutions of size 3*3 and expansion rates of 1, 2, and 3 respectively. Then, the sub-feature map E4 is subjected to feature extraction through a pooling layer, a 1*1 normal convolution and an upsampling operation. The outputs of the above operations are superimposed and then a 1*1 normal convolution operation is performed to obtain a multi-scale feature map M.

[0033] Step 4: Use the decoder to restore the sub-feature map to the size of the input image and output the segmentation result G1. The decoder includes three decoding layers, which perform an upsampling operation and two 3*3 deconvolution operations on the input, outputting G3, G2, and G1 from bottom to top. The third decoding layer receives the multi-scale feature map M output by the ASPP module and concatenates it with E3' after upsampling, outputting G3 with a size of 128*128 and 640 channels. The second decoding layer receives the G3 output by the third decoding layer and concatenates it with E2' after upsampling, outputting G2 with a size of 256*256 and 256 channels. The first decoding layer receives the G2 output by the second decoding layer and concatenates it with E1' after upsampling, outputting G1 with a size of 512*512 and 64 channels as the image segmentation result. This segmentation result is only used as a reference and cannot be used as the final diagnosis result.

[0034] like Figure 5 As shown, the attention gate receives the output E of the encoding layer and the output G of the decoding layer or the multi-scale feature map M, and outputs the weighted sub-feature map E i '=Rsampler(σ2(φ(σ1(W E E i +W G G i ))))E i , G4=M. Where σ1 and σ2 represent ReLu activation and Sigmoid activation functions respectively, W E 、W G They represent convolution with a stride of 2 and ordinary convolution, φ represents ordinary convolution, and Rsampler represents a resampling operation.

[0035] right Figure 3 The image to be segmented is marked and used as a label. In this embodiment, 400 different images to be segmented and their labels are selected and input into the image segmentation model based on the Unet structure for model training. Then, only the images to be segmented are input into the trained model, and the output segmentation results are as follows: Figure 6As shown, in order to illustrate the effectiveness of this method, the same image is input into the traditional Unet model, and the intersection-over-union ratio of the segmentation results output by the two models and the true label of the image is calculated. The comparison results are shown in the following table:

[0036] Unet model Image segmentation model based on Unet structure Iou1 0.926594 0.924557 Iou2 0.929274 0.930188 Iou3 0.940802 0.936039 Iou4 0.943500 0.933410 Iou5 0.919342 0.947328 Iou6 0.929326 0.945045 Iou7 0.910854 0.951882 Iou8 0.932546 0.937782 Iou9 0.951059 0.942199 Iou10 0.941914 0.942365 Iou11 0.937921 0.956051 Iou12 0.942106 0.919185 Iou13 0.965628 0.947861 Iou14 0.922584 0.955842 Iou15 0.945833 0.935879 Miou 0.874210 0.9403742

[0037] As can be seen from the above table, this method can increase the intersection-over-union ratio from 0.87 to 0.94, and the network training speed remains basically unchanged.

Claims

1. A medical image segmentation method based on feature reuse and attention gate, characterized by: The following steps are involved: Step 1: Build an image segmentation model based on the Unet structure, including encoder, decoder, ASPP module and attention gate; The encoder includes 4 coding layers, which are convolutional layer, DenseBlock1, DenseBlock2 and DenseBlock3 from top to bottom. The convolutional layer is connected to DenseBlock1 through downsampling operation, and DenseBlocks are connected through Transition. The number of layers of DenseBlock1 to DenseBlock3 is 6, 12 and 24 respectively. Step 2: Input the image to be segmented into the image segmentation model constructed in step 1, and encode the image to be segmented through the encoder to obtain sub-feature maps E1 to E4 with channel numbers D, (D+32*6), (D+32*(6+12)), and (D+32*(6+12+24)), where D is the number of channels of the image to be segmented; Step 3: Input the sub-feature map E4 obtained in step 2 into the ASPP module to obtain a multi-scale feature map M; Step 4: Use the decoder to upsample and deconvolve the multi-scale feature map M and the sub-feature map after the attention gate, and output the image segmentation result G1; The decoder includes three decoding layers, which perform two 3*3 deconvolution and ReLu activation operations on the input, and output G3, G2, and G1 from bottom to top. The input of each decoding layer is composed of the output of the previous decoding layer after upsampling and the output of the attention gate. The input of the lowest decoding layer is the multi-scale feature map M after upsampling, and the output of the multi-scale feature map M and the sub-feature map E3 after passing through the attention gate. The attention gate receives the output E of the encoding layer and the output G of the decoding layer or the multi-scale feature map M, and outputs the weighted sub-feature map E i '=Rsampler(σ2(φ(σ1(W E E i +W G G i ))))E i , G4=M; where σ1 and σ2 represent ReLu activation and Sigmoid activation functions respectively, W E 、W G They represent convolution with a stride of 2 and ordinary convolution, φ represents ordinary convolution, and Rsampler represents a resampling operation.

2. The medical image segmentation method based on feature reuse and attention gate according to claim 1, characterized in that: In the encoder, the convolution layer performs a 3*3 convolution and ReLu activation operation on the input data; the DenseBlock performs L nonlinear transformation operations on the input data, where L is the number of DenseBlock layers.

3. The medical image segmentation method based on feature reuse and attention gate according to claim 1, characterized in that: The Transition performs BN, ReLu activation, 1*1 convolution and average pooling operations in sequence.

4. The medical image segmentation method based on feature reuse and attention gate according to claim 1, characterized in that: The sub-feature map E i =H([x0,x1,...,x l ,...,x L-1 ]), i = 1, 2, 3, 4; where H() represents the nonlinear transformation of the input data, which is BN, ReLu activation, 1*1 convolution, BN, ReLu activation and 3*3 convolution; x0 represents the input data of DenseBlock, x l Represents the output of the lth layer in DenseBlock, x l =H([x0,x1,...,x l-1 ]), l=1,2,...,L, the number of layers of LDenseBlock; [x0,x1,...,x l-1 ] indicates connection in the channel dimension.

5. The medical image segmentation method based on feature reuse and attention gate according to claim 1, characterized in that: The ASPP module first reduces the dimension of the sub-feature map E4 through a normal convolution operation of size 1*1; then extracts features of different scales from the sub-feature map E4 through three 3*3 dilated convolutions; then extracts features from the sub-feature map E4 through a pooling layer, a 1*1 normal convolution and an upsampling operation; finally, the outputs of the above operations are superimposed and then a 1*1 normal convolution operation is performed to obtain a multi-scale feature map M.

6. The medical image segmentation method based on feature reuse and attention gate according to claim 5, characterized in that: In the ASPP module, the dilation rates of the dilated convolution are 1, 2, and 3 respectively.

7. A segmentation system for a medical image segmentation method based on feature reuse and attention gate according to any one of claims 1 to 6, characterized in that: It includes image acquisition module, image segmentation module and result display module; The image acquisition module is used to acquire the medical image to be segmented; The image segmentation module includes a trained Unet-based image segmentation model, which receives the medical image to be segmented, extracts sub-feature maps at different levels through an encoder, further extracts multi-scale feature maps through an ASPP module, assigns weights to different regions in the feature map through an attention gate, and finally restores it to the same size as the input image through a decoder, outputting the segmentation result; The result display module receives the segmentation result output by the image segmentation module and displays it on the screen.

Images