A medical image segmentation method based on an autoencoder and a generative adversarial network

Abstract

The application discloses a medical image segmentation method based on a self-encoder and a generative adversarial network, wherein a self-encoder including an encoder and a second decoder is arranged to reconstruct an input image, thereby providing an additional multi-scale feature map for a segmentation model; a suitable skip connection layer is designed to provide more abundant multi-scale features for the generation of a segmented image; a generative adversarial network is used to introduce an additional loss for the generated segmented image, so that the network can generate a better segmented image, and the quality of the generated segmented image is higher, thereby improving the precision of medical image segmentation. The encoder constructed by using a depth separable convolution and an attention mechanism can extract more abundant features with less parameter quantity and computation amount.

Description

Technical Field

[0001] This invention belongs to the field of deep learning image segmentation, and more specifically, relates to a medical image segmentation method based on autoencoders and generative adversarial networks. Background Technology

[0002] Organ / lesion segmentation in medical images is a crucial step in clinical applications. Image segmentation technology can extract the regions containing organs or lesions in medical images to assist medical experts in making corresponding medical diagnoses.

[0003] Existing methods for organ / lesion segmentation in medical images typically employ deep network models such as UNet and UNet++, but the accuracy of the image segmentation results needs improvement. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of existing technologies, the present invention provides a medical image segmentation method based on autoencoders and generative adversarial networks, thereby solving the technical problem of low accuracy of existing segmentation methods.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a medical image segmentation method based on an autoencoder and a generative adversarial network is provided, comprising:

[0006] Training phase:

[0007] Using medical images as training samples and segmented images of lesions or organs from the medical images as labels, the segmentation model and discriminator are trained.

[0008] The segmentation model includes an encoder, a first decoder, and a second decoder. The encoder includes convolutional layers and Gaussian downsampling layers connected in sequence, and the decoder includes upsampling layers and convolutional layers connected in sequence.

[0009] The features corresponding to each stage in the encoder are input to the network layer of the corresponding stage and the previous stage in the first decoder through a skip connection layer; the features corresponding to each stage in the first decoder are input to the network layers of the last two stages of each stage through a skip connection layer; the features corresponding to each stage in the second decoder are input to the network layer of the corresponding stage in the first decoder through a skip connection layer; wherein, each stage in the encoder is divided into downsampling layers, and each stage in the first and second decoders is divided into upsampling layers.

[0010] The loss function in the training phase includes the reconstruction error function of the reconstructed image of the medical image generated by the second decoder, the segmentation error function of the lesion or organ segmentation image of the medical image generated by the first decoder, and the loss function of the discriminator's true / false value prediction; wherein, in the training phase, the discriminator and the segmentation model together form a generative adversarial network;

[0011] Application phase:

[0012] The medical image to be segmented is input into the trained segmentation model to obtain the segmentation result.

[0013] Preferably, the convolutional layer is a depthwise separable convolutional layer with an added attention mechanism.

[0014] Preferably, during the training phase, the parameters of the segmentation model are updated using the backpropagation algorithm.

[0015] Preferably, the discriminator includes sequentially alternating convolutional layers and Gaussian downsampling layers, as well as global pooling layers and fully connected layers.

[0016] Preferably, the number of upsampling layers and downsampling layers is 4.

[0017] Preferably, with the fifth stage of the encoder as the center, the encoder, the first decoder, and the second decoder have a symmetrical structure.

[0018] According to a second aspect of the present invention, a medical image segmentation system based on an autoencoder and a generative adversarial network is provided, characterized in that it comprises: a computer-readable storage medium and a processor;

[0019] The computer-readable storage medium is used to store executable instructions;

[0020] The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in the first aspect.

[0021] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0022] 1. The medical image segmentation method based on autoencoders and generative adversarial networks provided by this invention reconstructs the input image by setting an autoencoder including an encoder and a second decoder, providing an additional multi-scale feature map for the segmentation model; by using skip connection layers, richer multi-scale features can be provided for the generation of segmented images; using generative adversarial networks can introduce additional loss to the generated segmented images, enabling the network to generate better segmented images, resulting in higher quality segmented images and further improving the accuracy of medical image segmentation.

[0023] 2. The medical image segmentation method based on autoencoders and generative adversarial networks provided by this invention uses an encoder constructed with depthwise separable convolution and attention mechanisms, which can extract richer features with fewer parameters and less computation. Attached Figure Description

[0024] Figure 1 A network architecture diagram of the segmentation model provided in an embodiment of the present invention;

[0025] Figure 2 Network layer information of the encoder provided in the embodiments of the present invention;

[0026] Figure 3 Network layer information of the decoder provided in the embodiments of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0028] In recent years, automated segmentation models based on deep learning have shown remarkable performance. Deep convolutional neural networks have developed many encoder-decoder segmentation models, such as FCNs, UNet, UNet++, and UNet 3+, with UNet being particularly widely used in medical image processing. These networks share similar key structures—encoder-decoder architecture and Skip Connections. The former ensures that the model can extract rich features at different scales and reconstruct segmentation results similar to the original image; the latter enables the joint application of features at different scales from different layers in the encoder and decoder.

[0029] To more effectively extract and utilize features at different scales, a stronger backbone network is needed. Simultaneously, depthwise separable convolutions can be used to reduce the number of network parameters and computational cost; compared to standard convolutions, depthwise separable convolutions significantly reduce the number of network parameters and computational cost while sacrificing only a small amount of accuracy. Furthermore, attention mechanisms can be used to enhance network performance. Autoencoders and generative adversarial networks (GANs) are mainstream methods in image generation, and segmented images in medical image segmentation can also be considered as generated images. Similar to networks like UNet, autoencoders and GANs also have encoder and decoder (referred to as generators in GANs) structures. Both autoencoders and GANs are unsupervised learning methods, meaning they do not require additional labeled data but can provide additional effective information for generating segmented images. Using an autoencoder to reconstruct the input image, the additional encoder can provide a multi-scale feature map. Using GAN strategies can introduce additional loss to the generated segmented image, enabling the network to generate better segmented images. Based on this, this invention provides a medical image segmentation method based on autoencoders and GANs, including:

[0030] Training phase:

[0031] Using medical images as training samples and segmented images of lesions or organs from the medical images as labels, the segmentation model and discriminator are trained.

[0032] The segmentation model includes an encoder, a first decoder, and a second decoder. The encoder includes convolutional layers and Gaussian downsampling layers connected in sequence, and the decoder includes upsampling layers and convolutional layers connected in sequence.

[0033] The features corresponding to each stage in the encoder are input to the network layer of the corresponding stage and the previous stage in the first decoder through a skip connection layer; the features corresponding to each stage in the first decoder are input to the network layers of the last two stages of each stage through a skip connection layer; the features corresponding to each stage in the second decoder are input to the network layer of the corresponding stage in the first decoder through a skip connection layer; wherein, each stage in the encoder is divided into downsampling layers, and each stage in the first and second decoders is divided into upsampling layers.

[0034] The loss function in the training phase includes the reconstruction error function of the reconstructed image of the medical image generated by the second decoder, the segmentation error function of the lesion or organ segmentation image of the medical image generated by the first decoder, and the loss function of the discriminator's true / false value prediction; wherein, in the training phase, the discriminator and the segmentation model (i.e., as a generator) together form a generative adversarial network.

[0035] Application phase:

[0036] The medical image to be segmented is input into the trained segmentation model to obtain the segmentation result.

[0037] Preferably, the convolutional layer is a depthwise separable convolutional layer with an added attention mechanism.

[0038] Preferably, during the training phase, the parameters of the segmentation model are updated using the backpropagation algorithm.

[0039] Preferably, the number of upsampling layers and downsampling layers is 4, that is, the encoder performs 4 upsampling operations, and the first decoder and the second decoder each perform 4 downsampling operations.

[0040] Preferably, with the fifth stage of the encoder as the center, the encoder, the first decoder, and the second decoder have a symmetrical structure.

[0041] Specifically, the segmentation network architecture (i.e., segmentation model) provided by this invention includes a main network and a discriminator network: the main network consists of an encoder and two decoders constructed from depthwise separable convolutions; the discriminator uses the same network structure as the encoder of the main network, with the addition of a global pooling layer and a fully connected layer.

[0042] Furthermore, the network architecture provided by this invention includes two sub-networks, such as... Figure 1 As shown, the left-hand network (the main network, i.e., the segmentation model) consists of an encoder and two decoders, while the right-hand network is an encoder (or discriminator). The encoder of the main network is used to extract multi-scale features, and the upper decoder (i.e., the first decoder) is used to generate segmented images. The lower decoder (i.e., the second decoder) and the encoder together form an autoencoder used to reconstruct the input image, where the lower decoder can provide additional multi-scale features for generating segmented images. The main network generates segmented images and reconstructs images; therefore, the main network can act as a generator (a network that can generate images) and, together with the discriminator, form a Generative Adversarial Network (GAN). The discriminator is used to determine whether the segmented image is a generated fake image or a real image from the dataset. This introduces additional unsupervised error into the main network (i.e., the segmentation model) to assist in the training process. That is, the discriminator is only used during training; after training, it is no longer used in practical applications. The design method of this network architecture specifically includes the following steps:

[0043] (1) Construct the main network using depthwise separable convolution and attention mechanisms.

[0044] The main network is constructed using depthwise separable convolutions and attention mechanisms. The encoder of the main network is divided into five stages, with downsampling as the segmentation point. The decoder is constructed using convolutions, also divided into four stages, with upsampling as the segmentation point (the fifth downsampling stage and the first upsampling stage are the same). The decoder and encoder have a symmetrical structure centered on the fifth stage of the encoder.

[0045] (2) Use autoencoders to provide additional multi-scale features

[0046] An autoencoder is a generative network architecture that is often trained in unsupervised mode using reconstruction error as the loss, and can generate realistic images. In this architecture, a second decoder and the encoder of the main network constitute the autoencoder structure, using the reconstruction error of the input image to be segmented as the loss. The second decoder also has four stages, corresponding one-to-one with the first decoder; while reconstructing the input image, the second decoder generates multi-scale feature maps, which can be input into the first decoder to provide richer features.

[0047] (3) Use better Skip connections

[0048] In UNet and subsequent improved networks, skip connections are used to provide multi-scale features for generating segmented images. In this architecture, we use more sophisticated connections to provide features to the decoder that generates segmented images:

[0049] 1) Skip connections are used to input the feature maps generated by the five stages of the main network into the first decoder. For example... Figure 1 As shown, the features of each stage of the encoder are only input to the corresponding stage of the decoder and the previous stage of the corresponding stage;

[0050] 2) The input features of each stage of the first decoder itself are fed into the output layers of the latter two stages using skip connections;

[0051] 3) The features of each stage of the autoencoder's decoder (i.e., the second decoder) are input into the corresponding stage of the first decoder using skip connections.

[0052] 4) Introducing unsupervised error using generative adversarial networks

[0053] Generative Adversarial Networks (GANs) are the most realistic network architectures for generating images. To improve the quality of segmented images generated by the main network, such as... Figure 1 As shown, the main network is used as the generator and the right discriminator is used as the generative adversarial network architecture, which introduces additional loss to the generated segmented images.

[0054] The method provided by this invention will be further illustrated below with a specific example:

[0055] S1: The encoder of the main network is constructed using alternating connected depthwise separable convolutional layers and Gaussian downsampling layers. Note that a Squeeze-and-Excitation attention mechanism is incorporated into the depthwise separable convolutions in the encoder. The order and number of network layers are as follows: Figure 2 As shown; two decoders are constructed using alternating convolutional layers and upsampling layers, with their order and number of network layers as follows. Figure 3 As shown, the discriminator uses the same network structure as the main network encoder, and adds a global pooling layer and a fully connected layer.

[0056] It is understandable that the network structures of the two decoders are the same, but the number of channels in each layer is different. Preferably, the number of channels in the upper decoder is twice that of the lower encoder.

[0057] S2: Input the image to be segmented into the main network to obtain a segmented image, including:

[0058] S21: In the encoder, the downsampling layer is used to divide the stage, and in the first decoder and the second decoder, the upsampling layer is used to divide the stage; the encoder of the main network is used to extract features corresponding to the scales of the five stages;

[0059] S22: Input the features from the five stages and the intermediate features from the second decoder into the network layer at the corresponding scale of the first decoder, i.e., according to... Figure 1 The skip connections shown input the features from the five stages and the intermediate features from the second decoder into the network layer corresponding to the stage of the first decoder;

[0060] S23: The output of the first decoder of the main network is the segmented image, and the output of the second decoder is the reconstructed image;

[0061] S3: Input the input image and the generated segmented image together into the discriminator, and the discriminator should identify it as false; input the input image and the corresponding real segmented image (i.e., training label) together into the discriminator, and the discriminator should identify it as true;

[0062] S4: The network loss consists of the image segmentation error, the image reconstruction error, and the discriminator loss, and the parameters are updated using the backpropagation algorithm;

[0063] S5: Repeat the above steps for all images in the dataset to complete the model training (the training objective is to generate the segmented image with the highest similarity to the label using the first decoder, and to minimize the reconstruction error between the reconstructed image generated by the second decoder and the medical images in the training set).

[0064] S6: Use the trained main network to segment medical images. The segmentation results include the segmented regions of the target lesions / organs.

[0065] This invention provides a medical image segmentation system based on an autoencoder and a generative adversarial network, comprising: a computer-readable storage medium and a processor;

[0066] The computer-readable storage medium is used to store executable instructions;

[0067] The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any of the above embodiments.

[0068] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A medical image segmentation method based on an auto-encoder and a generative adversarial network, characterized by, The application relates to a medical image segmentation method and device. A training phase is included. A segmentation model and a discriminator are trained by taking a medical image as a training sample and taking a lesion or organ segmentation image of the medical image as a label. The segmentation model includes an encoder, a first decoder and a second decoder. The encoder includes convolution layers and Gaussian down-sampling layers which are alternately connected. Features of corresponding scales in each stage of the encoder are input into network layers of corresponding stages and previous stages of the first decoder through a skip connection layer. Features of corresponding scales in each stage of the first decoder are input into network layers of the next two stages through a skip connection layer. Features of corresponding scales in each stage of the second decoder are input into network layers of corresponding stages of the first decoder through a skip connection layer.

2. The method of claim 1, wherein, The training phase includes a reconstruction error function of a reconstructed image of the medical image generated by the second decoder, a segmentation error function of the lesion or organ segmentation image of the medical image generated by the first decoder and a loss function of true and false value prediction of the discriminator.

3. The method of claim 1, wherein, In the training phase, the discriminator and the segmentation model jointly form a generative adversarial network.

4. The method of claim 1, wherein, An application phase is included.

5. The method according to any one of claims 1 to 4, wherein A medical image to be segmented is input into the trained segmentation model to obtain a segmentation result.

6. The method of claim 5, wherein, The convolution layer is a deep separable convolution layer with an attention mechanism. 7.A medical image segmentation system based on an autoencoder and a generative adversarial network, characterized by, In the training phase, the parameters of the segmentation model are updated by using a back propagation algorithm. The discriminator includes convolution layers and Gaussian down-sampling layers which are alternately connected, a global pooling layer and a full connection layer. The number of the up-sampling layers and the down-sampling layers is 4. The encoder, the first decoder and the second decoder are symmetric structures with the fifth stage of the encoder as the center. A computer readable storage medium and a processor are included. The computer readable storage medium is used for storing executable instructions. The processor is used for reading the executable instructions stored in the computer readable storage medium and executing the method in any one of claims 1-6.

Images