A lung CT lesion general automatic segmentation method based on a generative adversarial network

Abstract

This invention belongs to the field of image segmentation technology, and particularly relates to a general automatic segmentation method for lung CT lesions based on generative adversarial networks. It achieves general automatic segmentation of lung CT image lesions, reducing the cost of redesigning different segmentation methods for various types of lung lesion CT images. The method includes: segmenting the lesion region of the lung CT image using a generative network based on a fusion of U-net and an attention network, obtaining a segmented image of the lesion region; calculating the image loss and self-supervised rotation loss between the segmented image and the real lesion image based on a discriminative network with fused self-supervised rotation loss; and training and optimizing the generative and discriminative networks respectively based on the loss values ​​of the segmented image and the real lesion image to achieve automatic segmentation of the lung lesion region on the lung CT image.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology, and in particular relates to a general automatic segmentation method for lung CT lesions based on generative adversarial networks. Background Technology

[0002] Because lesions on lung CT images include various types of lung lesions such as pulmonary nodules, pneumonia, and tuberculosis, automatic segmentation methods are often limited by the diversity of lung lesion types, the complexity of grayscale changes in lesion tissue, and the uncertainty of lesion morphology and location. While emerging convolutional neural networks have the potential to provide a general segmentation method for lesion regions on lung CT images, they are currently limited by the small number of available training samples. Segmentation networks are often designed and trained only for a specific type of lung CT lesion, and cannot provide a universal automatic segmentation method for lesions on lung CT images.

[0003] Generative adversarial networks (GANs) are a novel type of network capable of effectively learning image features, and have the potential to provide a general segmentation method for different types of lesions on lung CT images. However, current GANs struggle to simultaneously and effectively learn both lesion morphology and semantic information on lung CT images, resulting in low network structure utilization. Furthermore, the inherent forgetting nature of the discriminator in GANs makes network training complex and time-consuming. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies by providing a general automatic segmentation method for lung CT lesions based on generative adversarial networks.

[0005] To achieve the above objectives, the present invention adopts the following technical solution, including:

[0006] A generative network based on the fusion of U-net and attention network is used to segment the lesion region of lung CT image, thereby obtaining a segmented image of the lesion region of the lung CT image.

[0007] Based on a discriminant network that incorporates self-supervised rotation loss, the image loss and self-supervised rotation loss between the segmented image and the real lesion image (delineated by experts) on the lung CT image are calculated.

[0008] The generator network and the discriminator network are trained and optimized based on the loss values ​​of the segmented image and the real lesion image, respectively, to achieve automatic segmentation of lung lesion areas on lung CT images.

[0009] Furthermore, the lung CT images include images of lung nodules, pneumonia, and tuberculosis.

[0010] Furthermore, the fusion of the U-net and the attention network generates a network to segment the lesion region of the lung CT image. Obtaining the segmented image of the lesion region of the lung CT image includes adding a spatial attention network to the shallow short connections of the U-net and adding a channel attention network to the deep short connections of the U-net.

[0011] Furthermore, using U-net as the backbone network, a spatial attention network module is added to the first short connection of the U-net network, and a channel attention network module is added to the second and third short connections of the U-net network.

[0012] Furthermore, the addition of a spatial attention network module to the first short connection of the U-net network includes the input image of the first short connection of the U-net network first passing through the spatial attention network module to learn the morphological information of the lesion area on the lung CT image.

[0013] The addition of a channel attention network module to the second and third short connections of the U-net network involves the input images of the second and third short connections of the U-net network first passing through the channel attention network module to learn the semantic structure information of the lesion area on the lung CT image.

[0014] Furthermore, the discriminative network based on fusion self-supervised rotation loss calculates the image loss and self-supervised rotation loss between the segmented image and the real lesion image based on the segmented image of the lesion region on the lung CT image and the real lesion image, including:

[0015] Based on the segmented image of the lesion area on the lung CT image and the actual lesion image (drawn by the expert), the images are rotated by N° and stitched together to obtain the one-hot encoding of the segmented image of the lesion area on the lung CT image, the actual lesion image (drawn by the expert), and the rotated image.

[0016]

[0017] Where, G(x) 0° Let G(x) represent the segmented image. N° Let R(x) represent the segmented image after rotation by N degrees. 0° R(x) represents the true image of the lesion. N° This represents the actual lesion image after rotation by N degrees.

[0018] Furthermore, this includes: Concat(G(x)) 0° , G(x) N° ), Concat(R(x) 0° , R(x) N° Input the discriminant network and calculate the network loss:

[0019]

[0020] Wherein, Conv_D represents the discriminant network, G_pro_loss and R_pro_loss represent the image losses of the segmented image and the real lesion image after passing through the discriminant network, respectively, and G_rot_pros and R_rot_pros represent the self-supervised rotation probability values ​​of the segmented image and the real lesion image after passing through the discriminant network, respectively.

[0021] Furthermore, this includes: a self-supervised rotation loss calculated by a discriminative network between the segmented image and the real lesion image, calculated by the following formula:

[0022]

[0023] Where binary_cross represents the loss calculation method, rot_labels represents the one-hot encoding of the image, G_Rot_loss represents the self-supervised rotation loss value of the segmented image after passing through the discriminant network, and R_Rot_loss represents the self-supervised rotation loss value of the real lesion image after passing through the discriminant network.

[0024] Furthermore, the lung lesion is any one or more of pulmonary nodules, pneumonia, and pulmonary tuberculosis.

[0025] Compared with the prior art, the present invention has the following advantages.

[0026] This invention enables universal automatic segmentation of lung CT image lesions, reducing the cost of redesigning different segmentation methods for various types of lung lesion CT images.

[0027] The fusion generative network proposed in this invention can leverage the advantages of spatial attention networks, channel attention networks, and U-net networks to improve the segmentation efficiency of lesion regions in lung CT images.

[0028] The self-supervised rotation loss designed in this invention can simultaneously learn the image loss between the segmented image and the real lesion image and the self-supervised rotation loss, thus avoiding the discriminator forgetting defect of generative adversarial networks. Attached Figure Description

[0029] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The scope of protection of the present invention is not limited to the following description.

[0030] Figure 1 The flowchart illustrates a general automatic segmentation method for lung CT lesions based on generative adversarial networks, as provided in this embodiment of the invention.

[0031] Figure 2This is a schematic diagram of automatic segmentation of lung nodule regions on a lung CT image provided in an embodiment of the present invention.

[0032] Figure 3 This is a schematic diagram of automatic segmentation of the pneumonia region on a lung CT image provided in an embodiment of the present invention.

[0033] Figure 4 This is a schematic diagram illustrating the automatic segmentation of the pulmonary tuberculosis region on a lung CT image, provided as an embodiment of the present invention. Detailed Implementation

[0034] This invention generates segmented images of lung lesion regions from input lung CT images using a generative network based on a fusion of U-net and an attention network. A discriminative network, incorporating self-supervised rotation loss, calculates the image loss between the segmented image and the actual lesion image, as well as the self-supervised rotation loss. Based on these image and rotation losses, the generative and discriminative networks are trained respectively to accurately segment lesion regions such as lung nodules, pneumonia, and tuberculosis on lung CT images.

[0035] like Figure 1 As shown, a detailed description is given of a general automatic segmentation method for lesion regions in lung CT images based on generative adversarial networks.

[0036] In step S101, the lesion region of the lung CT image is segmented from the lung CT image by a network generated by the fusion of U-net and attention network, and the segmented image of the lesion region on the lung CT image is obtained.

[0037] Here, lung CT images, including images of lung nodules, pneumonia, and tuberculosis, are used as input to the generative network.

[0038] The specific process of step S101 is as follows: Using U-net as the backbone network, a spatial attention network module is added to the first short connection of the U-net network. That is, the input image of the first short connection of the U-net network first passes through the spatial attention network module to learn the morphological information of the lesion area on the lung CT image.

[0039] Channel attention network modules are added to the second and third short connections of the U-net network. That is, the input images of the second and third short connections of the U-net network first pass through the channel attention network module to learn the semantic structure information of the lesion area on the lung CT image.

[0040] Through the interaction of the U-net and attention network in the fusion generation network, the lung CT lesion region is obtained from the input image, and the output of the generation network is a segmented image of the lung CT lesion region.

[0041] In step S102, by rotating and stitching the segmented image output in step S101 and the real lesion image delineated by the expert, a segmented image and a real lesion image obtained after rotation and stitching are obtained, and one-hot encodings of the segmented image and the real lesion image are obtained. The one-hot encoding is used as a label for the discriminant network to calculate the self-supervised rotation loss.

[0042] In this step, the segmented image G(x) and the true lesion image R(x) are rotated using the following formula:

[0043]

[0044] Where, G(x) 0° Let G(x) represent the segmented image. N° Let R(x) represent the segmented image after rotation by N degrees. 0° R(x) represents the true image of the lesion. N° This represents the actual lesion image after rotation by N degrees.

[0045] The one-hot encoding of the segmented image and the real lesion image is calculated by the following formula:

[0046]

[0047] Where, G(x) one-hot Let R(x) represent the image G(x) and the self-supervised encoding label when rotated N°. one-hot Represents the R(x) image and the self-supervised coding label when rotated N°.

[0048] Concat(G(x)) 0° , G(x) N° ) and Concat(R(x) 0° , R(x) N° Input the discriminant network and calculate the network loss:

[0049]

[0050] Wherein, Conv_D represents the discriminant network, G_pro_loss and R_pro_loss represent the image losses of the segmented image and the real lesion image after passing through the discriminant network, respectively, and G_rot_pros and R_rot_pros represent the self-supervised rotation probability values ​​calculated by the discriminant network for the segmented image and the real lesion image, respectively.

[0051] The self-supervised rotation loss calculated by the discriminant network for the segmented image and the real lesion image is calculated by the following formula:

[0052]

[0053] Where binary_cross represents the network loss calculation method, rot_labels represents the one-hot encoding of the image, G_Rot_loss represents the self-supervised rotation loss value of the segmented image after passing through the discriminant network, and R_Rot_loss represents the self-supervised rotation loss value of the real lesion image after passing through the discriminant network.

[0054] In step S103, based on the segmented image of the lesion area on the lung CT image and the real lesion image delineated by the expert, the image loss value and self-supervised rotation loss value of the discriminant network are used to train and optimize the generator network and the discriminant network respectively, so as to realize the automatic segmentation of lung lesion areas on lung CT images, including lung nodules, pneumonia and pulmonary tuberculosis.

[0055] like Figure 2 The image shows the automatic segmentation of pulmonary nodule regions on a lung CT image. a, b, and c represent the original lung CT image, the automatically segmented pulmonary nodule lesion image obtained by this invention, and a manually drawn image of a real lesion by an expert, respectively. The white curve represents the lesion region in the lung CT image. The arrow in b indicates the pulmonary nodule region not drawn by the expert, which was accurately segmented using this method. Therefore, this invention can achieve automatic segmentation results close to the expert-drawn real lesion image for pulmonary nodule lesion regions on lung CT images, and it also detects pulmonary nodule lesion regions missed by the expert.

[0056] like Figure 3 As shown, this represents the automatic segmentation of the pneumonia region on a lung CT image; where a, b, and c are the original lung CT image, the segmented image of the pneumonia lesion on the lung CT image obtained by this invention, and the actual lesion image manually drawn by an expert, respectively. The white curve represents the lesion region in the lung CT image. Therefore, it can be seen that for the pneumonia lesion region on a lung CT image, this method can achieve segmentation results close to those of the actual lesion image manually drawn by an expert.

[0057] like Figure 4 The diagram illustrates the automatic segmentation of pulmonary tuberculosis lesions on a lung CT image. a, b, and c represent the original lung CT image, the segmented image of pulmonary tuberculosis lesions obtained by this invention, and a manually drawn image of the actual lesion by an expert, respectively. The white curve represents the lesion region in the lung CT image. The arrow in b indicates the pulmonary tuberculosis region that the expert failed to draw; accurate segmentation was achieved according to this invention. Therefore, for pulmonary tuberculosis lesions on lung CT images, the method of this invention can achieve segmentation results close to those of manually drawn images of the actual lesion by an expert, and it detects pulmonary tuberculosis lesion regions missed by the expert.

[0058] It is understood that the above specific description of the present invention is only for illustrating the present invention and is not limited to the technical solutions described in the embodiments of the present invention. Those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention to achieve the same technical effect; as long as the use needs are met, they are all within the protection scope of the present invention.

Claims

1. A general automatic segmentation method for lung CT lesions based on generative adversarial networks, characterized in that: include: A generative network based on the fusion of U-net and attention network is used to segment the lesion region of lung CT image, thereby obtaining a segmented image of the lesion region of the lung CT image; Based on a discriminant network that incorporates self-supervised rotation loss, the image loss and self-supervised rotation loss between the segmented image and the real lesion image are calculated according to the segmented image and the real lesion image of the lesion region on the lung CT image. The generator network and the discriminator network are trained and optimized based on the loss values ​​of the segmented image and the real lesion image, respectively, so as to achieve automatic segmentation of lung lesion areas on lung CT images. The discriminative network based on fusion self-supervised rotation loss calculates the image loss and self-supervised rotation loss between the segmented image and the real lesion image on the lung CT image, including: Based on the segmented image of the lesion area on the lung CT image and the real lesion image, rotate the images by N° and stitch them together to obtain the one-hot encoding of the segmented image of the lesion area on the lung CT image, the real lesion image, and the rotated image; Where, G(x) 0° Let G(x) represent the segmented image. N° Let R(x) represent the segmented image after rotation by N degrees. 0° R(x) represents the true image of the lesion. N° This represents the actual lesion image after rotation by N°; Also includes: Concat(G(x)) 0° , G(x) N° ), Concat(R(x) 0° , R(x) N° Input the discriminant network and calculate the network loss: Wherein, Conv_D represents the discriminant network, G_pro_loss and R_pro_loss represent the image losses of the segmented image and the real lesion image after passing through the discriminant network, respectively, and G_rot_pros and R_rot_pros represent the self-supervised rotation probability values ​​of the segmented image and the real lesion image after passing through the discriminant network, respectively. It also includes: the self-supervised rotation loss calculated by the discriminant network between the segmented image and the real lesion image, calculated by the following formula: Where binary_Cross represents the loss calculation method, rot_labels represents the one-hot encoding of the image, G_Rot_loss represents the self-supervised rotation loss value of the segmented image after passing through the discriminant network, and R_Rot_loss represents the self-supervised rotation loss value of the real lesion image after passing through the discriminant network.

2. The general automatic segmentation method for lung CT lesions based on generative adversarial networks according to claim 1, characterized in that: The lung CT images include images of lung nodules, pneumonia, and tuberculosis.

3. The general automatic segmentation method for lung CT lesions based on generative adversarial networks according to claim 1, characterized in that: The fusion of U-net and attention network generates a network to segment lesion regions in lung CT images. Obtaining segmented images of lesion regions in lung CT images involves adding a spatial attention network to the shallow short connections of U-net and adding a channel attention network to the deep short connections of U-net.

4. The general automatic segmentation method for lung CT lesions based on generative adversarial networks according to claim 3, characterized in that: The addition of a spatial attention network to the shallow short connections of U-net and the addition of a channel attention network to the deep short connections of U-net include: Using U-net as the backbone network, a spatial attention network module is added to the first short connection of the U-net network, and a channel attention network module is added to the second and third short connections of the U-net network.

5. A general automatic segmentation method for lung CT lesions based on generative adversarial networks according to claim 4, characterized in that: The addition of a spatial attention network module to the first short connection of the U-net network includes the input image of the first short connection of the U-net network first passing through the spatial attention network module to learn the morphological information of the lesion area on the lung CT image; The addition of a channel attention network module to the second and third short connections of the U-net network involves the input images of the second and third short connections of the U-net network first passing through the channel attention network module to learn the semantic structure information of the lesion area on the lung CT image.

6. A general automatic segmentation method for lung CT lesions based on generative adversarial networks according to any one of claims 1-5, characterized in that: The lung lesions are any one or more of the following: pulmonary nodules, pneumonia, and pulmonary tuberculosis.

Images