Lung nodule risk prediction method and device based on generative adversarial network and unet

Abstract

The application provides a lung nodule risk prediction method and device based on a generative adversarial network and a UNet, the application processes healthy lung CT images and lung nodule diseased lung CT images through the generative adversarial network and the conditional generative adversarial network, the diversity of data samples is enriched, and the generalization ability of the model is improved; the UNet model receives the real and generated lung CT images as input, and segments the lesion area from the lung CT image; the skip connection enables the feature map to transmit information between the corresponding levels of the encoder and the decoder, thereby enhancing the segmentation ability of the model and the accuracy of the lesion area detection; the application can capture complex features and identify tiny lesions well, and the accuracy of lung nodule risk prediction is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of medical information processing and statistical analysis, and particularly relates to a lung nodule risk prediction method and device based on a generative adversarial network and a UNet. BACKGROUND

[0002] A lung nodule is a common abnormality in the lung, usually found on chest X-ray or CT scans. They usually appear as round or oval shadows in the lung, with diameters ranging from a few millimeters to a few centimeters. The occurrence of lung nodules can be caused by a variety of reasons, including infection, inflammation, benign tumors and malignant tumors. Given the diverse causes of lung nodules, accurate determination of their nature is crucial for the further treatment and prognosis of patients.

[0003] Currently, the prediction of lung nodules mainly relies on imaging examination, which can provide detailed imaging information of lung nodules, but requires higher experience and judgment of doctors. In order to improve the accuracy and efficiency of prediction, in recent years, automatic prediction systems based on artificial intelligence have gradually attracted attention. The existing prediction systems lack sufficient training data when facing data-scarce lesion types, resulting in insufficient model generalization ability and difficulty in dealing with complex clinical situations.

[0004] Patent application CN115049584A, aiming at the detection of lung nodules, adopts lung parenchyma segmentation, lung nodule segmentation based on UNet network and false positive screening based on 3DConvNets, reducing manual intervention and assisting risk prediction. However, the above method has the problem of small sample size. Due to the possible bias and imbalance of the data set, the model may have weak generalization ability, affecting the final prediction accuracy. SUMMARY

[0005] The purpose of the present application is to provide a lung nodule risk prediction method and device based on a generative adversarial network and a UNet to improve the accuracy of lung nodule risk prediction.

[0006] In order to achieve the above purpose, the technical solution adopted by the present application is:

[0007] A lung nodule risk prediction method based on a generative adversarial network and a UNet, comprising the following steps:

[0008] S1: obtaining CT images of healthy lungs, CT images of lung nodule diseased lungs and lesion types;

[0009] S2: processing the CT images of healthy lungs using a generative adversarial network to generate healthy image data; processing the CT images of lung nodule diseased lungs using a conditional generative adversarial network to generate diseased image data;

[0010] S3: training the UNet model by taking the CT image of the healthy lung, the CT image of the lung nodule diseased lung, the healthy image data and the diseased image data as inputs of the UNet model;

[0011] The UNet model comprises an encoder, a decoder and a connection layer connecting the encoder and the decoder; the encoder comprises a convolution layer, an activation layer and a maximum pooling layer; the decoder comprises a convolution layer, an activation layer and a deconvolution layer;

[0012] The training process of the UNet model comprises:

[0013] The CT image of the healthy lung, the CT image of the lung nodule diseased lung, the healthy image data and the diseased image data are taken as inputs of the encoder, and are subjected to convolution and activation processing to obtain a feature map F enc ;

[0014] The feature map F enc is processed by convolution, activation and deconvolution of the decoder to obtain a feature map F dec ;

[0015] The feature map F enc and the feature map F dec are connected to obtain a connected feature map F skip , F skip = F enc + F dec ;

[0016] The UNet model is iteratively optimized and trained using a loss function composed of cross-entropy loss and Dice loss to obtain a lung nodule risk prediction model.

[0017] The present application enriches the diversity of data samples and improves the generalization ability of the model by processing the CT images of the healthy lung and the lung nodule diseased lung through the generative adversarial network and the conditional generative adversarial network; the UNet model receives the real and generated lung CT images as inputs, and segments the lesion area from the lung CT images; the skip connection enables the feature map to transmit information between the corresponding levels of the encoder and the decoder, thereby enhancing the segmentation ability of the model and the accuracy of the lesion area detection; the present application can well capture complex features and identify tiny lesions, and improves the accuracy of lung nodule risk prediction.

[0018] Further, the generative adversarial network comprises a first generator and a first discriminator, the first generator comprises a full connection layer and a deconvolution layer, and the first discriminator comprises a convolution layer and a full connection layer.

[0019] Further, the conditional generative adversarial network comprises a second generator and a second discriminator, the second generator comprises a fusion layer, a full connection layer and a deconvolution layer; and the second discriminator comprises a fusion layer, a convolution layer and a full connection layer.

[0020] Further, the implementation process of generating the healthy image data by using the conditional generative adversarial network to process the CT image of the healthy lung comprises:

[0021] obtaining a randomly generated noise vector z, processing the noise vector z through convolution and full connection to generate a fake image G(z);

[0022] processing the CT image x1 of the healthy lung and the fake image G(z) through full connection and deconvolution respectively to obtain a discrimination result D(x1) of the CT image of the healthy lung and a discrimination result D(G(z)) of the fake image G(z);

[0023] iteratively optimizing the first generator and the first discriminator through a loss function to obtain the healthy image data.

[0024] By alternately optimizing the first generator and the first discriminator, the image quality generated by the first generator gradually improves, the healthy lung CT image data samples are enriched, the generalization ability of the model in the face of healthy samples is improved, and the accuracy of risk prediction is enhanced.

[0025] Further, the implementation process of generating the diseased image data by using the conditional generative adversarial network to process the CT image of the lung nodule diseased lung comprises:

[0026] obtaining a randomly generated noise vector z, fusing the noise vector z and the lesion type, processing the fused noise vector z and the lesion type through convolution and full connection to generate a fake image G(z|g);

[0027] fusing the CT image x of the lung nodule diseased lung and the lesion type, fusing the fake image G(z|g) and the lesion type;

[0028] processing the fused CT image of the lung nodule diseased lung and the lesion type through full connection and deconvolution to obtain a discrimination result D(x|g) of the CT image of the lung nodule diseased lung; and processing the fused fake image and the lesion type through full connection and deconvolution to obtain a discrimination result D(G(z|g)|g) of the fake image G(z|g);

[0029] iteratively optimizing the second generator and the second discriminator through a loss function to obtain the diseased image data.

[0030] The conditional generative adversarial network is used, a condition such as a disease course stage or a lesion type is introduced in a generation process, so that the generated image can be associated with specific disease characteristics, thereby improving the relevance and practicability of the generated image. The conditional generative adversarial network takes the condition vector as an additional input on the basis of the traditional GAN, and provides the generator and the discriminator with the condition vector respectively. The condition vector corresponds to the input diseased lung CT image, and the conditional generative adversarial network generates more diversified training data sets, which can improve the generalization ability of the model when facing different lesions, disease course stages and pathological types, thereby enhancing the practicability and accuracy of the risk prediction method.

[0031] Based on the same inventive concept, the present application also provides an electronic device comprising:

[0032] one or more processors;

[0033] a memory having one or more programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the steps of the lung nodule risk prediction method based on the generative adversarial network and the UNet.

[0034] Based on the same inventive concept, the present application also provides a computer readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the lung nodule risk prediction method based on the generative adversarial network and the UNet.

[0035] Compared with the prior art, the present application has the following advantages:

[0036] The present application processes the healthy lung CT image and the lung nodule diseased lung CT image through the generative adversarial network and the conditional generative adversarial network, enriches the diversity of data samples, and improves the generalization ability of the model; the UNet model receives the real and generated lung CT image as input, and segments the lesion area from the lung CT image; the skip connection enables the feature map to transmit information between the corresponding levels of the encoder and the decoder, thereby enhancing the segmentation ability of the model and the accuracy of the lesion area detection; the present application can well capture complex features and identify tiny lesions, and improve the accuracy of the lung nodule risk prediction. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The schematic diagram of the lung nodule risk prediction method based on the generative adversarial network and the UNet of the present application. DETAILED DESCRIPTION

[0038] The application will be described in detail below with reference to the embodiments. It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict. For the convenience of description, the following "up", "down", "left", "right" only indicate the up, down, left and right directions of the drawings themselves, and do not limit the structure.

[0039] Embodiments

[0040] As Figure 1 , the embodiment proposes a lung nodule risk prediction system based on a generative adversarial network and a UNet model. The system combines conditional generative adversarial networks and UNet models. The system can automatically generate and analyze lung CT images of lung nodules, improve lung nodule risk prediction efficiency and accuracy. The steps include:

[0041] Step 1: Data generation and enhancement

[0042] Step 1.1: Application of generative adversarial network (GAN):

[0043] First, use the generative adversarial network (GAN) to generate high-quality healthy lung CT image data. GAN consists of a generator and a discriminator.

[0044] Generator: The generator network Generator consists of multiple fully connected layers and deconvolution layers, which receives a randomly generated noise vector z as input and generates a fake image G(z) with the same size as the real healthy lung CT image. The goal of the generator is to generate realistic images so that the discriminator cannot distinguish whether these images are real or synthetic.

[0045] G(z) = Generator(z)

[0046] Discriminator: The discriminator network Discriminator consists of multiple convolutional layers and fully connected layers. The discriminator network (D) receives a real healthy lung CT image x1 or a generated fake image G(z) as input and outputs a probability value D(x1) indicating the likelihood that the input image is a real image.

[0047] D(x1) = Discriminator(x1)

[0048] The generator and discriminator are optimized through adversarial training. The specific process is as follows:

[0049] 1. Generator training:

[0050] The goal of the generator is to minimize the probability that the discriminator correctly identifies the fake image, so the loss function L G of the generator is defined as the logarithmic negative value of the probability output by the discriminator:

[0051]

[0052] where z is the input noise vector of the generator, p z (z) is the distribution of the noise, is the expected value of the noise, G(z) is the image generated by the generator, and D(G(z)) is the discrimination result of the generated image by the discriminator.

[0053] 2. Discriminator training:

[0054] The loss function L D of the discriminator is defined as the negative logarithm of the sum of the probability that the real image is discriminated as real and the probability that the fake image is discriminated as fake:

[0055]

[0056] where x1 is a real healthy lung CT image, p data (x1) is the distribution of the real image, is the expected value of the real data, and D(x1) is the discrimination result of the real image by the discriminator.

[0057] By alternately optimizing the generator and the discriminator, the quality of the images generated by the generator gradually improves, enriching the healthy lung CT image data samples, improving the generalization ability of the model in the face of healthy samples, and enhancing the accuracy of lung nodule risk prediction.

[0058] Step 1.2: Data diversity improvement:

[0059] Conditional GAN (cGAN) is used to introduce conditions such as disease stage or lesion type during the generation process, so that the generated images can be associated with specific disease characteristics, thereby improving the relevance and practicality of the generated images. cGAN, based on traditional GAN, takes a conditional vector as an additional input and provides it to the generator and discriminator respectively. The conditional vector corresponds to the input real lung CT image.

[0060] Conditional generation: the generator network of cGAN has an additional fusion layer compared to the generator network of GAN, which is used to fuse the randomly generated noise vector z and the conditional vector y (such as lesion type) to generate a fake diseased image G(z|y) that meets the conditions:

[0061] G(z|y) = Generator(z|y)

[0062] Conditional discrimination: the discriminator network of cGAN has an additional fusion layer compared to the discriminator network of GAN, which is used to fuse the real diseased CT image x or the generated fake diseased image and the conditional vector y, and output the probability that the real diseased image x is a real image under the given condition y, and discriminate the authenticity of the image:

[0063] D(x|y) = Discriminator(x|y)

[0064] The specific process is as follows:

[0065] 1. Condition generation:

[0066] The generator receives a randomly generated noise vector z and a condition vector y, and generates a synthetic image G(z|y) that meets the conditions.

[0067] 2. Condition discrimination:

[0068] The discriminator receives a real diseased lung CT image x or a generated fake image G(z|y), and simultaneously receives the corresponding condition input y, and outputs the discrimination result D(x|y) or D(G(z|y)|y).

[0069] The loss function of cGAN adds condition input to the traditional GAN loss function:

[0070]

[0071] where y is the condition vector, representing specific disease characteristics or lesion types, is the generator's expectation of noise z and condition y, is the discriminator's expectation of real data x and condition y.

[0072] By generating more diverse training data sets through cGAN, the model's generalization ability in the face of different lesions, disease stages, and pathological types can be improved, thereby enhancing the accuracy of nodule risk prediction.

[0073] Step 2: Deep learning model optimization

[0074] The UNet model receives real and generated lung CT images as input, segments the lesion area from the lung CT image and outputs it. It is composed of an encoder and a decoder, and through the jump connection, the feature map can transmit information between the corresponding layers of the encoder and the decoder, thereby enhancing the model's segmentation ability and the accuracy of lesion area detection. The following are the specific implementation steps and technical details:

[0075] Step 2.1: Encoder

[0076] The encoder part is composed of multiple convolutional layers, ReLU activation functions, and max-pooling layers. It receives real and generated lung CT images as input feature maps X, and the convolution kernel is K, then the convolution operation can be represented as:

[0077] Y ′ = f(W·X + b)

[0078] where W is the convolution kernel weight, b is the bias, and f is the ReLU activation function.

[0079] Step 2.2 Decoder

[0080] The decoder part is composed of multiple convolutional layers, ReLU activation functions, and deconvolutional layers. The spatial dimensions of the feature maps are gradually restored through deconvolution (up-sampling) operations, and the feature maps from the encoder part are combined through skip connections for processing. Before each up-sampling operation, a convolutional layer is added. Similar to the convolutional operations in the encoder part:

[0081] Z = f(W' · Y' + b')

[0082] where W' and b' are the convolution kernel weights and biases of the decoder part.

[0083] Step 2.3: Adding skip connections

[0084] Skip connections are established between corresponding layers of the encoder and decoder, directly passing the feature maps from the encoder part to the decoder part. This helps preserve high-resolution features and improves the accuracy of segmentation.

[0085] Let the feature maps output by the encoder part be F enc , and the feature maps output by the decoder part be F dec , then the skip connection operation can be represented as:

[0086] F skip = F enc + F dec

[0087] where F skip is the feature map after the skip connection, representing the element-wise addition of the encoder and decoder feature maps.

[0088] Step 2.4: Model output

[0089] The output feature map of the decoder is converted into a segmentation map using a 1x1 convolutional kernel, and a softmax function is applied for processing, achieving pixel-level classification. The output of each pixel point is a probability vector, representing the probability of the point belonging to the corresponding lesion type or healthy. Using the argmax function, the class with the highest probability is selected as the predicted class of the pixel point:

[0090] predicted_class(i, j) = argmax(softmax_outputs(i, j))

[0091] where i, j are the position of the pixel, and softmax_outputs(i, j) is the class probability vector output by the softmax function.

[0092] Finally, the predicted class of all pixels is used to construct a complete segmentation map. This map has the same size as the original image, but the value of each pixel represents its predicted class label.

[0093] Step 2.5: Model optimization

[0094] A combined loss function is used, combining cross-entropy loss and Dice loss, to consider both classification and segmentation accuracy:

[0095]

[0096] where y i and are the true label and predicted label, N is the total number of samples, Y and are the set of true labels and predicted labels.

[0097] During training, large-scale data synthesized by GAN and cGAN and real data collected are used to train the model to improve its generalization ability. During training, the Adam optimization algorithm is used to update the model parameters.

[0098]

[0099] where θ t is the current weight, η is the learning rate, m t and v t are the first and second moment estimates of the gradient, and ∈ is a constant to prevent division by zero.

[0100] Various evaluation indicators are used, including accuracy (Accuracy), precision (Precision), recall (Recall), and F1 score (F1 Score), to evaluate the performance of the model and ensure its performance on various clinical data.

[0101] Through the above specific embodiments, the combination of the generative adversarial network and the UNet model improves the accuracy of lung nodule risk prediction.

[0102] The embodiment also provides a cloud computing-based lung nodule risk prediction system. The system deploys complex image processing and model training tasks on the cloud, and the application of cloud computing enables the system to process larger-scale data sets and more complex models, effectively improving prediction efficiency and processing capacity. The method includes the following steps:

[0103] Step 1: Obtain lung CT image data from public medical databases and upload the data to a cloud storage server using a high-speed network.

[0104] Step 2: Preprocess the uploaded lung CT image data in the cloud, including image standardization, denoising, and data enhancement operations.

[0105] Step 3: Train and optimize the Generative Adversarial Network (GAN) and UNet models in the cloud to generate high-quality synthetic images and improve the predictive ability of the models.

[0106] Step 4: Use the optimized models to analyze and detect lesions in new lung CT image data in the cloud and generate prediction results.

[0107] The embodiment provides an electronic device, comprising:

[0108] one or more processors;

[0109] a memory having one or more programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the steps of the lung nodule risk prediction method based on the generative adversarial network and UNet.

[0110] In some implementations, the memory can be a high-speed random access memory (RAM) and can also include a non-volatile memory, such as at least one disk memory.

[0111] In other implementations, the processor can be a central processing unit (CPU), a digital signal processor (DSP), or various types of general-purpose processors, without limitation.

[0112] The embodiment provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the lung nodule risk prediction method based on the generative adversarial network and UNet.

[0113] The content illustrated in the above embodiments should be understood as these embodiments only for more clearly illustrating the present application, and not for limiting the scope of the present application, after reading the present application, various equivalent forms of the present application are modified by those skilled in the art, all fall within the scope defined by the claims attached to the present application.

Claims

1. A method for predicting the risk of lung nodules based on generative adversarial networks and UNet, characterized in that, Includes the following steps: S1: Obtain CT images of healthy lungs, CT images of lungs with pulmonary nodules, and lesion types; S2: Generative adversarial network is used to process the CT images of the healthy lungs to generate healthy image data; conditional generative adversarial network is used to process the CT images of the diseased lungs with pulmonary nodules to generate diseased image data. S3: Use the CT images of the healthy lungs, the CT images of the diseased lungs with pulmonary nodules, the healthy image data, and the diseased image data as input to the UNet model to train the UNet model; The UNet model includes an encoder, a decoder, and a connection layer connecting the encoder and decoder; the encoder includes a convolutional layer, an activation layer, and a max pooling layer. The decoder includes convolutional layers, activation layers, and deconvolutional layers; The training process of the UNet model includes: The CT images of the healthy lungs, the CT images of the diseased lungs with pulmonary nodules, the healthy image data, and the diseased image data are used as input to the encoder. After convolution and activation processing, feature map F is obtained. enc ; The feature map F is processed through convolution, activation, and deconvolution by the decoder. enc The feature map F is obtained. dec ; Feature map F enc and feature map F dec Perform the connection to obtain the connected feature map F. skip F skip =F enc +F dec ; The UNet model is iteratively optimized and trained using a loss function composed of cross-entropy loss and Dice loss to obtain a lung nodule risk prediction model. The conditional generative adversarial network includes a second generator and a second discriminator. The second generator includes a fusion layer, a fully connected layer, and a deconvolution layer. The second discriminator includes a fusion layer, a convolutional layer, and a fully connected layer. The process of generating diseased image data by using a conditional generative adversarial network to process CT images of the lungs with pulmonary nodules includes: Obtain a randomly generated noise vector z, fuse the noise vector z and the lesion type, and process the fused noise vector and lesion type through convolution and fully connected processing to generate a fake image G(z|y); The CT image x of the lung affected by the pulmonary nodule and the lesion type are fused together, and the spurious image G(z|y) and the lesion type are also fused together. By processing the fused CT images of lungs with pulmonary nodules and lesion types using fully connected and deconvolution methods, the discrimination result D(x|y) of the CT images of lungs with pulmonary nodules is obtained; by processing the fused fake images and lesion types using fully connected and deconvolution methods, the discrimination result D(G(z|y)|y) of the fake images is obtained. Diseased image data are obtained by iteratively optimizing the second generator and the second discriminator through the loss function.

2. The lung nodule risk prediction method based on generative adversarial networks and UNet according to claim 1, characterized in that, The generative adversarial network includes a first generator and a first discriminator. The first generator includes a fully connected layer and a deconvolutional layer, and the first discriminator includes a convolutional layer and a fully connected layer.

3. The lung nodule risk prediction method based on generative adversarial networks and UNet according to claim 2, characterized in that, The process of using a generative adversarial network to process the CT images of the healthy lungs and generate healthy image data includes: Obtain a randomly generated noise vector z, process the noise vector z through convolution and fully connected layers, and generate a fake image G(z); The CT image x1 of the healthy lung and the fake image G(z) are processed by fully connected and deconvolution respectively to obtain the discrimination result D(x1) of the CT image of the healthy lung and the discrimination result D(G(z)) of the fake image G(z). Healthy image data is obtained by iteratively optimizing the first generator and the first discriminator through the loss function.

4. An electronic device, characterized in that, include: One or more processors; A memory having stored one or more programs that, when executed by one or more processors, cause the one or more processors to perform the steps of the method according to any one of claims 1-3.

5. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-3.

Images