Pulmonary nodule risk prediction method and device based on generative adversarial network and UNet
By generating adversarial networks and conditionally generating adversarial networks, CT images are processed, combined with UNet models, the problem of insufficient model generalization ability caused by data scarcity and deviation in the prior art is solved, and the accuracy and segmentation accuracy of pulmonary nodule risk prediction are improved.
Patent Information
- Application Number
- CN202411041780.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-31
AI Technical Summary
The existing pulmonary nodule prediction system has insufficient generalization capabilities when facing data scarcity and deviations, resulting in low prediction accuracy, which is difficult to effectively respond in complex and changeable clinical situations.
Using the pulmonary nodule risk prediction method based on generative adversarial network and UNet, we can process healthy and diseased lung CT images by generating adversarial networks, generate rich and diverse data samples, and use the conditional generative adversarial network to introduce lesion types, and train UNet models To improve the generalization ability and segmentation accuracy of the model.
By enriching data samples and introducing lesion types, the generalization ability of the model and the accuracy of pulmonary nodule risk prediction are improved, complex characteristics can be better captured and micro lesions can be identified, and the accuracy of lesion area detection is enhanced.
Smart Images

Figure CN118942701A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of medical information processing and statistical analysis, and in particular to a method and device for predicting lung nodule risk based on a generative adversarial network and UNet. Background Art
[0002] Pulmonary nodules are a common abnormality of the lungs, usually found on chest X-rays or CT scans. They usually appear as round or oval shadows in the lungs, ranging from a few millimeters to a few centimeters in diameter. The appearance of pulmonary nodules may be caused by a variety of reasons, including infection, inflammation, benign tumors, and malignant tumors. Given the diverse causes of pulmonary nodules, accurately determining their nature is crucial for the patient's further treatment and prognosis.
[0003] Currently, the prediction of lung nodules mainly relies on imaging examinations. This method can provide detailed imaging information of lung nodules, but it requires high experience and judgment of doctors. In order to improve the accuracy and efficiency of prediction, automated prediction systems based on artificial intelligence have gradually attracted attention in recent years. Existing prediction systems lack sufficient training data when faced with lesion types with scarce data, resulting in insufficient model generalization ability and difficulty in coping with complex and changing clinical situations.
[0004] Patent application CN115049584A, for the detection of lung nodules, adopts lung parenchyma segmentation, lung nodule segmentation based on UNet network and false positive screening based on 3DConvNets, which reduces manual intervention and assists risk prediction. However, the above methods have the problem of small sample size. Due to the possible bias and imbalance of the data set, the model will have the problem of weak generalization ability, which affects the final prediction accuracy. Summary of the invention
[0005] The purpose of the present invention is to address the deficiencies in the prior art and provide a method and device for predicting the risk of pulmonary nodules based on a generative adversarial network and UNet, thereby improving the accuracy of pulmonary nodule risk prediction.
[0006] In order to achieve the above object, the technical solution adopted by the present invention is:
[0007] A pulmonary nodule risk prediction method based on generative adversarial network and UNet, comprising the following steps:
[0008] S1: Obtain CT images of healthy lungs, CT images of lungs with pulmonary nodules, and lesion types;
[0009] S2: using a generative adversarial network to process the CT image of the healthy lung to generate healthy image data; using a conditional generative adversarial network to process the CT image of the diseased lung with pulmonary nodules to generate diseased image data;
[0010] S3: using the CT image of the healthy lung, the CT image of the lung with pulmonary nodules, the healthy image data, and the diseased image data as inputs of a UNet model to train the UNet model;
[0011] The UNet model includes an encoder, a decoder, and a connection layer connecting the encoder and the decoder; the encoder includes a convolution layer, an activation layer, and a maximum pooling layer; the decoder includes a convolution layer, an activation layer, and a deconvolution layer;
[0012] The training process of the UNet model includes:
[0013] The CT image of the healthy lung, the CT image of the lung with pulmonary nodules, the healthy image data and the diseased image data are used as the input of the encoder, and after convolution and activation processing, a feature map F is obtained. enc ;
[0014] Process the feature map F through convolution, activation, and deconvolution of the decoder enc , and get the feature map F dec ;
[0015] The feature map F enc and feature map F dec Connect and get the connected feature map F skip , F skip =F enc +F dec ;
[0016] Using the loss function composed of cross entropy loss and Dice loss, the UNet model was iteratively optimized and trained to obtain the pulmonary nodule risk prediction model.
[0017] The present invention processes healthy lung CT images and CT images of diseased lungs with pulmonary nodules through generative adversarial networks and conditional generative adversarial networks, enriching the diversity of data samples and improving the generalization ability of the model; the UNet model receives real and generated lung CT images as input, and segments the lesion area from the lung CT images; through jump connections, the feature map can transfer information between the corresponding levels of the encoder and the decoder, thereby enhancing the segmentation ability of the model and the accuracy of lesion area detection; the present invention can well capture complex features and identify tiny lesions, thereby improving the accuracy of pulmonary nodule risk prediction.
[0018] Furthermore, the generative adversarial network includes a first generator and a first discriminator, the first generator includes a fully connected layer and a deconvolution layer, and the first discriminator includes a convolution layer and a fully connected layer.
[0019] Furthermore, the conditional generative adversarial network includes a second generator and a second discriminator, the second generator includes a fusion layer, a fully connected layer and a deconvolution layer; the second discriminator includes a fusion layer, a convolution layer and a fully connected layer.
[0020] Furthermore, the implementation process of using a generative adversarial network to process the healthy lung CT image to generate healthy image data includes:
[0021] Get a randomly generated noise vector z, process the noise vector z through convolution and full connection, and generate a fake image G(z);
[0022] The healthy lung CT image x1 and the fake image G(z) are processed by full connection and deconvolution respectively, and the discrimination result D(x1) of the healthy lung CT image and the discrimination result D(G(z)) of the fake image G(z) are obtained;
[0023] The first generator and the first discriminator are iteratively optimized through the loss function to obtain healthy image data.
[0024] By alternately optimizing the first generator and the first discriminator, the quality of images generated by the first generator is gradually improved, enriching healthy lung CT image data samples, improving the generalization ability of the model in the face of healthy samples, and enhancing the accuracy of risk prediction.
[0025] Furthermore, the conditional generative adversarial network is used to process the CT image of the diseased lung with pulmonary nodules, and the implementation process of generating diseased image data includes:
[0026] Obtain a randomly generated noise vector z, fuse the noise vector z and the lesion type, process the fused noise vector z and lesion type through convolution and full connection, and generate a false image G(z|g);
[0027] fusing the CT image x of the lung diseased by the pulmonary nodule and the lesion type, and fusing the false image G(z|g) and the lesion type;
[0028] The fused CT image of the diseased lung with pulmonary nodules and the lesion type are processed by full connection and deconvolution to obtain the discrimination result D(x|g) of the CT image of the diseased lung with pulmonary nodules; the fused false image and the lesion type are processed by full connection and deconvolution to obtain the discrimination result D(G(z|g)|g) of the false image G(z|g);
[0029] The second generator and the second discriminator are iteratively optimized through the loss function to obtain diseased image data.
[0030] Conditional generative adversarial networks are used to introduce conditions such as disease stage or lesion type into the generation process, so that the generated images can be associated with specific disease features, thereby improving the relevance and practicality of the generated images. Based on the traditional GAN, the conditional generative adversarial network provides the conditional vector as an additional input to the generator and discriminator respectively. The conditional vector corresponds to the input diseased lung CT image. By generating more diverse training data sets through conditional generative adversarial networks, the generalization ability of the model can be improved when facing different lesions, disease stages and pathological types, thereby enhancing the practicality and accuracy of the risk prediction method.
[0031] Based on the same inventive concept, the present invention also provides an electronic device, including:
[0032] one or more processors;
[0033] A memory having one or more programs stored thereon, which, when executed by the one or more programs, enables the one or more processors to implement the steps of a lung nodule risk prediction method based on a generative adversarial network and UNet.
[0034] Based on the same inventive concept, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a method for predicting lung nodule risk based on a generative adversarial network and UNet.
[0035] Compared with the prior art, the present invention has the following beneficial effects:
[0036] The present invention processes healthy lung CT images and CT images of diseased lungs with pulmonary nodules through generative adversarial networks and conditional generative adversarial networks, enriching the diversity of data samples and improving the generalization ability of the model; the UNet model receives real and generated lung CT images as input, and segments the lesion area from the lung CT images; through jump connections, the feature map can transfer information between the corresponding levels of the encoder and the decoder, thereby enhancing the segmentation ability of the model and the accuracy of lesion area detection; the present invention can well capture complex features and identify tiny lesions, thereby improving the accuracy of pulmonary nodule risk prediction. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 Schematic diagram of the lung nodule risk prediction method based on generative adversarial network and UNet of the present invention. DETAILED DESCRIPTION
[0038] The present invention will be described in detail below in conjunction with the embodiments. It should be noted that the embodiments and features of the embodiments of the present invention can be combined with each other without conflict. For the convenience of description, if the words "upper", "lower", "left" and "right" appear in the following, they only indicate that the upper, lower, left and right directions are consistent with the drawings themselves, and do not limit the structure.
[0039] Example
[0040] like Figure 1 This embodiment proposes a lung nodule risk prediction system based on a generative adversarial network and a UNet model. The system combines a conditional generative adversarial network and a UNet model. The system can automatically generate and analyze lung CT images of lung nodules, thereby improving the efficiency and accuracy of lung nodule risk prediction. The system includes the following steps:
[0041] Step 1: Data Generation and Augmentation
[0042] Step 1.1: Generative Adversarial Network (GAN) Application:
[0043] First, a generative adversarial network (GAN) is used to generate high-quality healthy lung CT image data. GAN consists of a generator and a discriminator.
[0044] Generator: The generator network Generator consists of multiple fully connected layers and deconvolution layers, receives a randomly generated noise vector z as input, and generates a fake image G(z) of the same size as a real healthy lung CT image. The goal of the generator is to generate realistic images so that the discriminator cannot distinguish whether these images are real or synthetic.
[0045] G(z)=Generator(z)
[0046] Discriminator: The discriminator network consists of multiple convolutional layers and fully connected layers. The discriminator network (D) receives the real healthy lung CT image x1 or the generated fake image G(z) as input and outputs a probability value D(x1) indicating the possibility that the input image is a real image.
[0047] D(x1)=Discriminator(x1)
[0048] The generator and discriminator are optimized through adversarial training. The specific process is as follows:
[0049] 1. Generator training:
[0050] The goal of the generator is to minimize the probability that the discriminator correctly identifies a fake image, so the loss function of the generator is L G It is defined as the negative logarithm of the probability output by the discriminator:
[0051]
[0052] Where z is the input noise vector of the generator, p z (z) is the distribution of noise, is the expected value of the noise, G(z) is the image generated by the generator, and D(G(z)) is the judgment result of the discriminator on the generated image.
[0053] 2. Discriminator training:
[0054] The loss function L of the discriminator D It is defined as the negative logarithm of the sum of the probability that a real image is judged as real and the probability that a fake image is judged as fake:
[0055]
[0056] Among them, x1 is the real healthy lung CT image, p data (x1) is the distribution of real images, is the expected value of the real data, and D(x1) is the discriminator's judgment result on the real image.
[0057] By alternately optimizing the generator and the discriminator, the quality of images generated by the generator is gradually improved, enriching the healthy lung CT image data samples, improving the generalization ability of the model in the face of healthy samples, and enhancing the accuracy of lung nodule risk prediction.
[0058] Step 1.2: Improve data diversity:
[0059] Conditional GAN (cGAN) is used to introduce conditions such as disease stage or lesion type during the generation process, so that the generated images can be associated with specific disease characteristics, thereby improving the relevance and practicality of the generated images. Based on the traditional GAN, cGAN takes the conditional vector as an additional input to the generator and discriminator respectively. The conditional vector corresponds to the input real lung CT image.
[0060] Conditional generation: The generator network of cGAN has an additional fusion layer compared to the generator network of GAN, which is used to fuse the randomly generated noise vector z and the conditional vector y (such as the lesion type) to generate a fake diseased image G(z|y) that meets the conditions:
[0061] G(z|y)=Generator(z|y)
[0062] Conditional discrimination: The discriminator network of cGAN has an additional fusion layer compared to the discriminator network of GAN, which is used to fuse the real diseased CT image x or the generated fake diseased image and the conditional vector y, and output the probability D(x|y) that the real diseased image x is the real image under the given condition y, to discriminate the authenticity of the image:
[0063] D(x|y)=Discriminator(x|y)
[0064] The specific process is as follows:
[0065] 1. Conditional generation:
[0066] The generator receives a randomly generated noise vector z and a conditional vector y and generates a synthetic image G(z|y) that meets the conditions.
[0067] 2. Condition determination:
[0068] The discriminator receives the real diseased lung CT image x or the generated fake image G(z|y), and simultaneously receives the corresponding conditional input y, and outputs the discrimination result D(x|y) or D(G(z|y)|y).
[0069] The loss function of cGAN adds conditional input based on the traditional GAN loss function:
[0070]
[0071] Among them, y is a conditional vector, representing a specific disease feature or lesion type, is the expectation of the generator for the noise z and condition y, is the discriminator's expectation of the true data x and conditional y.
[0072] By generating more diverse training data sets through cGAN, the generalization ability of the model can be improved when facing different lesions, disease stages and pathological types, thereby enhancing the accuracy of nodule risk prediction.
[0073] Step 2: Deep Learning Model Optimization
[0074] The UNet model receives real and generated lung CT images as input, segments the lesion area from the lung CT images and outputs them. It consists of an encoder and a decoder. The jump connection enables the feature map to transfer information between the corresponding layers of the encoder and decoder, thereby enhancing the segmentation ability of the model and the accuracy of lesion area detection. The following are the specific implementation steps and technical details:
[0075] Step 2.1: Encoder
[0076] The encoder part consists of multiple convolutional layers, ReLU activation functions, and maximum pooling layers. The real and generated lung CT images are received as input feature maps X, and the convolution kernel is K. The convolution operation can be expressed as:
[0077] Y ′ =f(W·X+b)
[0078] Among them, W is the convolution kernel weight, b is the bias, and f is the ReLU activation function.
[0079] Step 2.2 Decoder
[0080] The decoder part consists of multiple convolutional layers, ReLU activation functions, and deconvolutional layers. The spatial size of the feature map is gradually restored through deconvolution (upsampling) operations, and the feature map of the encoder part is processed through skip connections. Before each upsampling operation, a convolutional layer is added. Similar to the convolution operation of the encoder part:
[0081] Z=f(W′·Y′+b′)
[0082] Where W′ and b′ are the convolution kernel weights and biases of the decoder part.
[0083] Step 2.3: Add skip connections
[0084] A skip connection is established between the corresponding layers of the encoder and decoder to pass the feature maps of the encoder part directly to the decoder part. This helps preserve high-resolution features and improves the accuracy of segmentation.
[0085] Assume that the feature map output by the encoder is F enc , the feature map output by the decoder is F dec , then the skip connection operation can be expressed as:
[0086] F skip =F enc +F dec
[0087] Among them, F skip It is the feature map after skip connection, which represents the element-by-element addition of the encoder and decoder feature maps.
[0088] Step 2.4: Model output
[0089] The output feature map of the decoder is converted into a segmentation map using a 1x1 convolution kernel, and the softmax function is applied to achieve pixel-level classification. The output of each pixel is a probability vector, indicating the probability that the point belongs to the corresponding lesion type or is healthy. Using the argmax function, the category with the highest probability is selected as the predicted category of the pixel:
[0090] predicted_class(i,j)=argmax(softmax_outputs(i,j))
[0091] Among them, i, j are the positions of the pixels, and softmax_outputs(i, j) is the category probability vector output by the above softmax function.
[0092] Finally, the predicted categories of all pixels are used to construct a complete segmentation map. This map has the same size as the original image, but the value of each pixel represents its predicted category label.
[0093] Step 2.5: Model optimization
[0094] A combined loss function, a combination of cross entropy loss and Dice loss, is used to consider both classification and segmentation accuracy:
[0095]
[0096] Among them, y i and are the true label and the predicted label respectively, N is the total number of samples, Y and are the sets of true labels and predicted labels respectively.
[0097] During the training process, large-scale data synthesized by GAN and cGAN and real data collected are used for training to improve the generalization ability of the model. During the training process, the Adam optimization algorithm is used to update the model parameters.
[0098]
[0099] Among them, θ t is the current weight, η is the learning rate, m t and v t are the first-order moment estimate and the second-order moment estimate of the gradient, respectively, and ∈ is a constant to prevent division by zero.
[0100] A variety of evaluation indicators, including accuracy, precision, recall and F1 score, are used to evaluate model performance and ensure the performance of the model on various clinical data.
[0101] Through the above specific implementation methods, this embodiment improves the accuracy of risk prediction of pulmonary nodules by combining the generative adversarial network and the UNet model.
[0102] This embodiment also provides a pulmonary nodule risk prediction system based on cloud computing. The system deploys complex image processing and model training tasks in the cloud. The application of cloud computing enables the system to process larger data sets and more complex models, effectively improving prediction efficiency and processing capabilities. The method includes the following steps:
[0103] Step 1: Obtain lung CT image data from the public medical database and upload the data to a cloud storage server using a high-speed network;
[0104] Step 2: Preprocess the uploaded lung CT image data in the cloud, including image standardization, denoising, and data enhancement.
[0105] Step 3: Train and optimize the Generative Adversarial Network (GAN) and UNet models in the cloud to generate high-quality synthetic images and improve the model’s predictive capabilities.
[0106] Step 4: Use the optimized model to analyze and detect lesions on new lung CT image data in the cloud to generate prediction results.
[0107] This embodiment provides an electronic device, including:
[0108] one or more processors;
[0109] A memory having one or more programs stored thereon, which, when executed by one or more processors, enables the one or more processors to implement the steps of a lung nodule risk prediction method based on a generative adversarial network and UNet.
[0110] In some implementations, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.
[0111] In some other implementations, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.
[0112] This embodiment provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a method for predicting lung nodule risk based on a generative adversarial network and UNet.
[0113] The contents explained in the above embodiments should be understood as these embodiments are only used to more clearly illustrate the present invention, and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.
Claims
1. A pulmonary nodule risk prediction method based on generative adversarial network and UNet, characterized in that: The following steps are involved: S1: Obtain CT images of healthy lungs, CT images of lungs with pulmonary nodules, and lesion types; S2: using a generative adversarial network to process the CT image of the healthy lung to generate healthy image data; using a conditional generative adversarial network to process the CT image of the diseased lung with pulmonary nodules to generate diseased image data; S3: using the CT image of the healthy lung, the CT image of the lung with pulmonary nodules, the healthy image data, and the diseased image data as inputs of a UNet model to train the UNet model; The UNet model includes an encoder, a decoder, and a connection layer connecting the encoder and the decoder; the encoder includes a convolution layer, an activation layer, and a maximum pooling layer; The decoder comprises a convolutional layer, an activation layer and a deconvolutional layer; The training process of the UNet model includes: The CT image of the healthy lung, the CT image of the lung with pulmonary nodules, the healthy image data and the diseased image data are used as the input of the encoder, and after convolution and activation processing, a feature map F is obtained. enc ; Process the feature map F through convolution, activation, and deconvolution of the decoder enc , and get the feature map F dec ; The feature map F enc and feature map F dec Connect and get the connected feature map F skip , F skip =F enc +F dec ; Using the loss function composed of cross entropy loss and Dice loss, the UNet model was iteratively optimized and trained to obtain the pulmonary nodule risk prediction model.
2. The pulmonary nodule risk prediction method based on generative adversarial network and UNet according to claim 1, characterized in that: The generative adversarial network includes a first generator and a first discriminator, the first generator includes a fully connected layer and a deconvolution layer, and the first discriminator includes a convolution layer and a fully connected layer.
3. The pulmonary nodule risk prediction method based on generative adversarial network and UNet according to claim 1, characterized in that: The conditional generative adversarial network includes a second generator and a second discriminator, the second generator includes a fusion layer, a fully connected layer and a deconvolution layer; the second discriminator includes a fusion layer, a convolution layer and a fully connected layer.
4. The pulmonary nodule risk prediction method based on generative adversarial network and UNet according to claim 2, characterized in that: The implementation process of using a generative adversarial network to process the healthy lung CT image and generate healthy image data includes: Get a randomly generated noise vector z, process the noise vector z through convolution and full connection, and generate a fake image G(z); The healthy lung CT image x1 and the fake image G(z) are processed by full connection and deconvolution respectively, and the discrimination result D(x1) of the healthy lung CT image and the discrimination result D(G(z)) of the fake image G(z) are obtained; The first generator and the first discriminator are iteratively optimized through the loss function to obtain healthy image data.
5. The pulmonary nodule risk prediction method based on generative adversarial network and UNet according to claim 3, characterized in that: The conditional generative adversarial network is used to process the CT image of the lung diseased with pulmonary nodules, and the implementation process of generating diseased image data includes: Obtain a randomly generated noise vector z, fuse the noise vector z and the lesion type, process the fused noise vector and lesion type through convolution and full connection, and generate a false image G(z|y); fusing the CT image x of the diseased lung with pulmonary nodules and the lesion type, and fusing the false image G(z|y) and the lesion type; The fused CT image of the diseased lung with pulmonary nodules and the lesion type are processed by full connection and deconvolution to obtain the discrimination result D(xy) of the CT image of the diseased lung with pulmonary nodules; the fused false image and the lesion type are processed by full connection and deconvolution to obtain the discrimination result D(G(z|y)|y) of the false image G(z|y); The second generator and the second discriminator are iteratively optimized through the loss function to obtain diseased image data.
6. An electronic device, characterized in that: include: one or more processors; A memory having one or more programs stored thereon, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the steps of the method according to any one of claims 1 to 5.
7. A computer-readable storage medium, characterized in that: The computer program is stored therein, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.
Citation Information
Patent Citations
Medical image synthesis method based on double generative adversarial networks
CN107909621A
A spaceflight abnormal event detection method based on a generative adversarial network
CN109447263A
Medical image synthesis and classification method based on a conditional multi-judgment generative adversarial network
CN109493308A
Pulmonary nodule intelligent diagnosis method based on GAN and Unet network
CN111524144A
Retina optical coherence tomography image lesion intelligent detection and positioning method based on generative adversarial network
CN114581420A
Cited By
LSTM and GAN combination-based pulmonary nodule growth prediction method
CN120727289A
Fusion aid decision-making model construction method and system for pneumonia image interpretation
CN122023405A
