Pulmonary nodule image attribute growth sequence reconstruction method based on attribute interaction generation

Abstract

The invention provides a pulmonary nodule image attribute growth sequence reconstruction method based on attribute interaction generation, and relates to the field of medical image processing. An improved DeepLabv3 + segmentation model and a twin framework generation model are designed, so that the nodule generation accuracy is higher. By introducing the de-noising diffusion probability model, a large number of attribute editing vectors can be easily obtained, and then corresponding nodules are generated, so that the quality of the reconstructed image is improved, the SSIM index of the reconstructed image is higher, the score of perceptual image quality evaluation (LPIPS) is obviously reduced, and the method has the advantages of being high in practicability and the like. The method is expected to become a valuable tool for assisting doctors in analysis and promoting medical research, and meanwhile, expansion of an existing medical data set is realized.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing, and in particular relates to a lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation. Background Art

[0002] Lung cancer is a cancer with the highest morbidity and mortality. Its early symptoms are usually not obvious, and the cure rate of advanced lung cancer is extremely low, which seriously affects the patient's quality of life and survival rate. As the main early manifestation of lung cancer, the detection and analysis of lung nodules are crucial for the early detection and treatment of lung cancer, which can significantly improve the survival rate of patients. However, accurately assessing the nature of lung nodules remains a major problem in clinical detection.

[0003] CT scan is the main means of detecting lung nodules, and growth sequence refers to observing the changes in the size, shape and other characteristics of lung nodules by comparing consecutive CT scan images obtained from the same patient at different time points. This is critical for evaluating whether lung nodules have malignant potential. Although CT scan images have advantages in accurate detection, the high radiation doses associated with frequent CT examinations may pose potential health risks to patients. Therefore, how to reduce the frequency of CT scans while still being able to obtain sufficient imaging data for nodule growth trajectory analysis has become a key issue in the field of medical imaging.

[0004] The above problems can be effectively solved by using the growth sequence reconstruction method. While reducing the frequency of CT scans, sufficient image data can be obtained for nodule growth trajectory analysis. The traditional growth sequence reconstruction method mainly relies on the subjective judgment of radiologists or technicians based on image features, including the size, morphology, density and edge features of the nodules. Although this method can provide detailed analysis, its process is time-consuming and limited by the doctor's experience level, and it is prone to human errors. With the development of artificial intelligence, many novel technologies have emerged in the medical field, which have greatly improved the accuracy and efficiency of medical imaging analysis and promoted the development of medicine. The existing growth sequence reconstruction method is mainly based on deep learning algorithms, especially convolutional neural networks. By training a large amount of data, the characteristic information is gradually discovered, breaking the limitations of manual processing of medical images and saving a lot of time costs. However, the use of convolutional neural networks for growth sequence reconstruction has problems such as deformation of irrelevant areas, changes in background information, and uncertainty in generated samples during the nodule generation process. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a lung nodule image attribute growth sequence reconstruction method based on attribute interactive generation to solve the problems of irrelevant area deformation, background information change and uncertainty of generated samples in the nodule generation process of GAN-type algorithms.

[0006] The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation includes the following steps:

[0007] Step 1: Obtain medical datasets;

[0008] The medical data set includes a plurality of cases, each of which includes a CT scan sequence and nodule data. The CT scan sequence is three-dimensional image data, which includes a plurality of CT scan images, i.e., two-dimensional slices of the three-dimensional image data, and each CT scan image covers a different layer of the lungs; the nodule data includes the position, size and attribute information of each nodule in the CT scan image;

[0009] The position of each nodule is marked by three-dimensional coordinates (x, y, z), where (x, y) represents the plane position of the nodule in the CT scan image, x represents the abscissa of the plane position of the nodule, y represents the ordinate of the plane position of the nodule, and z represents the depth position of the CT scan image where the nodule is located in the CT scan sequence;

[0010] Step 2: Preprocess the medical data set to obtain the real mask and the real background image, and then obtain the preprocessed medical data set;

[0011] The preprocessing method is specifically as follows: first, cases with less than a set number of attribute information are removed, and cases with overlapping annotation areas are excluded; then, only nodules within a set size range are retained in the CT scan image of each case, and the CT scan image is resampled using a bilinear interpolation method. During this process, the nodules in the CT scan image are extracted separately using the annotation boundaries as a true mask, and at the same time, for CT scan images without nodules in the CT scan sequence, they are used as true background images; the cases with overlapping annotation areas refer to cases in which multiple doctors have annotated nodules at the same or adjacent positions in the same CT scan image, but the differences in their annotation ranges, shapes or boundaries exceed a set threshold;

[0012] Step 3: Cluster the CT scan images in the preprocessed medical dataset based on the attribute information and obtain several image pairs using the full permutation method;

[0013] The method of obtaining a plurality of image pairs using the full permutation method is specifically as follows: for each cluster, each CT scan image in the cluster is paired with CT scan images from other clusters;

[0014] The image pair includes two CT scan images with different attribute information, respectively represented as the first type of image I a and the second type of image I b , where the first type of image Ia The attribute information of the second type of image I b The attribute information is b;

[0015] Step 4: Obtain a training data set based on the acquired image pairs;

[0016] Specifically, the CNN encoder model is used to map the two CT scan images in each image pair to the latent space to obtain the source latent vectors, which are the first source latent vector δ(I a ) and the second source latent vector δ(I b ), calculate the first source potential vector δ(I a ) and the second source latent vector δ(I b ) and finally the first source potential vector δ(I a ) and the second source latent vector δ(I b ) are respectively subtracted from the intersection set V to obtain first training data d0 and second training data d1, and the first training data d0 and the second training data d1 form a training data pair, thereby obtaining a training data set including several training data pairs;

[0017] Step 5: Normalize the training data set to obtain a normalized training data set;

[0018] Step 6: Use the normalized training data set to train the denoising diffusion probability model, and use the trained denoising diffusion probability model to obtain the control vector;

[0019] The denoising diffusion probability model is used to gradually add Gaussian noise ε~N(0,I) to the input first training data d0 and the second training data d1, simulating the gradual degradation of the training data in the diffusion process. As time goes by, the noise level gradually increases until it reaches a predetermined noise level at time t. Then, the U-Net network is used as the denoising network for iterative denoising to obtain two restored new vectors d'0 and d'1. The restored new vector d' j The two latent vectors are processed by row scaling and shifting, j = 0, 1; the two latent vectors are added to the two source latent vectors respectively to obtain the control vector w i , i = 1, 2, where j represents the number of the latent vector, i represents the number of the control vector, N(0, I) represents the normal distribution, I represents the unit moment, and ε represents the Gaussian noise sampled from the normal distribution;

[0020] Control vector w i for:

[0021] w i =δ(I s )+Vs' (1)

[0022] Among them, Vs' is the processed potential vector, δ(I s ) is the source latent vector, s is the number of the source latent vector and s = 1, 2;

[0023] The loss function calculation formula of the denoising diffusion probability model is as follows:

[0024] L mse =(ε-ε(d0,d t )) 2 (2)

[0025] Among them, L mse represents the mean square error loss function, ε(d0,d t ) represents the noise predicted by the denoising network at time step t;

[0026] Step 7: Build an improved DeepLabv3+ segmentation model, use the preprocessed medical data set obtained in step 2 to train the improved DeepLabv3+ segmentation model, and use the trained improved DeepLabv3+ segmentation model to obtain the segmentation mask;

[0027] The improved DeepLabv3+ segmentation model includes a maximum pooling layer, an encoder and a decoder;

[0028] The maximum pooling layer is arranged before the encoder;

[0029] The encoder improves the ASPP module in the original DeepLabv3+ model encoder, optimizes the original three-layer dilated convolution rate combination of 6, 12 and 18 to 3, 6 and 12, and adds a CBAM attention mechanism at the output position of each dilated convolution and GAP;

[0030] The decoder performs 1*1 convolution on the shallow information obtained by the encoder backbone network ResNet101, and adds the CBAM attention mechanism at the 1*1 convolution output position of the original DeepLabv3+ model decoder to obtain enhanced shallow features, and performs 4 times bilinear interpolation upsampling on the output result of the encoder to obtain high-level features, and then merges these enhanced shallow features with the high-level features using the Concat function, undergoes a 3×3 convolution, and adds a sub-pixel convolution layer at the 3×3 convolution output position of the original DeepLabv3+ model decoder to obtain a segmentation mask;

[0031] The loss function used by the improved DeepLabv3+ segmentation model is a composite loss function that combines the binary cross entropy loss function BCE and Dice loss. The calculation formula of the composite loss function is as follows:

[0032] Loss=σ×BCE_loss+(1-σ)×Dice_loss (3)

[0033] Among them, Loss represents the composite loss function, σ is the weight parameter, BCE_loss represents the binary cross entropy loss function, and Dice_loss represents the Dice loss;

[0034] Step 8: Multiply the true mask pixel by pixel with the CT scan image in the preprocessed medical dataset to obtain a background image with holes;

[0035] Step 9: Use the full permutation method on the segmentation mask obtained in step 7 to generate several mask image pairs;

[0036] Specifically, for the segmentation mask in each cluster, pair it with the segmentation masks from other clusters to obtain a mask image pair including two segmentation masks with different attribute information;

[0037] Step 10: construct a twin framework generation model based on a pseudo-time window, train the twin framework generation model based on a pseudo-time window using a mask image pair and a control vector, and obtain two false mask images using the trained twin framework generation model based on a pseudo-time window; the false mask image is an intermediate state of the mask image pair of the twin framework generation model of the input pseudo-time window;

[0038] The pseudo-time window-based twin framework generation model inputs a set of mask image pairs each time, including the segmentation mask X a and segmentation mask X b , where X a is the segmentation mask with attribute information a, X b is a segmentation mask with attribute information b, and the output is two false mask images;

[0039] The pseudo-time window-based twin framework generation model includes two branches, each of which is an AttGAN network, including two sub-networks: a generator module and a discriminator module. The generator module includes G enc Encoder and G dec The decoder,discriminator module includes the discriminator D and the classifier C. The networks of the two branches share weight parameters and the forward propagation process is completely consistent;

[0040] The G enc The encoder is used to convert the input segmentation mask into a latent vector;

[0041] The G dec The decoder is based on G enc The latent vector transformed by the encoder edits the attribute information of the segmentation mask into the target attribute information, that is, the control vector w i, and obtain a false mask image; at the same time, reconstruction learning is performed, and the latent vector and attribute information are input into G dec The decoder generates a reconstructed image;

[0042] Then, the segmentation mask and the false mask image are input to the discriminator D for game training of true and false image discrimination. At the same time, the false mask image is input to the classifier C for fine-grained attribute discrimination.

[0043] The loss function of the pseudo-time window-based twin frame generation model retains the reconstruction loss and cross entropy loss in the original AttGAN. The reconstruction loss and cross entropy loss are calculated using the reconstructed image. At the same time, a pre-trained Resnet101 feature extraction network is used to add content loss on the basis of the reconstruction loss and cross entropy loss. The calculation formula is:

[0044]

[0045]

[0046] Among them, f(·) is a pre-trained Resnet101 feature extraction network, represents the segmentation mask with the lth attribute information a, represents the false mask image whose lth attribute information is w1; represents the segmentation mask with the lth attribute information b, represents the false mask image with the lth attribute information w2, n is the number of mask image pairs; L a According to the segmentation mask Calculate the content loss, L b For segmentation mask Calculated content loss;

[0047] Step 11: Use the background image with holes and the true mask to train the U-Net-based hole filling model, and use the trained U-Net-based hole filling model to obtain the filled image;

[0048] The U-Net-based hole filling model is used to perform partial convolution processing and feature fusion on the input background image with holes and the real mask, and use the real background image as a guide to obtain the filled image;

[0049] Define the pixel loss of the U-Net-based hole filling model, and the calculation formula is expressed as:

[0050]

[0051]

[0052] Among them, L hole is the pixel loss in the hole area, is the number of pixels on the true background image for guidance, M is the original mask, and I out is the filled image, I gt Guide the generation of images, To guide the number of pixels on the generated image, L valid is the pixel loss in the non-hole area;

[0053] Finally, we define the perceptual loss and use the pre-trained Resnet101 network to calculate the difference between the two images. The perceptual loss calculation formula is:

[0054]

[0055] in, represents the perceived loss, represents the feature map of the pth layer extracted by the pre-trained ResNet101 network, where I is I comp ,I out or I gt ; yes The number of elements in; P is the number of feature maps, that is, a total of P layers of extraction are performed, I comp is a background image with holes;

[0056] Step 12: Use the Alpha fusion method to superimpose the filled image and the false mask image to obtain the reconstructed image.

[0057] Compared with the prior art, the beneficial effects of the present invention are:

[0058] The present invention proposes a method for reconstructing the attribute growth sequence of lung nodule images based on attribute interaction generation, designs an improved DeepLabv3+ segmentation model and a twin framework generation model, so that the accuracy of nodule generation is higher. By introducing the denoising diffusion probability, a large number of attribute editing vectors can be easily obtained, and then the corresponding nodules can be generated. This invention improves the quality of the reconstructed image. The structural similarity (SSIM) index of the reconstructed image is higher, and the perceptual image quality evaluation (LPIPS) score is significantly reduced. It is expected to become a valuable tool to assist doctors in analysis and promote medical research, while realizing the expansion of existing medical data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is an architecture diagram of a lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation in an embodiment of the present invention;

[0060] Figure 2This is a diagram of the DDPM denoising process in an embodiment of the present invention;

[0061] Figure 3 This is a diagram of the improved DeepLabv3+ model architecture in an embodiment of the present invention;

[0062] Figure 4 This is a diagram of the generation model architecture of the twin framework in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The present invention is described in detail below with reference to the accompanying drawings and embodiments. The overall structure of the present invention is as follows: Figure 1 shown.

[0064] The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation includes the following steps:

[0065] Step 1: Obtain medical datasets;

[0066] The medical data set includes several cases, each of which includes a CT scan sequence and nodule data in DICOM format, the CT scan sequence is three-dimensional image data, including several CT scan images, i.e., two-dimensional slices of the three-dimensional image data, each CT scan image covers different layers of the lungs; the nodule data includes the position, size and attribute information of each nodule in the CT scan image, which is crucial for evaluating the nature of the nodule;

[0067] The position of each nodule is marked by three-dimensional coordinates (x, y, z), where (x, y) represents the plane position of the nodule in the CT scan image, x represents the abscissa of the plane position of the nodule, y represents the ordinate of the plane position of the nodule, and z represents the depth position of the CT scan image where the nodule is located in the CT scan sequence;

[0068] The attribute information includes occultness, edge, texture, sphericity, lobulation, burr shape and malignancy, and these detailed attribute information help to accurately simulate the growth sequence of nodules through the generative model. The growth sequence refers to the simulation of the morphological changes of lung nodules at different time points or development stages based on the attribute information.

[0069] In this embodiment, the LIDC-IDRI public dataset is used for experiments.

[0070] Step 2: Preprocess the medical data set to obtain the real mask and the real background image, and then obtain the preprocessed medical data set;

[0071] The preprocessing method is specifically as follows: first, remove cases with less than a set amount of attribute information, and exclude cases with overlapping annotated areas; then, only retain nodules within a set size range in the CT scan image of each case, and after the above-mentioned more detailed screening process, use bilinear interpolation to resample the CT scan image. This process uses the annotated boundaries to extract the nodules in the CT scan image separately as a true mask. At the same time, the lung tissue near the nodule is retained. This is because generative learning requires more information, so the model needs to be given information about the environment around the lung nodules in order to simulate the growth environment of the nodules and generate more realistic images. For CT scan images without nodule areas in the CT scan sequence, use them as the real background images in step 11 to guide the filling;

[0072] The case of overlapping annotation area refers to the case in which multiple doctors have annotated nodules in the same or adjacent positions in the same CT scan image, but there are significant differences in the annotation range, shape or boundary;

[0073] In this implementation, cases with at least three doctors’ attribute information were used to select nodules with a diameter of 9-30 mm;

[0074] Step 3: Cluster the CT scan images in the preprocessed medical dataset based on the attribute information and obtain several image pairs using the full permutation method;

[0075] The method of obtaining a plurality of image pairs using the full permutation method is specifically as follows: for each cluster, each CT scan image in the cluster is paired with CT scan images from other clusters;

[0076] The image pair includes two CT scan images with different attribute information, respectively represented as the first type of image I a and the second type of image I b , where the first type of image I a The attribute information of the second type of image I b The attribute information is b;

[0077] Step 4: Obtain a training data set based on the acquired image pairs;

[0078] Specifically, the CNN encoder model is used to map the two CT scan images in each image pair to the latent space to obtain the source latent vectors, which are the first source latent vector δ(I a ) and the second source latent vector δ(I b ), calculate the first source potential vector δ(I a ) and the second source latent vector δ(I b ) and finally the first source potential vector δ(Ia ) and the second source latent vector δ(I b ) are respectively subtracted from the intersection set V to obtain first training data d0 and second training data d1, and the first training data d0 and the second training data d1 form a training data pair, thereby obtaining a training data set including several training data pairs;

[0079] Step 5: Normalize the training data set to obtain a normalized training data set to ensure that the data is within a suitable range, which is helpful for the model learning process;

[0080] Step 6: Use the normalized training data set to train the denoising diffusion probability model (DDPM), and use the trained denoising diffusion probability model (DDPM) to obtain the control vector;

[0081] The denoising diffusion probability model includes a denoising process and a denoising process. The denoising process gradually adds Gaussian noise based on normal distribution. The denoising process uses U-Net as a denoising network to gradually remove noise and restore the details of the original data. The denoising diffusion probability model can learn complex image distributions. Given a data set containing different attribute information, training DDPM can capture the complex distribution of these images. In the present invention, DDPM is applied to training in the latent space to model multimodality in attribute changes.

[0082] The denoising diffusion probability model is used to gradually add Gaussian noise ε~N(0,I) to the input first training data d0 and the second training data d1, simulating the gradual degradation of the training data in the diffusion process. As time goes by, the noise level gradually increases until it reaches a predetermined noise level at time t, and then the U-Net network is used for iterative denoising. The denoising process is as follows: Figure 2 As shown, the two new vectors d'0 and d'1 are recovered, and the new vector d' j (j=0,1) The two processed latent vectors are obtained by scaling and shifting the rows. The two processed latent vectors are added to the two source latent vectors respectively to obtain the control vector w i (i=1,2), where j represents the number of the latent vector, i represents the number of the control vector, N(0,I) represents the normal distribution, I represents the unit moment, and ε represents the Gaussian noise sampled from the normal distribution;

[0083] According to the reparameterization technique, the process of adding Gaussian noise is as follows:

[0084]

[0085] Among them, d t represents the noisy training data at time step t; dt-1 represents the noisy training data at the previous time step t-1; α t is a number between 0 and 1, controlling how much noise is added at each step;

[0086] The new vector d' will be restored j The methods for scaling and shifting are:

[0087] V'=γ*d' j +β*V (2)

[0088] Among them, V' is the processed latent vector, γ is the diversity parameter, and β is the scale parameter;

[0089] Control vector w i for:

[0090] w i =δ(I s )+Vs' (3)

[0091] Among them, Vs' is the processed potential vector, δ(I s ) is the source latent vector, s is the number of the source latent vector and s = 1, 2;

[0092] In order to better restore the original data, the present invention selects U-Net as the denoising network and uses the mean square error loss function. The loss function calculation formula of the denoising diffusion probability model is as follows:

[0093] L mse =(ε-ε(d0,d t )) 2 (4)

[0094] Among them, L mse represents the mean square error loss function, ε(d0,d t ) represents the noise predicted by the denoising network at time step t;

[0095] Step 7: Build an improved DeepLabv3+ segmentation model, use the preprocessed medical data set obtained in step 2 to train the improved DeepLabv3+ segmentation model, and use the trained improved DeepLabv3+ segmentation model to obtain the segmentation mask;

[0096] In order to solve the problem of unsatisfactory segmentation of small nodules in the DeepLabv3+ model, the present invention adopts an improved DeepLabv3+ model, which mainly combines the ResNet101 network with CBAM to optimize the original DeepLabv3+ model. The improved DeepLabv3+ segmentation model is as follows: Figure 3 As shown, it includes a maximum pooling layer, an encoder and a decoder;

[0097] The maximum pooling layer is set before the encoder of the original DeepLabv3+ model to enhance the key feature focus and robustness. This method helps the improved DeepLabv3+ segmentation model to focus on the key parts of the image, improves the robustness of the improved DeepLabv3+ segmentation model to small changes in the input image, and reduces the complexity of subsequent calculations;

[0098] In order to enhance the segmentation effect of different targets, the encoder improves the ASPP module in the original DeepLabv3+ model encoder, optimizes the convolution rate combination of the original three-layer dilated convolution from 6, 12 and 18 to 3, 6 and 12, improves the deep feature information, and thus improves the segmentation accuracy of the network, and adds the CBAM attention mechanism at the output position of each dilated convolution and GAP;

[0099] The decoder performs 1*1 convolution on the shallow information obtained by the encoder backbone network ResNet101, and adds the CBAM attention mechanism at the 1*1 convolution output position of the original DeepLabv3+ model decoder to obtain enhanced shallow features, performs 4 times bilinear interpolation upsampling on the output result of the encoder to obtain high-level features, and then merges these enhanced shallow features with the high-level features using the Concat function, undergoes a 3×3 convolution, and adds a sub-pixel convolution layer (ESPCN) at the 3×3 convolution output position of the original DeepLabv3+ model decoder to obtain a high-resolution segmentation mask to improve segmentation accuracy;

[0100] The loss function used by the improved DeepLabv3+ segmentation model is a composite loss function that combines the binary cross entropy loss function (BCE) and the Dice loss. The BCE loss function compares the difference between the true label and the predicted label, while the Dice loss function mainly measures the similarity between the model prediction result and the true label. It can simultaneously optimize the pixel-level classification accuracy and the degree of regional overlap. The calculation formulas of BCE, Dice and composite loss functions are as follows:

[0101]

[0102] Among them, BCE_loss represents the binary cross entropy loss function, ∈ represents the pixel label of the true mask, represents the probability of the segmentation mask predicted by the improved DeepLabv3+ segmentation model;

[0103] The calculation formula of Dice loss is as follows:

[0104]

[0105] Among them, Dice_loss represents Dice loss;

[0106] The calculation formula of the composite loss function is as follows:

[0107] Loss=σ×BCE_loss+(1-σ)×Dice_loss (7)

[0108] Among them, Loss represents the composite loss function, and σ is the weight parameter used to balance the contribution of BCE_loss and Dice_loss;

[0109] In this embodiment, the evaluation indicators used for the segmentation model are IOU (Intersection Over Union) and Dice coefficient, which reach 78.0% and 86.8% respectively.

[0110] Step 8: Multiply the true mask pixel by pixel with the CT scan image in the preprocessed medical dataset to obtain a background image with holes;

[0111] Step 9: Use the full permutation method on the segmentation mask obtained in step 7 to generate several mask image pairs;

[0112] Specifically, for the segmentation mask in each cluster, pair it with the segmentation masks from other clusters to obtain a mask image pair including two segmentation masks with different attribute information;

[0113] Step 10: construct a twin framework generation model based on a pseudo-time window, train the twin framework generation model based on a pseudo-time window using a mask image pair and a control vector, and obtain two false mask images using the trained twin framework generation model based on a pseudo-time window; the false mask image is an intermediate state of the mask image pair of the twin framework generation model of the input pseudo-time window;

[0114] The pseudo-time window refers to the fact that as the number of training rounds increases, the model tracks the state changes of nodule data, and finally simulates the growth trend of nodules to generate an intermediate state at a certain moment. In simple terms, we use full permutations to combine data, and input a set of data pairs into the model each time. As the number of iterations increases, the model can analyze deep semantic information based on attributes and simulate the trend of nodule generation.

[0115] like Figure 4 As shown in the figure, the pseudo-time window-based twin framework generation model includes two branches, each of which is an AttGAN network, including two sub-networks: a generator module and a discriminator module. The generator module includes G enc Encoder and G dec The decoder and discriminator modules include the discriminator D and the classifier C. Like the basic deep twin neural network, the two branches of the network share weight parameters and the forward propagation process of the data is exactly the same.

[0116] The AttGAN network is a generative adversarial network that focuses on attribute editing. It can modify specific attributes of an image while keeping the main structure of the image unchanged. In the pseudo-time window-based twin framework generation model, the AttGAN network is used as the core structure of each branch, and through its generator and discriminator subnetworks, it can achieve accurate generation and discrimination of lung nodule images. The pseudo-time window-based twin framework generation model inputs a set of mask image pairs each time, including segmentation mask X a and segmentation mask X b , where X a is the segmentation mask with attribute information a, X b is the segmentation mask with attribute information b;

[0117] G enc The encoder is essentially a feature extraction network that converts the input segmentation mask into a latent vector Z1;

[0118] Z1=G enc (X a ) (8)

[0119] G dec The decoder is responsible for converting the latent vector into a target, converting the low-dimensional data back to the original data space. Through multiple iterations of this conversion process, an effective data distribution can be learned.

[0120] The G dec The decoder edits the attribute information of the segmentation mask into the target attribute information according to the input potential vector, that is, the control vector w i , get a false mask image, the process is expressed as:

[0121]

[0122] in, Represents a false mask image whose attribute information is w1;

[0123] In order to keep the image semantics other than the target attribute information unchanged, reconstruction learning is also required. The latent vector Z1 and attribute information a are input into G dec Generate reconstructed image The process is expressed as:

[0124]

[0125] Mask image X b Generate a fake mask image using the same method and reconstructed image

[0126] The segmentation mask and the false mask image are input to the discriminator D for game training of true and false image discrimination. At the same time, the false mask image is input to the classifier C for fine-grained attribute discrimination.

[0127] In the loss function, the reconstruction loss and cross entropy loss in the original AttGAN are retained. A pre-trained Resnet101 feature extraction network is used to add content loss to the reconstruction loss and cross entropy loss. The calculation formula is:

[0128]

[0129]

[0130] Among them, f(·) is a pre-trained Resnet101 feature extraction network, represents the segmentation mask with the lth attribute information a, represents the false mask image whose lth attribute information is w1; represents the segmentation mask with the lth attribute information b, represents the false mask image with the lth attribute information w2, n is the number of mask image pairs; L a According to the segmentation mask Calculate the content loss, L b For segmentation mask Calculated content loss;

[0131] Step 11: Use the background image with holes and the true mask to train the U-Net-based hole filling model, and use the trained U-Net-based hole filling model to obtain the filled image;

[0132] The U-Net-based hole filling model is used to perform partial convolution processing and feature fusion on the input background image with holes and the real mask, and use the real background image as a guide to obtain the filled image;

[0133] Specifically, the model replaces all convolutional layers with partial convolutional layers and uses nearest neighbor upsampling in the decoding stage. The jump connection connects two feature maps and two masks respectively as the feature and mask inputs of the next partial convolutional layer. The input of the last partial convolutional layer will contain the connection of the original input image with the hole and the original mask, enabling the model to replicate the pixel information around the hole and improve robustness.

[0134] The partial convolutional layer can arbitrarily define a binary mask of size C×H×W to be the same size as its associated image, and then the mask update is implemented using a fixed convolutional layer with the same kernel size as the partial convolution operation, but with the same weights set to 1 and no bias;

[0135] In this task, the goal of the loss function is to reconstruct and reassemble each pixel, that is, to predict the smoothness of the transition from the hole value to the surrounding environment. The background image I with holes in As the input of the model, the pixel loss of the U-Net-based hole filling model is defined, and the calculation formula is expressed as:

[0136]

[0137]

[0138] Among them, L hole is the pixel loss in the hole area, is the number of pixels on the real background image for guidance, M is the original mask (holes are 1), and I out is the filled image, I gt Guide the generation of images, To guide the number of pixels on the generated image, L valid is the pixel loss in the non-hole area;

[0139] Finally, the perceptual loss is defined, and the difference between two pictures is calculated through the pre-trained Resnet101 network. The perceptual loss calculation formula is expressed as:

[0140]

[0141] in, represents the perceived loss, represents the feature map of the pth layer extracted by the pre-trained ResNet101 network, where I is I comp ,I out or I gt ; yes The number of elements in; P is the number of feature maps, that is, a total of P layers of extraction are performed, I comp is a background image with holes;

[0142] Step 12: Use the Alpha fusion method to superimpose the filled image and the false mask image to obtain the reconstructed image;

[0143] The Alpha fusion method is a process of superimposing a false mask image as a foreground image and a filled image as a background image. It is a commonly used technique in image processing. The formula is as follows:

[0144]

[0145] Among them, C o represents the reconstructed image, and α is used to represent the foreground color, that is, The visibility (transparency) in the reconstructed image, the larger the α, the The greater the proportion in the result, when α=1, the result is exactly equal to (completely opaque); on the contrary, when α=0, Completely invisible in the result (fully transparent).

Claims

1. A method for reconstructing attribute growth sequences of pulmonary nodules images based on attribute interaction generation, characterized in that: The following steps are involved: Step 1: Obtain a medical data set; the medical data set includes several cases, each case includes a CT scan sequence and nodule data, the CT scan sequence is three-dimensional image data, including several CT scan images, i.e., two-dimensional slices of the three-dimensional image data, each CT scan image covers different layers of the lungs; the nodule data includes the position, size and attribute information of each nodule in the CT scan image; Step 2: Preprocess the medical data set to obtain the real mask and the real background image, and then obtain the preprocessed medical data set; Step 3: Cluster the CT scan images in the preprocessed medical dataset based on the attribute information and obtain several image pairs using the full permutation method; Step 4: Obtain a training data set based on the acquired image pairs; Step 5: Normalize the training data set to obtain a normalized training data set; Step 6: Use the normalized training data set to train the denoising diffusion probability model, and use the trained denoising diffusion probability model to obtain the control vector; Step 7: Build an improved DeepLabv3+ segmentation model, use the preprocessed medical data set obtained in step 2 to train the improved DeepLabv3+ segmentation model, and use the trained improved DeepLabv3+ segmentation model to obtain the segmentation mask; Step 8: Multiply the true mask pixel by pixel with the CT scan image in the preprocessed medical dataset to obtain a background image with holes; Step 9: Use the full permutation method on the segmentation mask obtained in step 7 to generate several mask image pairs; Step 10: construct a pseudo-time window-based twin framework generation model, train the pseudo-time window-based twin framework generation model using mask image pairs and control vectors, and obtain two false mask images using the trained pseudo-time window-based twin framework generation model; Step 11: Use the background image with holes and the true mask to train the U-Net-based hole filling model, and use the trained U-Net-based hole filling model to obtain the filled image; Step 12: Use the Alpha fusion method to superimpose the filled image and the false mask image to obtain the reconstructed image.

2. The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation according to claim 1 is characterized in that: The position of each nodule described in step 1 is marked by three-dimensional coordinates (x, y, z), where (x, y) represents the planar position of the nodule in the CT scan image, x represents the horizontal coordinate of the planar position of the nodule, y represents the vertical coordinate of the planar position of the nodule, and z represents the depth position of the CT scan image where the nodule is located in the CT scan sequence.

3. The lung nodule image attribute growth sequence reconstruction method based on attribute interactive generation according to claim 1 is characterized in that: The preprocessing method described in step 2 is specifically as follows: first, remove cases with less than a set number of attribute information, and exclude cases with overlapping annotation areas; then, only retain the nodules within the set size range in the CT scan image of each case, and use the bilinear interpolation method to resample the CT scan image. In this process, the nodules in the CT scan image are extracted separately using the annotation boundaries as the real mask. At the same time, for the CT scan images without nodules in the CT scan sequence, they are used as the real background image; the cases with overlapping annotation areas refer to the cases in which multiple doctors have annotated nodules in the same or adjacent positions in the same CT scan image, but the differences in their annotation ranges, shapes or boundaries exceed the set threshold.

4. The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation according to claim 1, characterized in that: The method of obtaining a plurality of image pairs using the full permutation method in step 3 is specifically as follows: for each cluster, pairing each CT scan image in the cluster with CT scan images from other clusters; The image pair includes two CT scan images with different attribute information, respectively represented as the first type of image I a and the second type of image I b , where the first type of image I a The attribute information of the second type of image I b The attribute information is b.

5. The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation according to claim 1, characterized in that: The step 4 is specifically as follows: using the CNN encoder model to map the two CT scan images in each image pair to the latent space to obtain source latent vectors, which are the first source latent vector δ(I a ) and the second source latent vector δ(I b ), calculate the first source potential vector δ(I a ) and the second source latent vector δ(I b ) and finally the first source potential vector δ(I a ) and the second source latent vector δ(I b ) are respectively subtracted from the intersection set V to obtain the first training data d0 and the second training data d1, and the first training data d0 and the second training data d1 form a training data pair, thereby obtaining a training data set including several training data pairs.

6. The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation according to claim 1, characterized in that: The denoising diffusion probability model in step 6 is used to gradually add Gaussian noise ε~N(0,I) to the input first training data d0 and the second training data d1, simulating the gradual degradation of the training data in the diffusion process. As time goes by, the noise level gradually increases until it reaches the predetermined noise level at time t. Then, the U-Net network is used as the denoising network for iterative denoising to obtain two restored new vectors d'0 and d'1. The restored new vector d' j The two latent vectors are processed by row scaling and shifting, j = 0, 1; the two latent vectors are added to the two source latent vectors respectively to obtain the control vector w i , i = 1, 2, where j represents the number of the latent vector, i represents the number of the control vector, N(0, I) represents the normal distribution, I represents the unit moment, and ε represents the Gaussian noise sampled from the normal distribution; Control vector w i for: w i =δ(I s )+Vs' (1) Among them, Vs' is the processed potential vector, δ(I s ) is the source latent vector, s is the number of the source latent vector and s = 1, 2; The loss function calculation formula of the denoising diffusion probability model is as follows: L mse =(ε-ε(d0,d t )) 2 (2) Among them, L mse represents the mean square error loss function, ε(d0,d t ) represents the noise predicted by the denoising network at time step t.

7. The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation according to claim 1, characterized in that: The improved DeepLabv3+ segmentation model described in step 7 includes a maximum pooling layer, an encoder, and a decoder; The maximum pooling layer is arranged before the encoder; The encoder improves the ASPP module in the original DeepLabv3+ model encoder, optimizes the original three-layer dilated convolution rate combination of 6, 12 and 18 to 3, 6 and 12, and adds a CBAM attention mechanism at the output position of each dilated convolution and GAP; The decoder performs 1*1 convolution on the shallow information obtained by the encoder backbone network ResNet101, and adds the CBAM attention mechanism at the 1*1 convolution output position of the original DeepLabv3+ model decoder to obtain enhanced shallow features, and performs 4 times bilinear interpolation upsampling on the output result of the encoder to obtain high-level features, and then merges these enhanced shallow features with the high-level features using the Concat function, undergoes a 3×3 convolution, and adds a sub-pixel convolution layer at the 3×3 convolution output position of the original DeepLabv3+ model decoder to obtain a segmentation mask; The loss function used by the improved DeepLabv3+ segmentation model is a composite loss function that combines the binary cross entropy loss function BCE and Dice loss. The calculation formula of the composite loss function is as follows: Loss=σ×BCE_loss+(1-σ)×Dice_loss (3) Among them, Loss represents the composite loss function, σ is the weight parameter, BCE_loss represents the binary cross entropy loss function, and Dice_loss represents the Dice loss.

8. The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation according to claim 1, characterized in that: The step 9 specifically includes: pairing the segmentation mask in each cluster with the segmentation masks from other clusters to obtain a mask image pair of segmentation masks including two different attribute information.

9. The lung nodule image attribute growth sequence reconstruction method based on attribute interactive generation according to claim 1, characterized in that: The false mask image in step 10 is an intermediate state of the mask image pair of the twin framework generation model of the input pseudo time window; The pseudo-time window-based twin framework generation model inputs a set of mask image pairs each time, including the segmentation mask X a and segmentation mask X b , where X a is the segmentation mask with attribute information a, X b is a segmentation mask with attribute information b, and the output is two false mask images; The pseudo-time window-based twin framework generation model includes two branches, each of which is an AttGAN network, including two sub-networks: a generator module and a discriminator module. The generator module includes G enc Encoder and G dec The decoder,discriminator module includes the discriminator D and the classifier C. The networks of the two branches share weight parameters and the forward propagation process is completely consistent; The G enc The encoder is used to convert the input segmentation mask into a latent vector; The G dec The decoder is based on G enc The latent vector transformed by the encoder edits the attribute information of the segmentation mask into the target attribute information, that is, the control vector w i , and obtain a false mask image; at the same time, reconstruction learning is performed, and the latent vector and attribute information are input into G dec The decoder generates a reconstructed image; Then, the segmentation mask and the false mask image are input to the discriminator D for game training of true and false image discrimination. At the same time, the false mask image is input to the classifier C for fine-grained attribute discrimination. The loss function of the pseudo-time window-based twin frame generation model retains the reconstruction loss and cross entropy loss in the original AttGAN. The reconstruction loss and cross entropy loss are calculated using the reconstructed image. At the same time, a pre-trained Resnet101 feature extraction network is used to add content loss on the basis of the reconstruction loss and cross entropy loss. The calculation formula is: Among them, f(·) is a pre-trained Resnet101 feature extraction network, represents the segmentation mask with the lth attribute information a, represents the false mask image whose lth attribute information is w1; represents the segmentation mask with the lth attribute information b, represents the false mask image with the lth attribute information w2, n is the number of mask image pairs; L a According to the segmentation mask Calculate the content loss, L b For segmentation mask Compute the content loss.

10. The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation according to claim 1, characterized in that: The U-Net-based hole filling model described in step 11 is used to perform partial convolution processing and feature fusion on the input background image with holes and the real mask, and use the real background image as a guide to obtain the filled image; Define the pixel loss of the U-Net-based hole filling model, and the calculation formula is expressed as: Among them, L hole is the pixel loss in the hole area, is the number of pixels on the true background image for guidance, M is the original mask, and I out is the filled image, I gt Guide the generation of images, To guide the number of pixels on the generated image, L valid is the pixel loss in the non-hole area; Finally, we define the perceptual loss and use the pre-trained Resnet101 network to calculate the difference between the two images. The perceptual loss calculation formula is: in, represents the perceived loss, represents the feature map of the pth layer extracted by the pre-trained ResNet101 network, where I is I comp ,I out or I gt ; yes The number of elements in; P is the number of feature maps, that is, a total of P layers of extraction are performed, I comp is a background image with holes.

Images