Attribute growing sequence reconstruction method for pulmonary nodule images based on attribute interaction generation

Abstract

The present invention provides a method for reconstructing attribute growth sequences of lung nodules based on attribute interaction generation, which relates to the field of medical image processing. An improved DeepLabv3+ segmentation model and a twin framework generation model are designed to achieve a higher accuracy of nodule generation. By introducing a denoising diffusion probability model, a large number of attribute editing vectors can be easily obtained to generate corresponding nodules. This invention improves the quality of reconstructed images. The structural similarity (SSIM) index of the reconstructed images is higher, and the perceptual image quality evaluation (LPIPS) score is significantly reduced. It is expected to become a valuable tool to assist doctors in analysis and promote medical research, while achieving the expansion of existing medical data sets.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing, and in particular relates to a lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation. Background Art

[0002] Lung cancer, one of the most common cancers with the highest morbidity and mortality, often presents with subtle early symptoms, while the cure rate for advanced lung cancer is extremely low, severely impacting patients' quality of life and survival. Detection and analysis of lung nodules, the primary early manifestation of lung cancer, are crucial for early detection and treatment, significantly improving patient survival. However, accurately assessing the nature of lung nodules remains a major challenge in clinical testing.

[0003] CT scans are the primary means of detecting lung nodules. Growth sequences compare consecutive CT scans of the same patient at different time points to observe changes in the size, shape, and other characteristics of lung nodules. This is critical for assessing whether a lung nodule has malignant potential. Although CT scan images offer advantages in accurate detection, the high radiation doses associated with frequent CT examinations can pose potential health risks to patients. Therefore, reducing the frequency of CT scans while still obtaining sufficient imaging data for nodule growth trajectory analysis has become a key issue in the field of medical imaging.

[0004] Growth sequence reconstruction methods can effectively address these issues, reducing the frequency of CT scans while acquiring sufficient imaging data for nodule growth trajectory analysis. Traditional growth sequence reconstruction methods rely primarily on subjective judgment by radiologists or technicians based on imaging features, including nodule size, morphology, density, and edge characteristics. While this method provides detailed analysis, it is time-consuming, limited by the physician's experience, and prone to human error. With the development of artificial intelligence, many novel technologies have emerged in the medical field, significantly improving the accuracy and efficiency of medical imaging analysis and promoting the advancement of medicine. Existing growth sequence reconstruction methods primarily rely on deep learning algorithms, particularly convolutional neural networks. By training large amounts of data, they gradually discover characteristic information, overcoming the limitations of manual medical image processing and saving significant time and cost. However, the use of convolutional neural networks for growth sequence reconstruction presents challenges such as deformation of irrelevant regions during nodule generation, alteration of background information, and uncertainty in generated samples. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides a lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation to solve the problems of GAN-type algorithms such as deformation of irrelevant areas, changes in background information, and uncertainty in generated samples during the nodule generation process.

[0006] The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation includes the following steps:

[0007] Step 1: Obtain medical dataset;

[0008] The medical dataset includes several cases, each of which includes a CT scan sequence and nodule data. The CT scan sequence is three-dimensional imaging data, which includes several CT scan images, i.e., two-dimensional slices of the three-dimensional imaging data, each CT scan image covering a different layer of the lungs; the nodule data includes the location, size, and attribute information of each nodule in the CT scan image;

[0009] The position of each nodule is marked by three-dimensional coordinates (x, y, z), where (x, y) represents the plane position of the nodule in the CT scan image, x represents the abscissa of the plane position of the nodule, y represents the ordinate of the plane position of the nodule, and z represents the depth position of the CT scan image where the nodule is located in the CT scan sequence;

[0010] Step 2: Preprocess the medical dataset to obtain the true mask and the true background image, and then obtain the preprocessed medical dataset;

[0011] The preprocessing method is specifically as follows: first, cases with less than a set number of attribute information are removed, and cases with overlapping annotated areas are excluded; then, only nodules within a set size range are retained in the CT scan image of each case, and the CT scan image is resampled using a bilinear interpolation method. During this process, the nodules in the CT scan image are extracted separately using the annotated boundaries as a true mask. At the same time, CT scan images without nodules in the CT scan sequence are used as the true background image; the cases with overlapping annotated areas refer to cases in which multiple doctors have annotated nodules in the same or adjacent positions in the same CT scan image, but the differences in their annotated ranges, shapes, or boundaries exceed a set threshold;

[0012] Step 3: Cluster the CT scan images in the preprocessed medical dataset based on attribute information and obtain several image pairs using the full permutation method;

[0013] The method of obtaining a plurality of image pairs using the full permutation method is specifically as follows: for each cluster, each CT scan image in the cluster is paired with CT scan images from other clusters;

[0014] The image pair includes two CT scan images with different attribute information, which are respectively represented as the first type of image I a and the second type of image I b , where the first type of image Ia The attribute information of the second type of image I b The attribute information is b;

[0015] Step 4: Obtain a training data set based on the obtained image pairs;

[0016] Specifically, the CNN encoder model is used to map the two CT scan images in each image pair to the latent space to obtain the source latent vectors, which are the first source latent vector δ(I a ) and the second source latent vector δ(I b ), calculate the first source potential vector δ(I a ) and the second source latent vector δ(I b ) of the intersection V, and finally the first source potential vector δ(I a ) and the second source latent vector δ(I b ) are respectively subtracted from the intersection set V to obtain the first training data d0 and the second training data d1, and the first training data d0 and the second training data d1 form a training data pair, thereby obtaining a training data set including several training data pairs;

[0017] Step 5: Normalize the training data set to obtain a normalized training data set;

[0018] Step 6: Use the normalized training data set to train the denoising diffusion probability model, and use the trained denoising diffusion probability model to obtain the control vector;

[0019] The denoising diffusion probability model is used to gradually add Gaussian noise ε~N(0,I) to the input first training data d0 and second training data d1, simulating the gradual degradation of the training data during the diffusion process. As time goes by, the noise level gradually increases until it reaches a predetermined noise level at time t. Then, the U-Net network is used as the denoising network for iterative denoising to obtain two new vectors d'0 and d'1. The restored new vector d' j The two latent vectors are obtained by row scaling and shifting, j = 0, 1; the two latent vectors are added to the two source latent vectors respectively to obtain the control vector w i , i = 1, 2, where j represents the number of the latent vector, i represents the number of the control vector, N(0, I) represents the normal distribution, I represents the unit moment, and ε represents the Gaussian noise sampled from the normal distribution;

[0020] Control vector w i for:

[0021] w i =δ(I s )+Vs' (1)

[0022] Among them, Vs' is the processed potential vector, δ(I s ) is the source latent vector, s is the number of the source latent vector and s = 1, 2;

[0023] The loss function calculation formula of the denoising diffusion probability model is as follows:

[0024] L mse =(ε-ε(d0,d t )) 2 (2)

[0025] Among them, L mse represents the mean square error loss function, ε(d0,d t ) represents the noise predicted by the denoising network at time step t;

[0026] Step 7: Build an improved DeepLabv3+ segmentation model, train the improved DeepLabv3+ segmentation model using the preprocessed medical dataset obtained in step 2, and obtain the segmentation mask using the trained improved DeepLabv3+ segmentation model;

[0027] The improved DeepLabv3+ segmentation model includes a maximum pooling layer, an encoder and a decoder;

[0028] The maximum pooling layer is arranged before the encoder;

[0029] The encoder improves the ASPP module in the original DeepLabv3+ model encoder, optimizes the convolution rate combination of the original three-layer dilated convolution from 6, 12, and 18 to 3, 6, and 12, and adds the CBAM attention mechanism at the output position of each dilated convolution and GAP;

[0030] The decoder performs 1*1 convolution on the shallow information obtained by the encoder backbone network ResNet101, and adds the CBAM attention mechanism at the 1*1 convolution output position of the original DeepLabv3+ model decoder to obtain enhanced shallow features. The output result of the encoder is upsampled by 4 times bilinear interpolation to obtain high-level features. These enhanced shallow features are then merged with the high-level features using the Concat function, and a 3×3 convolution is performed. A sub-pixel convolution layer is added at the 3×3 convolution output position of the original DeepLabv3+ model decoder to obtain a segmentation mask.

[0031] The loss function used by the improved DeepLabv3+ segmentation model is a composite loss function that combines the binary cross entropy loss function BCE and Dice loss. The calculation formula of the composite loss function is as follows:

[0032] Loss=σ×BCE_loss+(1-σ)×Dice_loss (3)

[0033] Among them, Loss represents the composite loss function, σ is the weight parameter, BCE_loss represents the binary cross entropy loss function, and Dice_loss represents the Dice loss;

[0034] Step 8: Multiply the ground truth mask pixel by pixel with the preprocessed CT scan image in the medical dataset to obtain a background image with holes.

[0035] Step 9: Use the full permutation method on the segmentation mask obtained in step 7 to generate several mask image pairs;

[0036] Specifically: for the segmentation mask in each cluster, pair it with the segmentation mask from other clusters to obtain a mask image pair including two segmentation masks with different attribute information;

[0037] Step 10: Construct a pseudo-time window-based twin framework generation model, train the pseudo-time window-based twin framework generation model using mask image pairs and control vectors, and use the trained pseudo-time window-based twin framework generation model to obtain two false mask images; the false mask images are the intermediate states of the mask image pairs of the pseudo-time window-based twin framework generation model;

[0038] The pseudo-time window-based twin framework generation model inputs a set of mask image pairs each time, including the segmentation mask X a and segmentation mask X b , where X a is the segmentation mask with attribute information a, X b is a segmentation mask with attribute information b, and the output is two false mask images;

[0039] The twin framework generation model based on pseudo time window includes two branches, each branch is AttGAN network, including two sub-networks: generator module and discriminator module. The generator module includes G enc Encoder and G dec The decoder and discriminator modules include the discriminator D and the classifier C. The networks of the two branches share weight parameters and the forward propagation process is exactly the same;

[0040] The G enc The encoder is used to convert the input segmentation mask into a latent vector;

[0041] The G dec The decoder is based on G enc The latent vector transformed by the encoder edits the attribute information of the segmentation mask into the target attribute information, that is, the control vector w i, get a false mask image; at the same time, perform reconstruction learning, input the potential vector and attribute information into G dec The decoder generates a reconstructed image;

[0042] Then, the segmentation mask and the false mask image are input to the discriminator D for game training of true and false image discrimination. At the same time, the false mask image is input to the classifier C for fine-grained attribute discrimination.

[0043] The loss function of the pseudo-time window-based twin frame generation model retains the reconstruction loss and cross entropy loss in the original AttGAN. The reconstruction loss and cross entropy loss are calculated using the reconstructed image. At the same time, a pre-trained Resnet101 feature extraction network is used to add content loss on the basis of the reconstruction loss and cross entropy loss. The calculation formula is:

[0044]

[0045]

[0046] Among them, f(·) is a pre-trained Resnet101 feature extraction network, Indicates the segmentation mask with the lth attribute information a, represents the false mask image with the lth attribute information w1; Indicates the segmentation mask with the lth attribute information b, represents the false mask image with the lth attribute information w2, n is the number of mask image pairs; L a According to the segmentation mask Calculate the content loss, L b For the segmentation mask Calculated content loss;

[0047] Step 11: Use the background image with holes and the true mask to train the U-Net-based hole filling model, and use the trained U-Net-based hole filling model to obtain the filled image;

[0048] The U-Net-based hole filling model is used to perform partial convolution processing and feature fusion on the input background image with holes and the real mask, and use the real background image as a guide to obtain the filled image;

[0049] Define the pixel loss of the U-Net-based hole filling model, and the calculation formula is expressed as:

[0050]

[0051]

[0052] Among them, L hole is the pixel loss in the hole area, is the number of pixels on the true background image for guidance, M is the original mask, and I out is the filled image, I gt Guide the generation of images, To guide the number of pixels on the generated image, L valid is the pixel loss in the non-hole area;

[0053] Finally, we define the perceptual loss and calculate the difference between the two images using the pre-trained Resnet101 network. The perceptual loss calculation formula is:

[0054]

[0055] in, represents the perceived loss, represents the feature map of the pth layer extracted by the pre-trained ResNet101 network, where I is I comp , I out or I gt ; yes The number of elements in; P is the number of feature maps, that is, a total of P layers are extracted, I comp is a background image with holes;

[0056] Step 12: Use the Alpha fusion method to superimpose the filled image and the false mask image to obtain the reconstructed image.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] This paper proposes a method for reconstructing lung nodule image attribute growth sequences based on attribute interaction generation. It designs an improved DeepLabv3+ segmentation model and a twin framework generative model to achieve higher accuracy in nodule generation. By introducing denoising diffusion probability, a large number of attribute edit vectors can be easily obtained, and then the corresponding nodules can be generated. This invention improves the quality of the reconstructed image, with a higher structural similarity (SSIM) index and a significantly lower perceptual image quality score (LPIPS). It is expected to become a valuable tool to assist doctors in analysis and promote medical research, while also expanding existing medical datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a diagram showing the architecture of a lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation in an embodiment of the present invention;

[0060] Figure 2DDPM denoising process diagram in an embodiment of the present invention;

[0061] Figure 3 This is a diagram of the improved DeepLabv3+ model architecture in an embodiment of the present invention;

[0062] Figure 4 This is a diagram of the generation model architecture of the twin framework in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The present invention is described in detail below with reference to the accompanying drawings and embodiments. The overall structure of the present invention is as follows: Figure 1 shown.

[0064] The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation includes the following steps:

[0065] Step 1: Obtain medical dataset;

[0066] The medical dataset includes several cases, each of which includes a CT scan sequence and nodule data in DICOM format. The CT scan sequence is three-dimensional imaging data, including several CT scan images, i.e., two-dimensional slices of the three-dimensional imaging data, each CT scan image covering a different layer of the lungs. The nodule data includes the location, size, and attribute information of each nodule in the CT scan image, which is crucial for assessing the nature of the nodule.

[0067] The position of each nodule is marked by three-dimensional coordinates (x, y, z), where (x, y) represents the plane position of the nodule in the CT scan image, x represents the abscissa of the plane position of the nodule, y represents the ordinate of the plane position of the nodule, and z represents the depth position of the CT scan image where the nodule is located in the CT scan sequence;

[0068] The attribute information includes occultness, margins, texture, sphericity, lobulation, burr shape, and malignancy. This detailed information helps the generative model accurately simulate the nodule growth sequence. The growth sequence simulates the morphological changes of lung nodules at different time points or stages of development based on this attribute information.

[0069] In this embodiment, the LIDC-IDRI public dataset is used for experiments.

[0070] Step 2: Preprocess the medical dataset to obtain the true mask and the true background image, and then obtain the preprocessed medical dataset;

[0071] The preprocessing method is specifically as follows: first, cases with less than a set number of attribute information are removed, and cases with overlapping annotated areas are excluded; then, only nodules within a set size range are retained in the CT scan image of each case. After the above-mentioned more detailed screening process, the CT scan image is resampled using a bilinear interpolation method. This process uses the annotated boundaries to extract the nodules in the CT scan image separately as a true mask. At the same time, the lung tissue near the nodule is retained. This is because generative learning requires more information, so the model needs to be given information about the environment around the lung nodule in order to simulate the growth environment of the nodule and generate a more realistic image. For CT scan images without nodule areas in the CT scan sequence, they are used as the real background image to guide the filling in step 11;

[0072] The case of overlapping annotation areas refers to the case in which multiple doctors have annotated nodules in the same or adjacent locations in the same CT scan image, but there are significant differences in their annotation ranges, shapes or boundaries;

[0073] In this implementation, cases with attribute information from at least three doctors were used, and nodules with a diameter of 9–30 mm were selected;

[0074] Step 3: Cluster the CT scan images in the preprocessed medical dataset based on attribute information and obtain several image pairs using the full permutation method;

[0075] The method of obtaining a plurality of image pairs using the full permutation method is specifically as follows: for each cluster, each CT scan image in the cluster is paired with CT scan images from other clusters;

[0076] The image pair includes two CT scan images with different attribute information, which are respectively represented as the first type of image I a and the second type of image I b , where the first type of image I a The attribute information of the second type of image I b The attribute information is b;

[0077] Step 4: Obtain a training data set based on the obtained image pairs;

[0078] Specifically, the CNN encoder model is used to map the two CT scan images in each image pair to the latent space to obtain the source latent vectors, which are the first source latent vector δ(I a ) and the second source latent vector δ(I b ), calculate the first source potential vector δ(I a ) and the second source latent vector δ(I b ) of the intersection V, and finally the first source potential vector δ(Ia ) and the second source latent vector δ(I b ) are respectively subtracted from the intersection set V to obtain the first training data d0 and the second training data d1, and the first training data d0 and the second training data d1 form a training data pair, thereby obtaining a training data set including several training data pairs;

[0079] Step 5: Normalize the training data set to obtain a normalized training data set to ensure that the data is within the appropriate range, which is helpful for the model learning process;

[0080] Step 6: Use the normalized training data set to train the denoised diffusion probability model (DDPM), and use the trained denoised diffusion probability model (DDPM) to obtain the control vector;

[0081] The denoising diffusion probability model includes a denoising process and a denoising process. The denoising process gradually adds Gaussian noise based on a normal distribution, and the denoising process uses U-Net as a denoising network to gradually remove noise and restore the details of the original data. The denoising diffusion probability model can learn complex image distributions. Given a dataset containing different attribute information, training DDPM can capture the complex distribution of these images. In this invention, DDPM is applied to training in the latent space to model multimodality in attribute changes.

[0082] The denoising diffusion probability model is used to gradually add Gaussian noise ε~N(0,I) to the input first training data d0 and second training data d1, simulating the gradual degradation of the training data during the diffusion process. As time goes by, the noise level gradually increases until it reaches a predetermined noise level at time t. Then, the U-Net network is used for iterative denoising. The denoising process is as follows: Figure 2 As shown, the two new vectors d'0 and d'1 are recovered, and the new vector d' j (j=0,1) row scaling and shifting are performed to obtain the two processed latent vectors, and the two processed latent vectors are added to the two source latent vectors respectively to obtain the control vector w i (i=1,2), where j represents the number of the latent vector, i represents the number of the control vector, N(0,I) represents the normal distribution, I represents the unit moment, and ε represents the Gaussian noise sampled from the normal distribution;

[0083] According to the reparameterization technique, the process of adding Gaussian noise is as follows:

[0084]

[0085] Among them, d t represents the noisy training data at time step t; dt-1 represents the noisy training data at the previous time step t-1; α t is a number between 0 and 1 that controls how much noise is added at each step;

[0086] The new vector d' will be restored j The methods for scaling and shifting are:

[0087] V'=γ*d' j +β*V (2)

[0088] Among them, V' is the processed latent vector, γ is the diversity parameter, and β is the scale parameter;

[0089] Control vector w i for:

[0090] w i =δ(I s )+Vs' (3)

[0091] Among them, Vs' is the processed potential vector, δ(I s ) is the source latent vector, s is the number of the source latent vector and s = 1, 2;

[0092] In order to better restore the original data, this paper selects U-Net as the denoising network and uses the mean square error loss function. The loss function calculation formula of the denoising diffusion probability model is as follows:

[0093] L mse =(ε-ε(d0,d t )) 2 (4)

[0094] Among them, L mse represents the mean square error loss function, ε(d0,d t ) represents the noise predicted by the denoising network at time step t;

[0095] Step 7: Build an improved DeepLabv3+ segmentation model, train the improved DeepLabv3+ segmentation model using the preprocessed medical dataset obtained in step 2, and obtain the segmentation mask using the trained improved DeepLabv3+ segmentation model;

[0096] In order to solve the problem of unsatisfactory segmentation of small nodules in the DeepLabv3+ model, this paper adopts an improved DeepLabv3+ model, which mainly combines the ResNet101 network with CBAM to optimize the original DeepLabv3+ model. The improved DeepLabv3+ segmentation model is as follows: Figure 3 As shown, it includes the maximum pooling layer, encoder and decoder;

[0097] The maximum pooling layer is set before the encoder of the original DeepLabv3+ model to enhance the key feature focus and robustness. This method helps the improved DeepLabv3+ segmentation model focus on the key parts of the image, improves the robustness of the improved DeepLabv3+ segmentation model to small changes in the input image, and reduces the complexity of subsequent calculations;

[0098] To enhance the segmentation effect of different targets, the encoder improves the ASPP module in the original DeepLabv3+ model encoder, optimizing the convolution rate combination of the original three-layer dilated convolutions from 6, 12, and 18 to 3, 6, and 12, thereby improving deep feature information and thus improving the segmentation accuracy of the network. At the same time, the CBAM attention mechanism is added to the output position of each dilated convolution and GAP.

[0099] The decoder performs 1*1 convolution on the shallow information obtained by the encoder backbone network ResNet101, and adds the CBAM attention mechanism at the 1*1 convolution output position of the original DeepLabv3+ model decoder to obtain enhanced shallow features. The output result of the encoder is upsampled by 4 times bilinear interpolation to obtain high-level features. These enhanced shallow features are then merged with the high-level features using the Concat function, and a 3×3 convolution is performed. A sub-pixel convolution layer (ESPCN) is added at the 3×3 convolution output position of the original DeepLabv3+ model decoder to obtain a high-resolution segmentation mask, thereby improving segmentation accuracy.

[0100] The loss function used by the improved DeepLabv3+ segmentation model is a composite loss function that combines the binary cross entropy loss function (BCE) and the Dice loss. The BCE loss function compares the difference between the true label and the predicted label, while the Dice loss function mainly measures the similarity between the model's prediction result and the true label. It can simultaneously optimize pixel-level classification accuracy and region overlap. The calculation formulas for BCE, Dice, and the composite loss function are as follows:

[0101]

[0102] Among them, BCE_loss represents the binary cross entropy loss function, ∈ represents the pixel label of the true mask, represents the probability of the segmentation mask predicted by the improved DeepLabv3+ segmentation model;

[0103] The calculation formula of Dice loss is as follows:

[0104]

[0105] Among them, Dice_loss represents Dice loss;

[0106] The calculation formula of the composite loss function is as follows:

[0107] Loss=σ×BCE_loss+(1-σ)×Dice_loss (7)

[0108] Among them, Loss represents the composite loss function, and σ is the weight parameter used to balance the contribution of BCE_loss and Dice_loss;

[0109] In this embodiment, the evaluation indicators used for the segmentation model are IOU (Intersection over Union) and Dice coefficient, which reach 78.0% and 86.8% respectively.

[0110] Step 8: Multiply the ground truth mask pixel by pixel with the preprocessed CT scan image in the medical dataset to obtain a background image with holes.

[0111] Step 9: Use the full permutation method on the segmentation mask obtained in step 7 to generate several mask image pairs;

[0112] Specifically: for the segmentation mask in each cluster, pair it with the segmentation mask from other clusters to obtain a mask image pair including two segmentation masks with different attribute information;

[0113] Step 10: Construct a pseudo-time window-based twin framework generation model, train the pseudo-time window-based twin framework generation model using mask image pairs and control vectors, and use the trained pseudo-time window-based twin framework generation model to obtain two false mask images; the false mask images are the intermediate states of the mask image pairs of the pseudo-time window-based twin framework generation model;

[0114] The pseudo-time window refers to the model tracking the state changes of nodule data as the number of training rounds increases, ultimately simulating the growth trend of the nodule and generating an intermediate state at a specific moment. Simply put, we use full permutations to combine data, inputting a set of data pairs into the model each time. As the number of iterations increases, the model can analyze deep semantic information based on attributes and simulate nodule formation trends.

[0115] like Figure 4 As shown, the pseudo-time window-based twin framework generation model includes two branches, each branch is an AttGAN network, including two sub-networks: the generator module and the discriminator module. The generator module includes G enc Encoder and G dec The decoder and discriminator modules include the discriminator D and the classifier C. Like the basic deep twin neural network, the two branches of the network share weight parameters and the forward propagation process of the data is exactly the same.

[0116] The AttGAN network is a generative adversarial network that focuses on attribute editing. It can modify specific attributes of an image while keeping the main structure of the image unchanged. In the pseudo-time window-based twin framework generation model, the AttGAN network is used as the core structure of each branch. Through its generator and discriminator sub-networks, it can achieve accurate generation and discrimination of lung nodule images. The pseudo-time window-based twin framework generation model inputs a set of mask image pairs each time, including the segmentation mask X a and segmentation mask X b , where X a is the segmentation mask with attribute information a, X b is the segmentation mask with attribute information b;

[0117] G enc The encoder is essentially a feature extraction network that converts the input segmentation mask into a latent vector Z1;

[0118] Z1=G enc (X a ) (8)

[0119] G dec The decoder is responsible for converting the latent vector into the target, converting the low-dimensional data back to the original data space. Through multiple iterations of this conversion process, an effective data distribution can be learned;

[0120] The G dec The decoder edits the attribute information of the segmentation mask into the target attribute information according to the input potential vector, that is, the control vector w i , get a false mask image, the process is expressed as:

[0121]

[0122] in, Indicates a false mask image with attribute information w1;

[0123] In order to keep the image semantics other than the target attribute information unchanged, reconstruction learning is required to input the latent vector Z1 and attribute information a into G dec Generate reconstructed image The process is expressed as:

[0124]

[0125] Mask image X b Generate a fake mask image using the same method and reconstructed image

[0126] The segmentation mask and the false mask image are input to the discriminator D for game training on true and false image discrimination. At the same time, the false mask image is input to the classifier C for fine-grained attribute discrimination.

[0127] In the loss function, the reconstruction loss and cross entropy loss in the original AttGAN are retained. A pre-trained Resnet101 feature extraction network is used to add content loss on the basis of the reconstruction loss and cross entropy loss. The calculation formula is:

[0128]

[0129]

[0130] Among them, f(·) is a pre-trained Resnet101 feature extraction network, Indicates the segmentation mask with the lth attribute information a, represents the false mask image with the lth attribute information w1; Indicates the segmentation mask with the lth attribute information b, represents the false mask image with the lth attribute information w2, n is the number of mask image pairs; L a According to the segmentation mask Calculate the content loss, L b For the segmentation mask Calculated content loss;

[0131] Step 11: Use the background image with holes and the true mask to train the U-Net-based hole filling model, and use the trained U-Net-based hole filling model to obtain the filled image;

[0132] The U-Net-based hole filling model is used to perform partial convolution processing and feature fusion on the input background image with holes and the real mask, using the real background image as a guide to obtain the filled image;

[0133] Specifically, the model replaces all convolutional layers with partial convolutional layers and uses nearest neighbor upsampling in the decoding stage. The skip connection connects two feature maps and two masks respectively as the feature and mask inputs of the next partial convolutional layer. The input of the last partial convolutional layer will contain the connection between the original input image and the hole and the original mask, enabling the model to replicate the pixel information around the hole and improve robustness.

[0134] The partial convolution layer can arbitrarily define a binary mask of size C×H×W, making it the same size as the image it is associated with, and then use a fixed convolution layer to implement the mask update, whose kernel size is the same as the partial convolution operation, but the weights are set to 1 and there is no bias;

[0135] In this task, the goal of the loss function is to reconstruct and reassemble each pixel, that is, to predict the smoothness of the transition from the hole value to the surrounding environment. The background image with holes I in As the input of the model, the pixel loss of the U-Net-based hole filling model is defined, and the calculation formula is expressed as:

[0136]

[0137]

[0138] Among them, L hole is the pixel loss in the hole area, is the number of pixels on the true background image for guidance, M is the original mask (holes are 1), I out is the filled image, I gt Guide the generation of images, To guide the number of pixels on the generated image, L valid is the pixel loss in the non-hole area;

[0139] Finally, the perceptual loss is defined, and the difference between two images is calculated through the pre-trained Resnet101 network. The perceptual loss calculation formula is expressed as:

[0140]

[0141] in, represents the perceived loss, represents the feature map of the pth layer extracted by the pre-trained ResNet101 network, where I is I comp , I out or I gt ; yes The number of elements in; P is the number of feature maps, that is, a total of P layers are extracted, I comp is a background image with holes;

[0142] Step 12: Use the Alpha fusion method to superimpose the filled image and the false mask image to obtain the reconstructed image;

[0143] Alpha fusion is a process of superimposing a false mask image as the foreground image and a filled image as the background image. It is a commonly used technique in image processing. The formula is as follows:

[0144]

[0145] Among them, C o Represents the reconstructed image, and α is used to represent the foreground color, i.e. The visibility (transparency) in the reconstructed image is larger, indicating The greater the proportion in the result, when α=1, the result is exactly equal to (completely opaque); on the contrary, when α=0, Completely invisible in the result (fully transparent).

Claims

1. A lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation, characterized in that: The following steps are involved: Step 1: Obtain a medical dataset; the medical dataset includes several cases, each of which includes a CT scan sequence and nodule data. The CT scan sequence is three-dimensional imaging data, including several CT scan images, i.e., two-dimensional slices of the three-dimensional imaging data, each CT scan image covering a different layer of the lungs; the nodule data includes the location, size, and attribute information of each nodule in the CT scan image; Step 2: Preprocess the medical dataset to obtain the true mask and the true background image, and then obtain the preprocessed medical dataset; Step 3: Cluster the CT scan images in the preprocessed medical dataset based on attribute information and obtain several image pairs using the full permutation method; Step 4: Obtain a training data set based on the obtained image pairs; Step 5: Normalize the training data set to obtain a normalized training data set; Step 6: Use the normalized training data set to train the denoising diffusion probability model, and use the trained denoising diffusion probability model to obtain the control vector; Step 7: Build an improved DeepLabv3+ segmentation model, train the improved DeepLabv3+ segmentation model using the preprocessed medical dataset obtained in step 2, and obtain the segmentation mask using the trained improved DeepLabv3+ segmentation model; Step 8: Multiply the ground truth mask pixel by pixel with the preprocessed CT scan image in the medical dataset to obtain a background image with holes. Step 9: Use the full permutation method on the segmentation mask obtained in step 7 to generate several mask image pairs; Step 10: Construct a pseudo-time window-based twin framework generation model, train the pseudo-time window-based twin framework generation model using mask image pairs and control vectors, and use the trained pseudo-time window-based twin framework generation model to obtain two false mask images; Step 11: Use the background image with holes and the true mask to train the U-Net-based hole filling model, and use the trained U-Net-based hole filling model to obtain the filled image; Step 12: Use the Alpha fusion method to superimpose the filled image and the false mask image to obtain the reconstructed image.

2. The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation according to claim 1 is characterized in that: The position of each nodule described in step 1 is marked by three-dimensional coordinates (x, y, z), where (x, y) represents the planar position of the nodule in the CT scan image, x represents the horizontal coordinate of the planar position of the nodule, y represents the vertical coordinate of the planar position of the nodule, and z represents the depth position of the CT scan image where the nodule is located in the CT scan sequence.

3. The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation according to claim 1 is characterized in that: The preprocessing method described in step 2 is specifically as follows: first, cases with less than a set number of attribute information are removed, and cases with overlapping annotation areas are excluded; then, only nodules within the set size range in the CT scan image of each case are retained, and the CT scan image is resampled using the bilinear interpolation method. In this process, the nodules in the CT scan image are extracted separately using the annotation boundary as the true mask. At the same time, for CT scan images without nodules in the CT scan sequence, they are used as the true background image; the cases with overlapping annotation areas refer to cases in which multiple doctors have annotated nodules in the same or adjacent positions in the same CT scan image, but the differences in their annotation ranges, shapes or boundaries exceed the set threshold.

4. The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation according to claim 1 is characterized in that: The method of obtaining a plurality of image pairs using the full permutation method in step 3 is specifically as follows: for each cluster, each CT scan image in the cluster is paired with a CT scan image from another cluster; The image pair includes two CT scan images with different attribute information, which are respectively represented as the first type of image I a and the second type of image I b , where the first type of image I a The attribute information of the second type of image I b The attribute information is b.

5. The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation according to claim 1 is characterized in that: The step 4 is specifically as follows: using the CNN encoder model to map the two CT scan images in each image pair to the latent space to obtain source latent vectors, which are the first source latent vector δ(I a ) and the second source latent vector δ(I b ), calculate the first source potential vector δ(I a ) and the second source latent vector δ(I b ) of the intersection V, and finally the first source potential vector δ(I a ) and the second source latent vector δ(I b ) are respectively subtracted from the intersection V to obtain the first training data d0 and the second training data d1, and the first training data d0 and the second training data d1 constitute a training data pair, thereby obtaining a training data set including several training data pairs.

6. The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation according to claim 1, characterized in that: The denoising diffusion probability model described in step 6 is used to gradually add Gaussian noise ε~N(0,I) to the input first training data d0 and the second training data d1, simulating the gradual degradation of the training data during the diffusion process. As time goes by, the noise level gradually increases until it reaches the predetermined noise level at time t. Then, the U-Net network is used as the denoising network for iterative denoising to obtain two new vectors d'0 and d'1. The restored new vector d' j The two latent vectors are obtained by row scaling and shifting, j = 0, 1; the two latent vectors are added to the two source latent vectors respectively to obtain the control vector w i , i = 1, 2, where j represents the number of the latent vector, i represents the number of the control vector, N(0, I) represents the normal distribution, I represents the unit moment, and ε represents the Gaussian noise sampled from the normal distribution; Control vector w i for: w i =δ(I s )+Vs' (1) Among them, Vs' is the processed potential vector, δ(I s ) is the source latent vector, s is the number of the source latent vector and s = 1, 2; The loss function calculation formula of the denoising diffusion probability model is as follows: L mse =(ε-ε(d0,d t )) 2 (2) Among them, L mse represents the mean square error loss function, ε(d0,d t ) represents the noise predicted by the denoising network at time step t.

7. The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation according to claim 1 is characterized in that: The improved DeepLabv3+ segmentation model described in step 7 includes a max pooling layer, an encoder, and a decoder; The maximum pooling layer is arranged before the encoder; The encoder improves the ASPP module in the original DeepLabv3+ model encoder, optimizes the convolution rate combination of the original three-layer dilated convolution from 6, 12, and 18 to 3, 6, and 12, and adds the CBAM attention mechanism at the output position of each dilated convolution and GAP; The decoder performs 1*1 convolution on the shallow information obtained by the encoder backbone network ResNet101, and adds the CBAM attention mechanism at the 1*1 convolution output position of the original DeepLabv3+ model decoder to obtain enhanced shallow features. The output result of the encoder is upsampled by 4 times bilinear interpolation to obtain high-level features. These enhanced shallow features are then merged with the high-level features using the Concat function, and a 3×3 convolution is performed. A sub-pixel convolution layer is added at the 3×3 convolution output position of the original DeepLabv3+ model decoder to obtain a segmentation mask. The loss function used by the improved DeepLabv3+ segmentation model is a composite loss function that combines the binary cross entropy loss function BCE and Dice loss. The calculation formula of the composite loss function is as follows: Loss=σ×BCE_loss+(1-σ)×Dice_loss (3) Among them, Loss represents the composite loss function, σ is the weight parameter, BCE_loss represents the binary cross entropy loss function, and Dice_loss represents the Dice loss.

8. The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation according to claim 1 is characterized in that: The step 9 specifically includes: pairing the segmentation mask in each cluster with the segmentation masks from other clusters to obtain a mask image pair including two segmentation masks with different attribute information.

9. The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation according to claim 1, characterized in that: The false mask image in step 10 is an intermediate state of the mask image pair of the twin framework generation model of the input pseudo time window; The pseudo-time window-based twin framework generation model inputs a set of mask image pairs each time, including the segmentation mask X a and segmentation mask X b , where X a is the segmentation mask with attribute information a, X b is a segmentation mask with attribute information b, and the output is two false mask images; The twin framework generation model based on pseudo time window includes two branches, each branch is AttGAN network, including two sub-networks: generator module and discriminator module. The generator module includes G enc Encoder and G dec The decoder and discriminator modules include the discriminator D and the classifier C. The networks of the two branches share weight parameters and the forward propagation process is exactly the same; The G enc The encoder is used to convert the input segmentation mask into a latent vector; The G dec The decoder is based on G enc The latent vector transformed by the encoder edits the attribute information of the segmentation mask into the target attribute information, that is, the control vector w i , get a false mask image; at the same time, perform reconstruction learning, input the potential vector and attribute information into G dec The decoder generates a reconstructed image; Then, the segmentation mask and the false mask image are input to the discriminator D for game training of true and false image discrimination. At the same time, the false mask image is input to the classifier C for fine-grained attribute discrimination. The loss function of the pseudo-time window-based twin frame generation model retains the reconstruction loss and cross entropy loss in the original AttGAN. The reconstruction loss and cross entropy loss are calculated using the reconstructed image. At the same time, a pre-trained Resnet101 feature extraction network is used to add content loss on the basis of the reconstruction loss and cross entropy loss. The calculation formula is: Among them, f(·) is a pre-trained Resnet101 feature extraction network, Indicates the segmentation mask with the lth attribute information a, represents the false mask image with the lth attribute information w1; Indicates the segmentation mask with the lth attribute information b, represents the false mask image with the lth attribute information w2, n is the number of mask image pairs; L a According to the segmentation mask Calculate the content loss, L b For the segmentation mask Compute the content loss.

10. The lung nodule image attribute growth sequence reconstruction method based on attribute interaction generation according to claim 1, characterized in that: The U-Net-based hole filling model described in step 11 is used to perform partial convolution processing and feature fusion on the input background image with holes and the true mask, and use the true background image as a guide to obtain the filled image; Define the pixel loss of the U-Net-based hole filling model, and the calculation formula is expressed as: Among them, L hole is the pixel loss in the hole area, is the number of pixels on the true background image for guidance, M is the original mask, and I out is the filled image, I gt Guide the generation of images, To guide the number of pixels on the generated image, L valid is the pixel loss in the non-hole area; Finally, we define the perceptual loss and calculate the difference between the two images using the pre-trained Resnet101 network. The perceptual loss calculation formula is: in, represents the perceived loss, represents the feature map of the pth layer extracted by the pre-trained ResNet101 network, where I is I comp , I out or I gt ; yes The number of elements in; P is the number of feature maps, that is, a total of P layers are extracted, I comp is a background image with holes.

Images