Lung nodule growth prediction method based on combination of lstm and gan

Abstract

The application discloses a lung nodule growth prediction method based on combination of LSTM and GAN, relates to the technical field of image analysis and processing, and comprises the following steps: collecting multi-time chest CT data, performing labeling and auditing to form a data set; performing pretreatment on the labeled CT data to obtain standardized lung nodule ROIs; segmenting out lung nodule regions by using a three-dimensional image segmentation network to generate three-dimensional segmentation masks; inputting the segmentation masks of multiple time nodes into an LSTM network to extract time sequence feature vectors representing dynamic evolution of the nodule; based on the time sequence feature vectors and random noise, a generator synthesizes predicted lung nodule images at future time points, and a discriminator optimizes model parameters by joint optimization of multiple loss functions; and inputting three-dimensional segmentation masks at the current and historical time points to generate predicted lung nodule images at future time points. The application can effectively improve the situation that the prior art is insufficient in utilization of time sequence information and lacks high-quality generation and inference capability.

Description

A lung nodule growth prediction method based on the combination of LSTM and GAN Technical Field

[0001] This invention relates to the field of image analysis and processing technology, specifically a lung nodule growth prediction method based on a combination of LSTM and GAN. Background Technology

[0002] Pulmonary nodules are important indicators for early medical screening and diagnosis, typically detected and followed up using chest CT scans. Long-term observation of the size, shape, and density of pulmonary nodules helps determine disease progression and subsequent treatment plans. However, traditional pulmonary nodule detection and follow-up rely heavily on physician experience and manual measurements, often introducing subjectivity and error. Furthermore, pulmonary nodules may show no significant changes in the short term during long-term follow-up, but may undergo abrupt changes during medium- or long-term follow-up. Therefore, accurately predicting the future morphology and growth trend of pulmonary nodules is crucial for improving the efficiency of early diagnosis and intervention.

[0003] Currently, many methods for detecting and predicting lung nodules still rely on traditional rule-based or classical machine learning approaches. These methods require designers to manually extract and filter features from medical images (such as shape, texture, and density). However, medical images typically contain rich and subtle features, making it difficult for humans to simultaneously consider both subjective judgment and the completeness of objective information, resulting in insufficient prediction accuracy. Furthermore, lung nodule growth is not independent; it exhibits dynamic characteristics that evolve over time. Most existing technologies focus on static image analysis at single or multiple time points, failing to fully utilize time-series information and unable to accurately predict the future growth trend of nodules in the early stages. In lung nodule growth prediction, researchers, in addition to judging the growth rate or trend, are more concerned with potential morphological changes in nodules. However, most existing methods only predict at the numerical or classification level, lacking the ability to generate realistic images of future nodules. Therefore, when clinicians need to observe "what form a future nodule might evolve into," the reference value of existing methods is limited. Summary of the Invention

[0004] The purpose of this invention is to provide a method for predicting lung nodule growth based on a combination of LSTM and GAN, in order to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a lung nodule growth prediction method based on a combination of LSTM and GAN, the method comprising the following steps:

[0006] Step 1: Collect multi-time series chest CT image data, annotate and review them to form a dataset;

[0007] Step 2: Preprocess the labeled CT image data to obtain standardized ROIs for lung nodules;

[0008] Step 3: Use a 3D image segmentation network to segment the lung nodule region from the preprocessed ROI and generate a 3D segmentation mask for the lung nodule;

[0009] Step 4: Input the segmentation masks of multiple time nodes into the LSTM network in chronological order to extract the temporal feature vectors representing the dynamic evolution of nodules;

[0010] Step 5: Based on temporal feature vectors and random noise, the generator synthesizes predicted future lung nodule images, and the discriminator optimizes model parameters through multiple loss functions.

[0011] Step 6: Input the 3D segmentation masks of the current and historical moments to generate the predicted lung nodule images for future moments.

[0012] In step 1, multi-temporal chest CT image data containing pulmonary nodules are collected from hospitals or databases; the multi-temporal chest CT image data includes continuous tracking scans at at least two time points (e.g., initial diagnosis, 3 months, 6 months, 12 months, etc.);

[0013] Radiologists use medical imaging software to manually or semi-automatically annotate the 3D ROI of lung nodules and generate annotation masks in NIfTI format;

[0014] The annotation results are independently reviewed by at least two physicians to ensure the consistency and accuracy of the annotations, forming an annotated dataset that includes time-series CT images and corresponding masks.

[0015] In step 2, the preprocessing specifically includes: denoising, standardizing, and registering the labeled CT images and masks to ensure that the same nodule at different time points is spatially aligned;

[0016] ROI cropping: Cropping three-dimensional regions of interest containing lung nodules from CT images based on labeled masks;

[0017] Resampling unifies CT images to a preset voxel size (e.g., 1×1×1mm^3), reducing spatial resolution inconsistencies caused by different equipment or scanning parameters;

[0018] Truncate the CT values ​​to a window width range of [-1000HU, 400HU] or wider (e.g., [-1200, 600]) and then normalize them (01 or -11) to maintain the stability of the network input.

[0019] In three-dimensional space, the data is randomly flipped, rotated, translated, Gamma corrected, and noise injected to improve the robustness of the model to different scanning conditions and lesion morphologies.

[0020] The preprocessed ROI image sequence of lung nodules was obtained.

[0021] In step 3, the labeled dataset is divided into a training set, a validation set, and a test set (e.g., 8:1:1).

[0022] Three-dimensional segmentation of lung nodules is achieved using any of the following networks:

[0023] (1) Automated segmentation based on nnU-Net: Automatically configure 3D U-Net or 2D+3D hybrid architecture according to the size, resolution, number of channels, etc. of the training data, eliminating the tedious manual parameter debugging;

[0024] The nnU-Net script is invoked to perform patch segmentation, data augmentation, and multi-fold cross-validation; the optimal model weights are automatically selected based on the Dice score; the test set data is segmented using the inference script built into nnU-Net, and the model automatically performs post-processing such as patch stitching and small connected component removal to obtain clean 3D segmentation results;

[0025] (2) Global feature fusion based on TransUNet: The Vision Transformer module is fused into the U-Net encoder, and the global self-attention mechanism is used to capture the long-distance association between lung nodules and surrounding tissues;

[0026] A multi-scale skip connection structure similar to that of traditional U-Net is adopted to fuse shallow local features with deep global features; for scenarios with small and irregular lung nodules, skip connections help to preserve details;

[0027] Set the patch size according to the GPU memory (e.g., [128, 128, 128]) and select an appropriate loss function: Dice + Cross Entropy or Focal Dice; use learning rate decay or cosine annealing strategy for training to ensure smooth convergence.

[0028] (3) Layered attention segmentation based on Swin-UNet: The input 3D patch is divided into blocks, and multi-head self-attention is performed sequentially in the local window, and the layers are stacked to form a layered structure; after each stage of the encoder is completed, it works with the U-Net decoder to output a high-resolution segmentation map.

[0029] If the 3D version of the Swing Transformer is used directly, the computational load needs to be considered, which can be alleviated by using downsampling strategies or 2.5D variants;

[0030] When using 3D Attention, it is more sensitive to interactions in remote regions and is suitable for capturing the edges of low-density, morphologically blurred nodules.

[0031] Set the Patch sampling and batch size; after training, perform 3D inference on the test set to obtain the voxel-level segmentation mask.

[0032] In step 4, the hidden state vectors on the time series are extracted using a three-dimensional segmentation mask with multiple time nodes as the input LSTM mesh, capturing the dynamic evolution patterns of nodules at different stages, and outputting a temporal feature vector that encodes the dynamic evolution law of nodules.

[0033] The generator receives a temporal feature vector from an LSTM network and combines it with a fixed-length random noise vector. It employs a multi-scale convolutional structure or a hybrid structure of convolution and Transformer, and integrates an Attention mechanism to generate a predicted future lung nodule image.

[0034] The discriminator performs binary classification on the real future lung nodule images and the future lung nodule images predicted by the generator, adaptively improving the generator's capabilities; iterative adversarial training is performed on the generated results to continuously optimize the generator until the discriminator and the generator converge.

[0035] The complete generator loss includes adversarial loss, pixel-level reconstruction loss, structural similarity loss, feature matching loss, and 3D slice consistency loss. The multi-level and multi-angle loss function design has the following characteristics: overall shape and intensity distribution: L1 loss; preservation of lung nodule structural features: SSIM loss; edges and textures: gradient loss; semantic realism: feature matching loss; anatomical coherence in 3D space: slice consistency loss. A phased strategy is adopted during training, first training with reconstruction loss weights, and then adding adversarial loss weights.

[0036] In step 6, the LSTM mesh outputs a temporal feature vector using the current and historical 3D segmentation masks as input; and the generator synthesizes the predicted future lung nodule image using the temporal feature vector as input.

[0037] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention introduces an LSTM network to extract features from lung nodule images at different times, enabling it to capture the dynamic evolution of nodules over time and achieve accurate prediction of lung nodule morphology in the short and medium term; This invention utilizes cutting-edge segmentation algorithms to perform high-precision and robust lung nodule ROI segmentation in three-dimensional space, laying a solid data foundation for subsequent growth prediction; Compared with previous methods that could only output numerical trends, this invention, through GAN, can directly generate complete lung nodule images at future times, making it more intuitive and valuable for clinical diagnosis; After large-scale data training and testing, this invention… The inference phase requires minimal computation time to predict data from new patients; this solution is applicable to different types of CT equipment and image data from different hospitals, and can be quickly deployed in medical institutions through appropriate transfer learning or fine-tuning; this invention automatically generates prediction results through deep learning, saving doctors a significant amount of measurement and judgment time in early follow-up; doctors can compare the generated images with actual follow-up images for more targeted examinations or treatments; the network structure of this invention has good scalability, applicable not only to lung nodules but also to temporal image prediction of other organs, providing new ideas and technological expansion for deep learning in the field of medical image analysis. Attached Figure Description

[0038] Figure 1 is a schematic flowchart of the lung nodule growth prediction method based on the combination of LSTM and GAN of the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Example: As shown in Figure 1, this invention provides a technical solution for predicting lung nodule growth based on a combination of LSTM and GAN. The method includes the following steps:

[0041] Step 1: Collect multi-time series chest CT image data, annotate and review them to form a dataset;

[0042] Step 2: Preprocess the labeled CT image data to obtain standardized ROIs for lung nodules;

[0043] Step 3: Use a 3D image segmentation network to segment the lung nodule region from the preprocessed ROI and generate a 3D segmentation mask for the lung nodule;

[0044] Step 4: Input the segmentation masks of multiple time nodes into the LSTM network in chronological order to extract the temporal feature vectors representing the dynamic evolution of nodules;

[0045] Step 5: Based on temporal feature vectors and random noise, the generator synthesizes predicted future lung nodule images, and the discriminator optimizes model parameters through multiple loss functions.

[0046] Step 6: Input the 3D segmentation masks of the current and historical moments to generate the predicted lung nodule images for future moments.

[0047] In step 1, multi-temporal chest CT image data containing pulmonary nodules are collected from hospitals or databases; the multi-temporal chest CT image data includes continuous tracking scans at at least two time points (e.g., initial diagnosis, 3 months, 6 months, 12 months, etc.);

[0048] Radiologists use medical imaging software to manually or semi-automatically annotate the 3D ROI of lung nodules and generate annotation masks in NIfTI format;

[0049] The annotation results are independently reviewed by at least two physicians to ensure the consistency and accuracy of the annotations, forming an annotated dataset that includes time-series CT images and corresponding masks.

[0050] In step 2, the preprocessing specifically includes: denoising, standardizing, and registering the labeled CT images and masks to ensure that the same nodule at different time points is spatially aligned;

[0051] ROI cropping: Cropping three-dimensional regions of interest containing lung nodules from CT images based on labeled masks;

[0052] Resampling unifies CT images to a preset voxel size (e.g., 1×1×1mm^3), reducing spatial resolution inconsistencies caused by different equipment or scanning parameters;

[0053] Truncate the CT values ​​to a window width range of [-1000HU, 400HU] or wider (e.g., [-1200, 600]) and then normalize them (01 or -11) to maintain the stability of the network input.

[0054] In three-dimensional space, the data is randomly flipped, rotated, translated, Gamma corrected, and noise injected to improve the robustness of the model to different scanning conditions and lesion morphologies.

[0055] The preprocessed ROI image sequence of lung nodules was obtained.

[0056] In step 3, the labeled dataset is divided into a training set, a validation set, and a test set (e.g., 8:1:1).

[0057] Three-dimensional segmentation of lung nodules is achieved using any of the following networks:

[0058] (1) Automated segmentation based on nnU-Net: Automatically configure 3D U-Net or 2D+3D hybrid architecture according to the size, resolution, number of channels, etc. of the training data, eliminating the tedious manual parameter debugging;

[0059] The nnU-Net script is invoked to perform patch segmentation, data augmentation, and multi-fold cross-validation; the optimal model weights are automatically selected based on the Dice score; the test set data is segmented using the inference script built into nnU-Net, and the model automatically performs post-processing such as patch stitching and small connected component removal to obtain clean 3D segmentation results;

[0060] (2) Global feature fusion based on TransUNet: The Vision Transformer module is fused into the U-Net encoder, and the global self-attention mechanism is used to capture the long-distance association between lung nodules and surrounding tissues;

[0061] A multi-scale skip connection structure similar to that of traditional U-Net is adopted to fuse shallow local features with deep global features; for scenarios with small and irregular lung nodules, skip connections help to preserve details;

[0062] Set the patch size according to the GPU memory (e.g., [128, 128, 128]) and select an appropriate loss function: Dice + Cross Entropy or Focal Dice; use learning rate decay or cosine annealing strategy for training to ensure smooth convergence.

[0063] (3) Layered attention segmentation based on Swin-UNet: The input 3D patch is divided into blocks, and multi-head self-attention is performed sequentially in the local window, and the layers are stacked to form a layered structure; after each stage of the encoder is completed, it works with the U-Net decoder to output a high-resolution segmentation map.

[0064] If the 3D version of the Swing Transformer is used directly, the computational load needs to be considered, which can be alleviated by using downsampling strategies or 2.5D variants;

[0065] When using 3D Attention, it is more sensitive to interactions in remote regions and is suitable for capturing the edges of low-density, morphologically blurred nodules.

[0066] Set the Patch sampling and batch size; after training, perform 3D inference on the test set to obtain the voxel-level segmentation mask.

[0067] In step 4, the hidden state vectors on the time series are extracted using a three-dimensional segmentation mask with multiple time nodes as the input LSTM mesh, capturing the dynamic evolution patterns of nodules at different stages, and outputting a temporal feature vector that encodes the dynamic evolution law of nodules.

[0068] The generator receives a temporal feature vector from an LSTM network and combines it with a fixed-length random noise vector. It employs a multi-scale convolutional structure or a hybrid structure of convolution and Transformer, and integrates an Attention mechanism to generate a predicted future lung nodule image.

[0069] The discriminator performs binary classification on the real future lung nodule images and the future lung nodule images predicted by the generator, adaptively improving the generator's capabilities; iterative adversarial training is performed on the generated results to continuously optimize the generator until the discriminator and the generator converge.

[0070] The complete generator loss includes adversarial loss, pixel-level reconstruction loss, structural similarity loss, feature matching loss, and 3D slice consistency loss. The multi-level and multi-angle loss function design has the following characteristics: overall shape and intensity distribution: L1 loss; preservation of lung nodule structural features: SSIM loss; edges and textures: gradient loss; semantic realism: feature matching loss; anatomical coherence in 3D space: slice consistency loss. A phased strategy is adopted during training, first training with reconstruction loss weights, and then adding adversarial loss weights.

[0071] In step 6, the LSTM mesh outputs a temporal feature vector using the current and historical 3D segmentation masks as input; and the generator synthesizes the predicted future lung nodule image using the temporal feature vector as input.

[0072] In this embodiment, deep learning is used to learn the growth patterns of lung nodules at different time series, and within the framework of generative adversarial networks, lung nodule images are synthesized 3 months, 6 months, and 12 months later, thereby providing a reference for clinical diagnosis and treatment planning.

[0073] Step 1: Collect multi-time series chest CT image data, annotate and review them to form a dataset;

[0074] Collect data containing multi-time-series chest CT images of pulmonary nodules from hospitals or databases. The data should include continuous follow-up scans at at least two time points (e.g., initial diagnosis, 3 months, 6 months, 12 months, etc.).

[0075] Under the guidance of radiologists, nodule regions are marked on the CT scans of each patient; professional medical imaging software is used to draw lines or semi-automatically mark the three-dimensional ROIs of the lung nodules; and senior physicians conduct cross-review to ensure the consistency and accuracy of the markings.

[0076] Patients are managed hierarchically according to their patient ID numbers. CT images and annotation masks are stored in NIfTI or other common formats. Data is divided into training, validation, and test sets (e.g., 8:1:1) to ensure balanced distribution.

[0077] Step 2: Preprocess the labeled chest CT image data to obtain standardized ROIs for lung nodules;

[0078] The annotated CT images and the annotated mask are denoised, standardized, and registered to ensure that the same nodule at different time points is spatially aligned.

[0079] ROI cropping: Cropping three-dimensional regions of interest containing lung nodules from CT images based on labeled masks;

[0080] Resampling unifies CT images to a preset voxel size (e.g., 1×1×1mm^3), reducing spatial resolution inconsistencies caused by different equipment or scanning parameters;

[0081] Truncate the CT values ​​to a window width range of [-1000HU, 400HU] or wider (e.g., [-1200, 600]) and then normalize them (01 or -11) to maintain the stability of the network input.

[0082] Random flipping, rotation, translation, Gamma correction, and noise injection are performed in three-dimensional space to improve the robustness of the model to different scanning conditions and lesion morphologies.

[0083] By masking some non-lung tissue areas and retaining only the nodular structures within the lungs, the probability of false positives generated by the model outside the lungs is further reduced.

[0084] Obtain the preprocessed ROI image sequence of lung nodules;

[0085] Step 3: Use a 3D image segmentation network to segment the lung nodule region from the standardized lung nodule ROI and generate a 3D segmentation mask for the lung nodule;

[0086] This invention can use any of the following representative three-dimensional network structures to automatically segment the lung nodule region; the specific implementation can be selected according to the actual situation:

[0087] 1) nnU-Net (Automation Pipeline):

[0088] Automatic configuration: nnU-Net will automatically configure 3D U-Net or 2D+3D hybrid architecture according to the size, resolution, number of channels, etc. of the training data, eliminating the need for tedious manual parameter tuning;

[0089] Training process: The nnU-Net script is called to perform patch splitting, data augmentation, and multi-fold cross-validation; the optimal model weights are automatically selected based on metrics such as Dice score;

[0090] Inference and Post-processing: The test set data is segmented using the inference script built into nnU-Net. The model automatically performs post-processing such as patch stitching and small connected component removal to obtain clean 3D segmentation results.

[0091] 2) TransUNet (a hybrid structure of Transformer and U-Net):

[0092] Network encoder: The Vision Transformer module is integrated into the U-Net encoder to capture the long-distance correlation between lung nodules and surrounding tissues using a global self-attention mechanism;

[0093] Skip connections and decoding: A multi-scale skip connection structure similar to the traditional U-Net is adopted to fuse shallow local features with deep global features. For scenarios with small and irregular lung nodules, skip connections help preserve details;

[0094] Training hyperparameters: Set an appropriate patch size (e.g., [128, 128, 128]) based on GPU memory, and select a suitable loss function (Dice+Cross Entropy, Focal Dice, etc.).

[0095] It can be combined with learning rate decay or cosine annealing strategies to ensure smooth training convergence.

[0096] 3) Swin-UNet (Hierarchical Transformer Network):

[0097] The basic principle of Swin Transformer is to divide the input 3D patch into blocks, perform multi-head self-attention sequentially within local windows, and stack them layer by layer to form a hierarchical structure; after each stage of the encoder is completed, it works with the U-Net decoder to output a high-resolution segmentation map.

[0098] 3D implementation details: If the 3D version of the Swing Transformer is used directly, the amount of computation needs to be considered. This can be alleviated by using downsampling strategies or 2.5D variants. When using 3D Attention, it is more sensitive to interactions in remote regions and is suitable for capturing the edges of low-density, morphologically blurred nodules.

[0099] Network training and inference: Similar to TransUNet, the patch sampling and batch size need to be carefully set; after training, 3D inference is performed on the test set to obtain voxel-level segmentation masks.

[0100] By leveraging cutting-edge segmentation algorithms, high-precision and robust ROI segmentation of lung nodules is performed in three-dimensional space, laying a solid data foundation for subsequent growth prediction.

[0101] Step 4: Input the segmentation masks of multiple time nodes into the LSTM network in chronological order to extract the temporal feature vectors representing the dynamic evolution of nodules;

[0102] Temporal feature extraction (LSTM part): Input lung nodule images or their features (such as multi-scale sensing features) at different time points into the LSTM network to extract hidden state vectors in the time series and capture the dynamic evolution pattern of nodules at different stages.

[0103] Step 5: Based on temporal feature vectors and random noise, the generator synthesizes predicted future lung nodule images, and the discriminator optimizes model parameters through multiple loss functions.

[0104] The LSTM output serves as a potential encoding of future nodule morphology, providing high-dimensional feature input for GAN generation.

[0105] Generative Adversarial Networks (GANs):

[0106] Generator: Receives time-series feature encodings from an LSTM network and combines them with random noise vectors (to enrich the diversity of generation) to generate lung nodule images for the next 3 months, 6 months, and 12 months;

[0107] Discriminator: The discriminator performs binary classification on real future lung nodule images (from labeled data) and images generated by the generator, thereby adaptively improving the generator's capabilities;

[0108] Through adversarial training, the generator can gradually learn to synthesize more realistic images of future lung nodules.

[0109] Loss function design: Combining adversarial loss (GAN Loss) and reconstruction loss, the generator can not only fool the discriminator, but also maintain similarity to the real target image at the pixel level or feature level; to improve the prediction accuracy of nodule details and contours, an attention mechanism can be incorporated into the model structure to focus on feature learning of key parts.

[0110] Step 6: Input the current and historical 3D segmentation masks to generate predicted lung nodule images for future moments;

[0111] Process and implementation description:

[0112] Training phase: Multi-patient, multi-time series lung nodule data are divided into training and validation sets; first, an LSTM network is used to pre-train the sequence data to learn the potential representation of lung nodules in the time dimension; then, it is combined with a generative adversarial network, and the generator is iteratively trained on the generation results through a discriminator to continuously optimize the generator; when the discriminator and generator converge, a model with good predictive ability for unknown lung nodule sequences can be obtained.

[0113] Inference phase: The model receives the patient's CT nodule image at time T0 and the image at time T1 (e.g., 3 months later), and can use the LSTM network to extract temporal features; then, it generates lung nodule prediction images at time T2 (e.g., 6 months later) through a generative adversarial network, and can also directly predict the nodule form 3, 6, and 12 months later at time T0 to assist doctors in decision-making.

[0114] The LSTM module (temporal feature extraction) inputs the patient's nodule ROIs at different time points T0, T1, T2... into the feature extraction network to obtain high-level semantic features; further, the feature sequence is input into the LSTM unit to capture the dynamic pattern of lung nodule morphology evolution over time; the hidden layer output of the LSTM is used as the condition vector of the GAN generator.

[0115] The GAN part (future image generation) generator takes the temporal features output by LSTM and random noise vector as input; it adopts a multi-scale convolutional structure or a hybrid structure of convolution + Transformer, and integrates the Attention mechanism to generate lung nodule prediction maps that can reflect the details of real lesions.

[0116] Architecture 1: Multi-scale convolution generator: This architecture focuses on capturing features at different scales through multi-scale convolutional layers and attention mechanisms, making it particularly suitable for generating lung nodule structures with rich details.

[0117] Architecture 2: Convolutional + Transformer hybrid generator; This architecture combines the local processing capabilities of convolutional neural networks with the global modeling capabilities of Transformers, making it particularly suitable for capturing long-range dependencies and complex structures of lung nodules.

[0118] GANs can directly generate complete images of lung nodules at future moments, making them more intuitive and valuable for clinical diagnosis.

[0119] The discriminator takes real-future lung nodule images and generator-generated images as input and performs adversarial binary classification.

[0120] The parameters of the generator and discriminator are jointly optimized using adversarial loss (GAN Loss) and reconstruction loss. The complete generator loss includes adversarial loss, pixel-level reconstruction loss, structural similarity loss, feature matching loss, and 3D slice consistency loss. This multi-level, multi-angle loss function design ensures that the generated lung nodule images simultaneously possess the following characteristics: good overall morphology and intensity distribution (L1 loss), preservation of important structural features of lung nodules (SSIM loss), clear edges and rich texture (gradient loss), semantic realism (feature matching loss), and anatomical coherence in 3D space (slice consistency loss). A phased strategy is adopted during training: first, high reconstruction loss weights are used to ensure the medical accuracy of the generated images, and then adversarial loss weights are increased to improve image realism and detail.

[0121] After training and testing on large-scale data, the inference phase requires minimal computation time to predict data from new patients. This solution is applicable to different types of CT equipment and imaging data from various hospitals, and can be rapidly deployed in medical institutions through appropriate transfer learning or fine-tuning.

[0122] The network structure of this invention has good scalability and is not only applicable to lung nodules, but also to temporal image prediction of other organs or lesions, such as follow-up of heart disease and prediction of brain lesion evolution, providing new ideas and technological expansion for deep learning in the field of medical image analysis.

[0123] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A lung nodule growth prediction method based on a combination of LSTM and GAN, characterized in that: The method includes the following steps: Step 1, collecting multi-time-series chest CT image data, labeling and reviewing it to form a dataset; Step 2, preprocessing the labeled chest CT image data to obtain standardized lung nodule ROIs; Step 3, using a 3D image segmentation network to segment lung nodule regions from the standardized lung nodule ROIs, generating a 3D segmentation mask for lung nodules; Step 4, inputting the segmentation masks from multiple time nodes into an LSTM network in chronological order to extract temporal feature vectors representing the dynamic evolution of nodules; Step 5, based on the temporal feature vectors and random noise, the generator synthesizes predicted future lung nodule images, and the discriminator optimizes model parameters through multiple loss functions; Step 6, inputting the current and historical 3D segmentation masks to generate predicted future lung nodule images; In Step 3, the 3D image segmentation network includes: global feature fusion based on TransUNet: fusing Vision in the U-Net encoder. The Transformer module uses a global self-attention mechanism to capture the long-distance correlation between lung nodules and surrounding tissues; it adopts a multi-scale skip connection structure similar to the traditional U-Net to fuse shallow local features with deep global features; it sets the patch size according to the GPU memory and selects the loss function: Dice+Cross Entropy or Focal Dice; it is trained with learning rate decay or cosine annealing strategies; in step 5, the multiple loss functions include adversarial loss, pixel-level reconstruction loss, structural similarity loss, feature matching loss, and 3D slice consistency loss; the multi-level, multi-angle loss function design has the following characteristics: overall shape and intensity distribution: L1 loss, preservation of lung nodule structural features: SSIM loss, edges and texture: gradient loss, semantic realism: feature matching loss, anatomical coherence in 3D space: slice consistency loss.

2. The lung nodule growth prediction method based on the combination of LSTM and GAN according to claim 1, characterized in that: In step 1, multi-time series chest CT image data containing pulmonary nodules are collected from hospitals or databases; the multi-time series chest CT image data includes continuous tracking scans at at least two time points; radiologists use medical imaging software to manually or semi-automatically annotate the pulmonary nodules with three-dimensional ROIs and generate annotation masks in NIfTI format; at least two physicians independently review the annotation results to form an annotation dataset containing time series chest CT images and corresponding annotation masks.

3. The lung nodule growth prediction method based on the combination of LSTM and GAN according to claim 2, characterized in that: In step 2, the preprocessing specifically includes: denoising, standardizing, and registering the labeled chest CT images and the labeled mask to ensure that the same nodule at different time points is spatially aligned; ROI cropping: cropping the three-dimensional region of interest containing the lung nodule from the CT image based on the labeled mask; unifying the CT image to a preset voxel size through resampling; truncating the CT values ​​to a preset window width range and performing normalization processing; randomly flipping, rotating, translating, performing Gamma correction, and injecting noise into the data in three-dimensional space to enhance data diversity; and obtaining a standardized lung nodule ROI image sequence after preprocessing.

4. The lung nodule growth prediction method based on the combination of LSTM and GAN according to claim 3, characterized in that: In step 3, the labeled dataset is divided into a training set, a validation set, and a test set; The 3D segmentation network for lung nodules includes: automated segmentation based on nnU-Net: automatically configuring 3D U-Net or 2D+3D hybrid architecture according to the size, resolution, and number of channels of the training data; calling nnU-Net scripts to perform patch segmentation, data augmentation, and multi-fold cross-validation; automatically selecting the optimal model weights based on the Dice score; segmenting the test set data through the inference script built into nnU-Net, and the model automatically performs post-processing to obtain the 3D segmentation results.

5. The lung nodule growth prediction method based on the combination of LSTM and GAN according to claim 4, characterized in that: The 3D segmentation network for lung nodules also includes: hierarchical attention segmentation based on Swin-UNet: the input 3D patch is divided into blocks, and multi-head self-attention is performed sequentially within local windows, stacking layer by layer to form a hierarchical structure; after each stage of the encoder is completed, it works with the U-Net decoder to output a high-resolution segmentation map; when directly using the 3D version of the Swin Transformer, the computational load is reduced through downsampling strategies or 2.5D variants; when using 3D Attention, it is used to capture the edges of low-density, morphologically blurred nodules; the patch sampling and batch size are set; after training, 3D inference is performed on the test set to obtain a voxel-level segmentation mask.

6. The lung nodule growth prediction method based on the combination of LSTM and GAN according to claim 5, characterized in that: In step 4, the hidden state vectors on the time series are extracted using a three-dimensional segmentation mask with multiple time nodes as the input LSTM mesh, capturing the dynamic evolution patterns of nodules at different stages, and outputting a temporal feature vector that encodes the dynamic evolution law of nodules.

7. The lung nodule growth prediction method based on the combination of LSTM and GAN according to claim 6, characterized in that: In step 5, the generator receives the temporal feature vector from the LSTM network and combines it with a fixed-length random noise vector. It uses a multi-scale convolutional structure or a hybrid structure of convolution and Transformer, and integrates the Attention mechanism to generate a predicted future lung nodule image. The discriminator performs binary classification on the real future lung nodule image and the future lung nodule image predicted by the generator, adaptively improving the generator's capabilities. The generated results are subjected to iterative adversarial training to continuously optimize the generator until the discriminator and the generator converge. A phased strategy was adopted during training, first training with reconstruction loss weights, and then adding adversarial loss weights.

8. The lung nodule growth prediction method based on the combination of LSTM and GAN according to claim 7, characterized in that: In step 6, the LSTM mesh outputs a temporal feature vector using the current and historical 3D segmentation masks as input; and the generator synthesizes the predicted future lung nodule image using the temporal feature vector as input.

Images