Radiation pneumonia prediction method and system based on deep learning

Abstract

The application provides a kind of based on deep learning's radiation pneumonia prediction method and system.The method comprises the following steps: S1, obtains DICOM image, radiotherapy data, clinical data and immunotherapy time series data, and carries out pre-processing;S2, constructs multi-channel three-dimensional structure tensor based on DICOM image;S3, obtains multi-channel input tensor X according to planned CT volume, dose volume and three-dimensional structure tensor;obtain time series characteristic vector and clinical phenotype characteristic vector according to radiotherapy data and clinical data;S4, construct fusion prediction model, and process three-dimensional structure tensor, time series characteristic vector and clinical phenotype characteristic vector, obtain the risk probability of radiation pneumonia;S5, based on the dynamic update of input and re-prediction of updated patient data.The application realizes the deep fusion and dynamic update of multi-modal information, can provide high-precision, risk prediction with spatially interpretable, and provides decision support for clinical intervention.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, and in particular relates to a method and system for predicting radiation pneumonia based on deep learning. Background Technology

[0002] Radiation pneumonitis (RP) is a common and potentially life-threatening complication of radiation therapy in lung cancer patients, making prevention more crucial than treatment. Statistics show that 10-20% of lung cancer patients undergoing radiation therapy develop moderate to severe radiation pneumonitis, with a mortality rate as high as 50% for severe cases. In recent years, immune checkpoint inhibitors combined with radiotherapy have become an important treatment strategy for locally advanced and metastatic lung cancer. The principle is that radiotherapy not only locally kills tumors but also triggers a "remote effect"—by altering the tumor microenvironment, it activates a systemic anti-tumor immune response; while immunosuppressants further release the brakes on the immune system, enhancing this effect. However, this synergistic effect is a double-edged sword: the activated immune system, while attacking the tumor, may also overreact to radiation-damaged normal lung tissue, leading to earlier, more severe, or even atypical radiation pneumonitis. It is precisely this alteration of the disease pattern by combined therapy that makes the incidence, timing, and clinical manifestations of radiation pneumonitis difficult to predict, thus challenging traditional risk prediction models based on radiotherapy alone. Currently, there is a lack of unified standards regarding the timing and dosage of combined immunotherapy and radiotherapy. Therefore, how to predict and promptly manage radiation pneumonitis under a comprehensive treatment model has become a serious clinical challenge. Against this backdrop, accurately identifying high-risk patients and implementing effective interventions is of paramount clinical importance.

[0003] Existing patent CN117745717A discloses a method and system for predicting radiation-induced pneumonia using dosimetry and deep learning features, including image data acquisition and preprocessing, region of interest (ROI) delineation, feature extraction (radiomics feature extraction), feature screening and dimensionality reduction, and construction and validation of machine learning models. While using radiomics features to classify and predict pneumonia patients induced by immunotherapy or radiotherapy yields a relatively low AUC, its effect on improving clinical decision-making is minimal.

[0004] In summary, the existing technology has the following drawbacks: (1) Feature extraction relies on manual delineation: The delineation (segmentation) of ROI is cumbersome and highly subjective, and there are differences between different delineators. This directly affects the stability and repeatability of subsequent feature extraction, and is one of the biggest sources of error.

[0005] (2) Limitations of “handmade” features: Image omics features are essentially “handmade”. Although they are numerous, they may not be able to capture the most predictive and complex deep abstract patterns in images, and their information extraction capabilities have an upper limit.

[0006] (3) Weak ability to process multimodal data fusion: Although existing technologies attempt to combine radiomics features with clinical data, they are usually simple stitches and lack an end-to-end framework to deeply and efficiently fuse spatially heterogeneous data such as dose distribution maps. Dose information is often simplified to a scalar parameter (such as MLD), losing its rich spatial distribution information.

[0007] (4) Insufficient feature mining of combined treatment modalities: Most radiomics models are trained based on traditional radiotherapy data, and their feature sets may not be able to fully characterize the complex biological response of unique immune pneumonia and radiation pneumonia caused by immunotherapy combined with radiotherapy. Summary of the Invention

[0008] This invention aims to overcome the limitations of traditional radiomics, which relies on manually delineating ROIs, resulting in poor feature repeatability, and the difficulty of capturing deep patterns in images by manually designed radiomics features. It provides a method and system for predicting radiation pneumonitis based on multimodal deep learning, so as to achieve accurate, non-invasive, and early prediction of RP in lung cancer patients receiving immunotherapy combined with radiotherapy.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: A deep learning-based method for predicting radiation-induced pneumonia includes the following steps: S1. Acquire DICOM images, radiotherapy data, clinical data, and immunotherapy time series data, and perform preprocessing; S2. Based on the planned CT volume obtained from DICOM images, a three-dimensional binary mask is automatically segmented using a three-dimensional segmentation network. Derived regions are obtained based on the three-dimensional binary mask. An isodose shell is generated based on the dose volume of radiotherapy data and a preset dose threshold, and a multi-channel three-dimensional structure tensor is constructed. S3. Obtain the multi-channel input tensor X based on the planned CT volume, dose volume, and three-dimensional structural tensor; obtain the temporal feature vector based on radiotherapy data and immunotherapy time series data; and obtain the clinical phenotype feature vector based on clinical data. S4. Construct a fusion prediction model and process the three-dimensional structural tensor, temporal feature vector and clinical phenotype feature vector to obtain the risk probability of radiation pneumonia; The fusion prediction model includes an image-dose dual-stream encoder for extracting image and dose fusion features, a graph neural network encoder for modeling inter-regional relationships, a conditional fusion module for fusing temporal and clinical information, and a classification head that outputs the risk probability of radiation pneumonitis. S5. Dynamically update the input and re-predict based on updated patient data; and / or, input the dose-volume data of different candidate radiotherapy into the fusion prediction model to assess and compare their corresponding radiation pneumonitis risks.

[0010] Furthermore, step S1 specifically includes: S101. Acquire DICOM images, radiotherapy data, clinical data, and immunotherapy time-series data; among which, DICOM images and radiotherapy data include: DICOM CT sequences, RTDOSE, RTSTRUCT, and RTPLAN; S102. Perform automatic desensitization processing on DICOM images and clinical data, and generate study IDs to associate them with patient data; S103. Using geometric transformation, spatial registration and resampling are performed between the dose volume obtained by RTDOSE analysis and the planned CT volume to obtain a geometrically consistent three-dimensional voxel mesh. S104. The HU value of the planned CT volume is clipped and standardized by z-score, and the radiotherapy dose volume is normalized according to the prescription dose.

[0011] Furthermore, step S2 specifically includes: S201. Automatic segmentation of the planned CT volume using a three-dimensional segmentation network, with a three-dimensional binary mask composed of anatomical structure mask and target area structure mask. S202. Perform set operations on the anatomical structure mask and the target area structure mask to obtain the derived region; S202. Generate an isodose shell based on the dose volume obtained from RTDOSE analysis and a preset dose threshold; S230. Channelization encoding is performed based on the three-dimensional binary mask, derived region, and dose shell to form a three-dimensional structural tensor. S203. The planned CT volume, dose volume and three-dimensional structural tensor are spliced ​​together in the channel dimension to form a voxel-level multi-channel input tensor X.

[0012] Furthermore, in step S3, the construction of the time-series feature vector is specifically as follows: taking the day of the first radiotherapy session as the time zero point, constructing a time event sequence of combined treatment based on the fractionation schedule in RTPLAN and the immunotherapy time-series data, and using a time position encoding method to convert the time event sequence into a fixed-length time-series feature vector.

[0013] Furthermore, the construction of the fusion prediction model in step S4 specifically includes: S401, Image-Dose Dual-Stream Coding: The planned CT volume and 3D structural tensor are encoded using 3D CNN or 3D Swin-Transformer to obtain multi-scale image feature maps; the normalized radiotherapy dose volume is encoded using 3D convolution or Transformer to obtain multi-scale dose feature maps; the multi-scale image feature maps and multi-scale dose feature maps are fused using a cross-attention mechanism to obtain multi-scale voxel features, which are then aggregated into a unified voxel backbone feature Z; S402, Graph Neural Network Encoding: Using the derived region and isodose shell region mask defined in step S2 as nodes, edges are defined based on spatial adjacency or dose correlation to construct a graph structure; node features are initialized using voxel backbone features Z and encoded through a graph neural network to obtain structural graph features; S403, Temporal-Clinical Condition Fusion: Fusion of temporal feature vectors and clinical phenotype feature vectors with voxel backbone features Z. Fusion methods include feature-level modulation mechanisms or attention-based fusion. S404, Classification Prediction: Input the fused voxel backbone features Z into the classification head, process it through global pooling and the Sigmoid function, and output the risk probability of radiation pneumonia.

[0014] Furthermore, step S404 also includes an uncertainty estimation step: estimating the uncertainty of the risk probability through Dropout or model ensemble, and outputting a confidence interval or uncertainty score.

[0015] Furthermore, step S4 also includes a model interpretability output step, specifically including: Based on the class activation map method, a spatial heat map is generated on the planned CT volume to visualize lung regions that contribute significantly to high-risk prediction. The influence of each clinical variable and time series feature on the prediction results was evaluated using the feature contribution analysis method.

[0016] Furthermore, the dynamic update in step S5 specifically involves: during the treatment process, periodically acquiring updated image and cumulative dose data, reconstructing the voxel-level multi-channel three-dimensional structural tensor and temporal feature vector, inputting the clinical phenotypic feature vector, the reconstructed multi-channel input tensor and temporal feature vector into the trained fusion prediction model to obtain the updated risk probability and risk change trend; and / or, making minor adjustments to the last few layers of the fusion prediction model to adapt to individual changes.

[0017] Furthermore, the planning linkage in step S5 specifically involves replacing the RTDOSE dose-volume corresponding to different candidate radiotherapy plans into the voxel-level multichannel input tensor during the radiotherapy planning stage. The risk probability corresponding to each candidate plan is calculated through a fusion prediction model and compared with the risk probability of the baseline plan. The risk change is then output to assist clinical decision-making.

[0018] A deep learning-based system for predicting radiation-induced pneumonia includes: The data acquisition and preprocessing module is used to acquire and preprocess multimodal medical data, and complete spatial registration and normalization. The automatic segmentation and tensor construction module is used to perform automatic segmentation, generate isodose shells, and construct voxel-level multichannel input tensors. The fusion prediction model module includes an image-dose dual-stream encoder, a graph neural network encoder, a time-to-clinical condition fusion module, and a classification head, which are used for feature fusion and risk prediction. An interpretable output module is used to provide spatial heatmaps, feature contribution and uncertainty information; The dynamic update and planning linkage module is used to support dynamic model updates, multi-plan risk assessment, and clinical decision support.

[0019] The beneficial effects of this invention are as follows: (1) Reduced dependence on sketching: Through automatic segmentation and human-machine collaboration, the subjective differences and workload caused by manual ROI are greatly reduced, and the stability and repeatability of features are improved.

[0020] (2) End-to-end multimodal deep fusion: The three-dimensional dose voxel is directly used as the channel input and fused with CT under multi-scale attention. The dose information is no longer simplified to a single scalar such as MLD / V20, and the spatial distribution is fully utilized.

[0021] (3) Combined therapy timing modeling: explicit time coding of the start and stop of immunotherapy and the dosing rhythm to adapt to the dynamic impact of immuno-radiotherapy interaction on RP risk, making up for the lack of characterization of combined therapy mode by traditional radiomics.

[0022] (4) Explainable and calibrable: Provides spatial thermal zone visualization and Tab feature contribution, supplemented by temperature scaling / depth integration to achieve probabilistic calibration, which is convenient for clinical use and threshold management.

[0023] (5) Dynamic prediction closed loop: Supports rolling updates of risks and record keeping as treatment progresses, forming a continuous learning closed loop that closely aligns with real clinical pathways.

[0024] (6) Deployability: The system is modular and the interface is standardized (DICOM / HL7 / FHIR), which can be quickly deployed on the institute's GPU server or compliant cloud, facilitating multi-center collaboration and generalization. Attached Figure Description

[0025] Figure 1 This is a flowchart of the radiation pneumonia prediction method provided by the present invention; Figure 2 This is a Swin-UNet network structure diagram of the radiation pneumonia prediction method provided by the present invention; Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following embodiments are for illustrative purposes only and not for limiting the invention. Those skilled in the art can make various modifications and substitutions to the following embodiments without departing from the spirit of the invention.

[0027] Please refer to Figure 1 and Figure 2 The deep learning-based method for predicting radiation pneumonia proposed in this invention includes steps such as data acquisition and preprocessing, structural tensor quantization, multimodal feature construction, fusion prediction model construction and training, inference and interpretable output, and dynamic update and planning linkage.

[0028] S1. Data Acquisition and Preprocessing S101, Data Acquisition (1) Obtain DICOM images and radiotherapy data related to radiotherapy for lung cancer patients based on the hospital's Picture Archiving and Communication System (PACS) and Radiotherapy Planning System (TPS), including: DICOM CT sequence: This refers to the three-dimensional planning CT volume used to develop treatment plans, serving as the basic input for the "image channel" in subsequent models; RTDOSE (Radiotherapy Dose Volume): A three-dimensional dose distribution stored in voxel grid form, used to construct "dose channels" and participate in isodose shell derivation and voxel-level multichannel tensor construction in subsequent steps; RTSTRUCT (Radiotherapy Structure Set): Contains structural outlines of the lungs, tumor target area (GTV / CTV / PTV) drawn by physicians, used to train or correct automatic segmentation networks, and as a reference in structural tensor quantization; RTPLAN (Radiation Therapy Plan): Includes information such as beam parameters, fractionation scheme, and prescribed dose, used to construct a radiotherapy fractionation schedule and cumulative dose information.

[0029] (2) Simultaneously, extract clinical data and immunotherapy time-series data from the electronic medical record system or tumor treatment information system: Clinical data: such as age, gender, smoking history, pulmonary function test (PFT), inflammatory markers (such as CRP, NLR), and history of lung disease (COPD, interstitial lung disease, etc.); Immunotherapy timing data: including the type of immunotherapy drug, administration time, pause time, and end time, used to characterize the dynamic effects of combination therapy.

[0030] S102, Registration and Resampling (1) To protect patient privacy, DICOM images and clinical data are automatically desensitized.

[0031] Read the fields from the DICOM image file header, including PatientName, PatientID, BirthDate, StudyInstanceUID, and SeriesInstanceUID, which describe the patient's identity and image sequence. Then delete or replace the corresponding tags with meaningless placeholders.

[0032] StudyInstanceUID or SeriesInstanceUID can be transformed using a one-way hash function with a fixed salt value. For example: read the original UID string; concatenate the preset salt value string with the original UID; input it into a hash function (such as SHA-256) to obtain a new UID; the transformed UID is used as a research UID to uniquely identify the image sequence without revealing the real identity.

[0033] (2) Desensitization of clinical data: Direct identification fields such as name, contact information, and ID number in electronic medical records are deleted, and only non-direct identification features related to modeling (such as age group, gender, laboratory indicators, etc.) are retained.

[0034] (3) Research ID mapping table generation: For each patient, the system generates a 128-bit random number based on a secure random number generator and forms a random UUID according to the UUIDv4 rule, which serves as the research ID (ResearchID); or the original patient ID is concatenated with the salt value and then hashed one way to obtain the ResearchID.

[0035] A mapping table of "Research ID – Original Identity Information" is established. This table is stored only in a restricted environment within the hospital for clinical retrospective analysis when necessary and is not shared externally with training data or models. The Research ID is used to associate the patient's CT volume, dose volume, structural mask, clinical data, and immunotherapy time series data throughout the modeling process.

[0036] S103, voxel grid, coordinates and registration By analyzing PixelSpacing (the actual physical spacing between rows and columns of the image), SliceThickness (the thickness parameter of adjacent reconstructed slices in the normal direction), ImagePositionPatient (IPP, the three-dimensional coordinates of the first pixel in the patient coordinate system), and ImageOrientationPatient (IOP, the direction cosines of the row and column directions in the patient coordinate system) in the DICOM CT sequence, the three-dimensional spatial position of each CT voxel in the patient coordinate system can be determined, thereby obtaining the CT voxel coordinates.

[0037] Similarly, the voxel size, IPP, IOP, and GridFrameOffsetVector (the offset of each dose slice relative to the reference plane) of the dose volume are read from RTDOSE to obtain the three-dimensional coordinates of each dose voxel in the patient coordinate system, i.e., the radiotherapy dose voxel coordinates.

[0038] This embodiment establishes an affine geometric transformation based on the aforementioned CT voxel coordinates and radiotherapy dose voxel coordinates. Through this geometric transformation, the dose volume obtained via RTDOSE is spatially registered and resampled with the planned CT volume to obtain a geometrically consistent three-dimensional voxel mesh. Specifically:

[0039] Using the patient coordinate system as the intermediate coordinate system, the radiotherapy dose voxel coordinates are mapped to the CT voxel grid based on information such as IPP and IOP. Trilinear interpolation was used to resample the dose volume in RTDOSE to the same voxel size and CT voxel grid as the planned CT volume.

[0040] In this step, a voxel refers to the smallest spatial unit in a 3D CT volume or 3D dose volume; the planned CT volume refers to the 3D image data reconstructed from a DICOM CT sequence; the CT voxel grid refers to a regular 3D raster defined by PixelSpacing and SliceThickness; and the dose volume refers to the 3D dose distribution stored in RTDOSE in the form of a voxel raster. The output of this step is a spatially aligned and resampled CT volume and dose volume, laying a geometrically consistent standardized foundation for the subsequent construction of a voxel-level multichannel input that fuses image, dose, and structural information.

[0041] S104, Intensity and Dose Normalization Planned CT volume intensity normalization: Using HU (Hunsfield Unit) as the measure of planned CT volume intensity, the HU value is first cropped to a preset range, such as −1000, 400-1000, 400−1000, 400 or other clinically reasonable ranges, to remove extreme values ​​caused by air and metal artifacts; then z-score normalization is performed on the cropped planned CT volume intensity.

[0042] Dose-volume normalization: Dose volumes aligned to the CT voxel grid can be linearly normalized to the actual physical dose (in Gy) or the prescribed dose. For example, the voxel dose can be divided by the prescribed dose for the case so that the normalized dose value falls within the [0,1] interval for cross-case comparisons.

[0043] The normalized planned CT volume and dose volume will be used as one of the input channels for subsequent deep learning models.

[0044] S2, Automatic Segmentation Model S201. In this embodiment, the planned CT volume after preprocessing in step S1 is subjected to automatic organ and target area segmentation to obtain a three-dimensional binary mask composed of anatomical structure mask and target area structure mask.

[0045] The automatic segmentation model can employ existing 3D segmentation networks, such as 3D U-Net and nnU-Net. The model's input is the planned CT volume after HU clipping and z-score normalization, and the output is a multi-channel 3D binary mask of the same size as the CT voxel mesh. The first type of channel is the anatomical structure mask (e.g., left lung, right lung, etc.), and the second type of channel is the radiotherapy-related target / dose-related structure mask (e.g., GTV, CTV, PTV, etc.). Each channel can be in binary form (0 / 1) or probabilistic form (0~1), and the binary mask can be obtained by thresholding.

[0046] When training this segmentation network, clinicians are used as labels in RTSTRUCT, and a hybrid loss function including Dice loss and voxel-level cross-entropy is employed for optimization. For severely imbalanced classes, weighted loss or Tversky loss can be introduced. This specification does not limit the specific network topology or loss function.

[0047] S202, Derivative Regions and Iso-dose Shell Generation (1) A derived region is generated by geometric operations of the anatomical structure mask and the target area structure mask. In this embodiment, a derived region is further constructed to characterize the spatial distribution of dose and the dose received by the lung parenchyma: the target area is removed from the lung parenchyma, and the left and right lung masks are merged to obtain the entire lung tissue region; the tumor target area mask is subtracted from the lung tissue mask to obtain the region of "lung tissue minus tumor volume", which is denoted as the Lung-GTV region, and is used to focus on the dose exposure of non-tumor lung tissue.

[0048] (2) Generation of the isodose shell (Vx region) mask: An isodose shell is generated based on the dose volume and preset dose thresholds. A set of dose thresholds {x1, x2, ...} (e.g., 5Gy, 10Gy, 20Gy, 30Gy, etc.) are preset on the dose volume D, which is consistent with the voxel grid of the planned CT. The dose volume D can be obtained from the three-dimensional dose distribution exported by the radiotherapy planning system, for example, by reading the RTDOSE file and mapping it to the same voxel grid as the planned CT through coordinate registration and resampling. For any threshold xi, the corresponding three-dimensional binary mask M_{Vx(i)} is generated by voxel-level comparison, where voxels that satisfy D≥xi are assigned a value of 1, and the remaining voxels are assigned a value of 0. To construct a "shell" morphology, a set difference operation can be performed on masks corresponding to adjacent thresholds to obtain annular regions, for example, M_{shell(i)} = M_{Vx(i)} \ M_{Vx(i+1)}; and a set operation can be performed with organ or target area masks to obtain derived regions, for example, Lung-GTV = M_{Lung} \ M_{GTV}, or the intersection of M_{shell(i)} and Lung-GTV can be used to confine the dose shell within the lung parenchyma. The above set operations can be implemented using difference, intersection, and necessary morphological processing (such as erosion / expansion), and this specification does not limit the specific implementation method. The generated isodose shell and derived regions are used to subsequently construct a structural tensor and structural relationship diagram to characterize the contribution of different dose distributions to the risk of radiation pneumonitis.

[0049] (3) Multichannel three-dimensional structure tensor The three-dimensional binary mask composed of the anatomical structure mask and the target area structure mask, the derived region, and the isodose shell are one-hot encoded according to the channel dimension to form a multi-channel three-dimensional structural tensor. This three-dimensional structural tensor is completely consistent with the planned CT volume in spatial dimensions, and the voxel value on each channel is 0 or 1, indicating whether the voxel belongs to the corresponding structure.

[0050] S3, Multimodal Feature Construction S301. Obtaining voxel-level multichannel input tensors The planned CT volume, dose volume, and 3D structure tensor are stitched together channel-wise along the voxel dimension to form a voxel-level multi-channel input tensor X. The size of the voxel-level multi-channel input tensor X is... Where C is the number of channels, and H, W, and D are the dimensions of the CT voxel grid in the three directions, respectively.

[0051] S302. Obtain the temporal feature vector. Based on the radiotherapy fractionation schedule and immunotherapy timing data recorded in RTPLAN, a time event sequence for combined therapy was constructed. Taking the day of the first radiotherapy fraction as the zero point, each radiotherapy fraction, each immunotherapy administration or pause event was regarded as a time-stamped event, and its relative time (such as the number of days since the first radiotherapy), current cumulative dose, cumulative number of administrations, and other information were recorded. All time-stamped times were used to construct a time event sequence.

[0052] The temporal event sequence of the combined treatment is converted into a fixed-length temporal feature vector T using temporal position encoding (e.g., relative position encoding based on sine / cosine functions) or other temporal embedding methods. This temporal feature vector T will be used as global conditional information in subsequent models to characterize the dynamic impact of the combined immunotherapy and radiotherapy regimen on the risk of radiation pneumonitis.

[0053] S303. Obtaining Clinical Phenotypic Feature Vectors Variables associated with the risk of radiation pneumonitis were selected from clinical data to obtain a clinical phenotypic feature vector, including continuous numerical variables and categorical variables. Continuous numerical variables include, for example, age, BMI, baseline lung function indicators (FEV1, FVC, DLCO, etc.), and inflammation-related laboratory indicators (CRP, NLR, albumin, etc.). Categorical variables include, for example, gender, smoking status (never / past / current), tumor stage, history of underlying diseases (present / absent), and types of concomitant medications.

[0054] For continuous numerical variables, missing values ​​can be imputed based on the statistical results of the training set (e.g., using the median or mean), and z-score standardization can be performed. For categorical variables, they can be converted into numerical representations using one-hot encoding or ordinal encoding. If necessary, a binary feature called a "missing value flag" can be added to each variable to indicate whether the variable is missing.

[0055] S4. Construct a fusion prediction model and process the data to obtain the prediction probability value.

[0056] S401, Image-Dose Dual-Stream Encoder (Voxel Backbone) (1) Input the planned CT volume and the three-dimensional structural tensor into a 3D CNN or 3D Swin-Transformer for processing to obtain multi-scale image feature maps, as follows: The planned CT volume and its three-dimensional structural tensor are input into a 3D CNN, and three-dimensional convolution, normalization, nonlinear activation and downsampling are performed sequentially to form a multi-scale image feature map, which can be denoted as F_img(s), where s represents the network level (e.g., 1 / 2, 1 / 4, 1 / 8, 1 / 16 downsampling scale). The planned CT volume and its 3D structural tensor are input into a 3D Swin-Transformer and divided into fixed-size 3D blocks (e.g., 4×4×4 voxels). Each block is linearly mapped to obtain an initial embedding, which is then encoded using a windowed multi-head self-attention and shifting window mechanism. This path also outputs a multi-scale image feature map F_img(s).

[0057] (2) The normalized dose volume is used as input, and the feature is encoded using a three-dimensional convolution or Transformer structure to output a multi-scale dose feature map F_dose(s).

[0058] (3) Cross-attention fusion: At each scale s, the multi-scale image feature map F_img(s) is used as the query, and the multi-scale dose feature map F_dose(s) is used as the key and value. The multi-scale voxel features are fused through the multi-head cross-attention module to obtain the multi-scale voxel features denoted as Z(s). Then, the multi-scale voxel features Z(s) are aggregated into a unified voxel backbone feature Z by using a top-down or bottom-up feature pyramid aggregation method.

[0059] In this specification, the voxel backbone feature Z is the high-dimensional feature representation of the image-dose dual-stream encoder after fusing image and dose information, and it forms the basis for subsequent structural diagram construction and temporal-clinical fusion.

[0060] S402, Graph Neural Network Encoding: Using the derived regions and isodose shells defined in step S2 as nodes, edges are defined based on spatial adjacency or dose correlation to construct a graph structure; node features are initialized using voxel backbone features Z, and encoded through a graph neural network to obtain the structural graph features, as follows: (1) Node construction: For each derived region code node, the voxel backbone feature Z is used to perform statistical aggregation in the region (e.g., to calculate the average pooling or max pooling of the features in the region) to obtain the initial feature vector of the node; at the same time, traditional dosimetric indicators such as the volume (number of voxels) and average dose of the derived region can be added as supplementary features to obtain the node.

[0061] (2) Edge relationship definition: When two derived regions are directly adjacent in space or share a boundary, an edge is established in the figure to represent the anatomical adjacency relationship; an edge can also be established based on the correlation of dose or features (e.g., the average dose correlation coefficient between two regions exceeds a threshold) to reflect dose-function coupling.

[0062] (3) Graph Neural Network Encoding: Constructing a graph based on nodes and edges We employ existing graph neural network (GNN) structures (such as graph convolutional networks and graph attention networks) to update node features, thereby obtaining the interaction representations between regions. The output of the GNN can be denoted as the structural graph feature H_G, which is used to supplement the voxel backbone feature Z in characterizing the relationships between regions.

[0063] S403, Timing-Clinical Condition Fusion The temporal feature vector T and clinical phenotype feature vector C obtained in step S3 are fused with the voxel backbone feature Z. The specific method for fusing the temporal feature vector T, clinical phenotype feature vector C, and voxel backbone feature Z is as follows:

[0064] (1) Feature-level modulation mechanism: First, the temporal feature vector T and the clinical phenotype feature vector C are concatenated along the feature dimension, and then input into a fully connected neural network to obtain the channel scale coefficient and bias (similar to the FiLM structure) used to adjust the voxel backbone feature Z. That is, each channel of the voxel backbone feature Z is subjected to the following modulation mechanism: The affine transformation allows different time sequences and clinical states to alter the model's focus on different channel features.

[0065] (2) An attention-based fusion approach can also be adopted, in order to As a query, Z serves as both a key and a value, selectively enhancing features associated with high-risk patterns across voxel or region dimensions through an attention mechanism.

[0066] S404, Classification Header and Uncertainty Estimation Header The fused voxel backbone features Z are input into the classification head and subjected to global pooling (averaging or maximizing across the spatial dimension) to obtain a global feature vector. This global feature vector is then input into a sigmoid function to obtain a probability value r between 0 and 1, representing the risk probability of a patient developing radiation pneumonitis (e.g., RP ≥ 2) within a predetermined observation time window. The output of the classification head can be formally represented as:

[0067] in, The aforementioned fused high-dimensional feature vector, Forward mapping of the classification head, This represents the Sigmoid function. The probability r can be further categorized into risk levels such as "low risk," "medium risk," and "high risk" based on thresholds determined in the validation set.

[0068] In this embodiment, the prediction model is trained using data from lung cancer patients who have completed radiotherapy and have follow-up information. For each patient, a training sample is constructed, which includes: preprocessed planned CT volume, dose volume, and structural tensor, forming a voxel-level multichannel input tensor X; the corresponding temporal feature vector T; the clinical phenotypic feature vector C; and the radiation pneumonitis label y (e.g., whether RP ≥ 2 occurred within a given time window).

[0069] The fusion prediction model is trained using training data, with the optimization objective being the binary cross-entropy loss between the classifier head output probability r and the true label y. Focal Loss and other techniques are employed to address class imbalance. During training, conventional data augmentation techniques can be used to moderately perturb the training samples, such as applying slight affine transformations or intensity perturbations to the CT volume and dose volume, to enhance the model's robustness.

[0070] The trained model is evaluated using performance metrics, including AUROC, accuracy, sensitivity, specificity, and Brier score.

[0071] In this embodiment, to improve the interpretability of the model output, the following information may also be output: Spatial heatmap interpretation: Based on the voxel backbone feature Z and classification output r, a heatmap is generated on the planned CT scan using methods such as class activation maps, highlighting lung regions that contribute significantly to high-risk prediction. Physicians can observe the heatmap to understand the spatial distribution of what the model considers high-risk areas.

[0072] Contribution of clinical and temporal features: By using feature perturbation or contribution analysis methods (such as gradient-based sensitivity analysis), the impact of each clinical variable and temporal feature on the final prediction result is evaluated, and the features with the highest contribution and their positive or negative effects on risk are output, which helps to explain the decision-making basis of the model.

[0073] Uncertainty alerts (e.g., with the uncertainty estimation module enabled): If Dropout or model ensemble methods are used to estimate prediction uncertainty, a confidence interval for the risk probability or an uncertainty score can be provided for each case. When the uncertainty is high, it can alert physicians to interpret the prediction results with caution.

[0074] S5, Dynamic Update of Fusion Prediction Model In clinical practice, patients may undergo new imaging examinations (such as CBCT) during radiotherapy, the cumulative dose may increase with each fraction, and the immunotherapy regimen may also be adjusted. This embodiment supports dynamic updates to the prediction model and plan linkage analysis.

[0075] Dynamic updates (online or nearline inference): During treatment, new incremental images (e.g., registered CBCT or synthetic images) and updated cumulative dose volumes are periodically acquired, and the voxel-level multichannel input tensor is reconstructed accordingly. and the latest time series feature vector .

[0076] Will By inputting the pre-trained fusion prediction model, the updated risk probability r_t can be obtained. By comparing it with the risk probability r_{t-1} at the previous time point, the trend of risk change can be obtained.

[0077] When necessary, the last few layers of the model can be lightly tweaked without changing the main network structure, so that the model can better adapt to changes in individual patients at different times. This tweaking can only be done in a controlled environment within the hospital.

[0078] Planning Linkage and Sensitivity Analysis: In the radiotherapy planning stage, this embodiment can receive one or more candidate radiotherapy plans, each corresponding to a different RTDOSE. By replacing the dose volume of the candidate plan in the dose channel position of the multi-channel input tensor X, while keeping other channels and features unchanged, and inputting it into the fusion prediction model for inference, the corresponding risk probability r(k) can be obtained.

[0079] The risk change Δr(k) is obtained by comparing r(k) with the risk probability r(base) of the current or baseline plan. The model can output the risk probability, risk change, and changes in key dosimetric indicators for each candidate plan, assisting physicians in selecting the option with a lower risk of radiation pneumonitis while ensuring tumor control among multiple plans.

[0080] After accumulating a certain number of new cases, the model can be retrained or incrementally learned under compliance with regulations, and the decision threshold can be recalibrated based on the new data, so that the model can maintain good predictive performance and clinical usability in different centers and at different time periods.

[0081] A deep learning-based system for predicting radiation-induced pneumonia includes: The data acquisition and preprocessing module is used to acquire and preprocess multimodal medical data, and complete spatial registration and normalization. The automatic segmentation and tensor construction module is used to perform automatic segmentation, generate isodose shells, and construct voxel-level multichannel input tensors. The fusion prediction model module includes an image-dose dual-stream encoder, a graph neural network encoder, a time-to-clinical condition fusion module, and a classification head, which are used for feature fusion and risk prediction. An interpretable output module is used to provide spatial heatmaps, feature contribution and uncertainty information; The dynamic update and planning linkage module is used to support dynamic model updates, multi-plan risk assessment, and clinical decision support.

[0082] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A deep learning-based method for predicting radiation-induced pneumonia, characterized in that, Includes the following steps: S1. Acquire DICOM images, radiotherapy data, clinical data, and immunotherapy time series data, and perform preprocessing; S2. Based on the planned CT volume obtained from DICOM images, a three-dimensional binary mask is automatically segmented using a three-dimensional segmentation network. Derived regions are obtained based on the three-dimensional binary mask. An isodose shell is generated based on the dose volume of radiotherapy data and a preset dose threshold, and a multi-channel three-dimensional structure tensor is constructed. S3. Obtain the multi-channel input tensor X based on the planned CT volume, dose volume, and three-dimensional structural tensor; obtain the temporal feature vector based on radiotherapy data and immunotherapy time series data; and obtain the clinical phenotype feature vector based on clinical data. S4. Construct a fusion prediction model and process the three-dimensional structural tensor, temporal feature vector and clinical phenotype feature vector to obtain the risk probability of radiation pneumonia; The fusion prediction model includes an image-dose dual-stream encoder for extracting image and dose fusion features, a graph neural network encoder for modeling inter-regional relationships, a conditional fusion module for fusing temporal and clinical information, and a classification head that outputs the risk probability of radiation pneumonitis. S5. Dynamically update the input and re-predict based on updated patient data; and / or, input the dose-volume data of different candidate radiotherapy into the fusion prediction model to assess and compare their corresponding radiation pneumonitis risks.

2. The deep learning-based radiation pneumonia prediction method according to claim 1, characterized in that, Step S1 specifically includes: S101. Acquire DICOM images, radiotherapy data, clinical data, and immunotherapy time-series data; among which, DICOM images and radiotherapy data include: DICOM CT sequences, RTDOSE, RTSTRUCT, and RTPLAN; S102. Perform automatic desensitization processing on DICOM images and clinical data, and generate study IDs to associate them with patient data; S103. Using geometric transformation, spatial registration and resampling are performed between the dose volume obtained by RTDOSE analysis and the planned CT volume to obtain a geometrically consistent three-dimensional voxel mesh. S104. The HU value of the planned CT volume is clipped and standardized by z-score, and the radiotherapy dose volume is normalized according to the prescription dose.

3. The deep learning-based method for predicting radiation-induced pneumonia according to claim 2, characterized in that, Step S2 specifically includes: S201. Automatic segmentation of the planned CT volume using a three-dimensional segmentation network, with a three-dimensional binary mask composed of anatomical structure mask and target area structure mask. S202. Perform set operations on the anatomical structure mask and the target area structure mask to obtain the derived region; S202. Generate an isodose shell based on the dose volume obtained from RTDOSE analysis and a preset dose threshold; S230. Channelization encoding is performed based on the three-dimensional binary mask, derived region, and dose shell to form a three-dimensional structural tensor. S203. The planned CT volume, dose volume and three-dimensional structural tensor are spliced ​​together in the channel dimension to form a voxel-level multi-channel input tensor X.

4. The deep learning-based method for predicting radiation-induced pneumonia according to claim 1, characterized in that, In step S3, the construction of the time-series feature vector is specifically as follows: taking the day of the first radiotherapy session as the time zero point, constructing a time event sequence of combined treatment based on the fractionation schedule in RTPLAN and the immunotherapy time sequence data, and using the time position encoding method to convert the time event sequence into a fixed-length time-series feature vector.

5. The deep learning-based method for predicting radiation-induced pneumonia according to claim 1, characterized in that, The construction of the fusion prediction model in step S4 specifically includes: S401, Image-Dose Dual-Stream Coding: The planned CT volume and 3D structural tensor are encoded using 3D CNN or 3D Swin-Transformer to obtain multi-scale image feature maps; the normalized radiotherapy dose volume is encoded using 3D convolution or Transformer to obtain multi-scale dose feature maps; the multi-scale image feature maps and multi-scale dose feature maps are fused using a cross-attention mechanism to obtain multi-scale voxel features, which are then aggregated into a unified voxel backbone feature Z; S402, Graph Neural Network Encoding: Using the derived region and isodose shell region mask defined in step S2 as nodes, edges are defined based on spatial adjacency or dose correlation to construct a graph structure; node features are initialized using voxel backbone features Z and encoded through a graph neural network to obtain structural graph features; S403, Temporal-Clinical Condition Fusion: Fusion of temporal feature vectors and clinical phenotype feature vectors with voxel backbone features Z. Fusion methods include feature-level modulation mechanisms or attention-based fusion. S404, Classification Prediction: Input the fused voxel backbone features Z into the classification head, process it through global pooling and the Sigmoid function, and output the risk probability of radiation pneumonia.

6. The deep learning-based method for predicting radiation-induced pneumonia according to claim 5, characterized in that, Step S404 also includes an uncertainty estimation step: by using Dropout or model ensemble, the uncertainty of the risk probability is estimated and the confidence interval or uncertainty score is output.

7. The deep learning-based method for predicting radiation-induced pneumonia according to claim 1, characterized in that, Step S4 also includes a model interpretability output step, specifically including: Based on the class activation map method, a spatial heat map is generated on the planned CT volume to visualize lung regions that contribute significantly to high-risk prediction. The influence of each clinical variable and time series feature on the prediction results was evaluated using the feature contribution analysis method.

8. The deep learning-based method for predicting radiation-induced pneumonia according to claim 1, characterized in that, The dynamic update in step S5 specifically involves: during the treatment process, periodically acquiring updated image and cumulative dose data, reconstructing the voxel-level multi-channel three-dimensional structural tensor and temporal feature vector, inputting the clinical phenotypic feature vector, the reconstructed multi-channel input tensor and temporal feature vector into the trained fusion prediction model to obtain the updated risk probability and risk change trend; and / or, making minor adjustments to the last few layers of the fusion prediction model to adapt to individual changes.

9. The deep learning-based method for predicting radiation-induced pneumonia according to claim 1, characterized in that, In step S5, the planning linkage specifically involves replacing the RTDOSE dose-volume corresponding to different candidate radiotherapy plans into the voxel-level multichannel input tensor during the radiotherapy planning stage. The risk probability corresponding to each candidate plan is calculated through a fusion prediction model and compared with the risk probability of the baseline plan. The risk change is then output to assist clinical decision-making.

10. A deep learning-based system for predicting radiation-induced pneumonia, characterized in that, include: The data acquisition and preprocessing module is used to acquire and preprocess multimodal medical data, and complete spatial registration and normalization. The automatic segmentation and tensor construction module is used to perform automatic segmentation, generate isodose shells, and construct voxel-level multichannel input tensors. The fusion prediction model module includes an image-dose dual-stream encoder, a graph neural network encoder, a time-to-clinical condition fusion module, and a classification head, which are used for feature fusion and risk prediction. An interpretable output module is used to provide spatial heatmaps, feature contribution and uncertainty information; The dynamic update and planning linkage module is used to support dynamic model updates, multi-plan risk assessment, and clinical decision support.

Images