A method and system for multi-modal prognosis analysis of lymphoma

Abstract

The application discloses a lymphoma multi-modal prognosis analysis method and system, the method obtains image, gene and pathological modal data of a target patient; the three types of modal data are respectively input into an image encoder, a gene encoder and a pathological encoder, potential representations of each modal are obtained, and the potential representations are fused according to modal weights to obtain a unified potential representation of the target patient; the unified potential representation is appended to the end of a historical patient potential representation base; a similarity fusion matrix between patients is calculated based on the expanded patient node feature set, an adjacency matrix is generated according to a graph threshold value; the unified potential representation of each patient is taken as a node feature, and the adjacency matrix is taken as a graph structure, graph neural network prognosis reasoning is performed, and a PFS24 prognosis probability or a PFS12 risk grouping result is output. Through technical means such as image standardization, spatial consistency verification, multi-modal field alignment and numerical coding, quality gating and the like, end-to-end traceable analysis from an image to a prognosis is realized.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing, deep learning lesion segmentation, multimodal medical data fusion, and graph neural network prognostic analysis, specifically to a method and system for multimodal prognostic analysis of lymphoma. Background Technology

[0002] PET / CT scans simultaneously reflect the metabolic activity and anatomical location of lymphoma lesions, serving as crucial evidence for lymphoma staging, treatment evaluation, and prognosis. Clinically, tumor burden is described using indicators such as SUVmax (maximum standardized uptake value), MTV (metabolic tumor volume), TMTV (total metabolic tumor volume), and TLG (total glycolysis). These indicators depend on the accurate determination of the lesion region. Manual and semi-automatic delineation are time-consuming and affected by factors such as the reader's experience, blurred lesion boundaries, normal high-uptake tissue, and differences in acquisition methods using different equipment.

[0003] Self-configurable 3D segmentation models such as nn-UNet have been used for automatic lesion segmentation in medical images, but they still have the following shortcomings when directly used as benchmark models for PET / CT lymphoma segmentation in clinical services.

[0004] First, the baseline model itself does not constrain the actual DICOM (Digital Imaging and Communication Standard for Medicine) screening, SUV conversion, and CT resampling process, which can easily lead to inconsistencies between PET, SUV, CTres (resampled CT), and mask space. Raw PET images record the distribution of radioactivity, which must be converted to SUV based on information such as injection dose, injection time, acquisition time, and patient weight before comparisons can be made between different patients. CT images typically have different spatial resolutions and voxel sizes than PET images; directly using them as segmentation input can cause the segmentation network to fail to correctly align the spatial information of the two modalities. If the aforementioned conversion and resampling are performed by different operators at different times, it is highly likely to produce deviations in the image spatial coordinates.

[0005] Second, baseline model outputs typically focus only on segmentation masks, lacking standardized nomenclature, spatial verification, and result traceability that link MTV / TMTV, SUV statistics, and radiomics features. The binary mask output by segmentation models only indicates "whether it is a lesion" at the pixel level, but clinicians need metabolic volume, tumor burden, and omics features calculated based on this mask in physical units. Currently, there is a lack of a mechanism to automatically chain mask generation, volume calculation, and omics extraction within the same spatial reference frame.

[0006] Third, the baseline model lacks sufficient online quality gating for small lesions, high physiological uptake backgrounds, and abnormal sequence quality, making it easy for low-reliability segmentation results to be passed on to subsequent prognostic models. In clinical PET / CT images, physiologically high-uptake areas such as reactive lymph node hyperplasia, myocardial uptake, and urinary system excretion may be visually difficult to distinguish from lesions, potentially leading to false positives in the segmentation model. Without spatial consistency checks and product integrity checks, these low-quality segmentation results will directly enter subsequent prognostic calculations, resulting in unreliable final conclusions.

[0007] On the other hand, lymphoma prognosis is not only related to PET / CT imaging features, but also to genetic and pathological features. Traditional prognostic models often employ simple feature concatenation and unimodal modeling, making it difficult to handle the differences in dimensionality, scale, sparsity, and sample distribution among radiomics, genetic, and pathological data. Radiomics features are typically continuous numerical values ​​ranging from hundreds to thousands of dimensions; genetic data includes mixed types such as mutation states and expression levels, and contains a large number of undetected sites; pathological data includes various forms such as semi-quantitative immunohistochemical indicators, pathology report text, and pathological images. Directly concatenating these heterogeneous features into a single long vector input to a classifier makes it difficult for the model to effectively learn complementary information across modalities.

[0008] Graph neural networks (GNNs) and graph convolutional networks (GCNs) can leverage the similarity relationships between patients for graph propagation. However, benchmark GNNs typically assume a complete patient graph already exists. In online single-patient prediction scenarios, a single new patient cannot form an effective graph structure on its own; it must be placed into a historical patient similarity network, ensuring that node features, adjacency matrices, and PFS (progression-free survival) time-series configurations are consistent with those used in the training phase. Specifically, GNN training relies on a pre-constructed graph structure where nodes represent patients and edges represent the similarity relationships between patients. During training, the graph structure is built based on all patients in the training set. However, in the online inference phase, a new patient, as an isolated node, cannot be directly input into the trained GNN model because the model's graph structure is fixed, and the new node lacks connections to other nodes in the graph.

[0009] Therefore, an algorithm is needed to impose end-to-end constraints on the PET / CT lymphoma lesion segmentation model and the lymphoma multimodal prognostic prediction model, so that the segmentation results, metabolic indicators, omics features, multimodal deep characterization, patient similarity map and prognostic output can be continuously generated, verified and traced under the same patient identifier. Summary of the Invention

[0010] The purpose of this invention is to provide a method and system for multimodal prognostic analysis of lymphoma, in order to solve the technical problem that when a multimodal prognostic prediction model for lymphoma based on GNN fails to form an effective graph structure during online inference for a single patient, the lack of graph connections between new patients and historical patients leads to the failure of the GNN propagation mechanism.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] A multimodal prognostic analysis method for lymphoma includes the following steps:

[0013] Acquire imaging modality data, genetic modality data, and pathological modality data of the target patient, and generate a unified patient sample identifier;

[0014] The image modality data, gene modality data, and pathology modality data are respectively input into the image encoder, gene encoder, and pathology encoder to obtain the latent representation of each modality. The latent representations of each modality are fused according to modality weights to form a unified latent representation of the target patient, which serves as the node feature of the graph neural network prognostic model.

[0015] The unified potential representation of the target patient is added to the potential representation disk of historical patients. The potential representation disk of historical patients is composed of the unified potential representations of each historical patient in the training set. The unified potential representations of each historical patient and the unified potential representation of the target patient are in the same feature space.

[0016] Based on the patient node feature set with the added unified potential representation of the target patient, a patient similarity fusion matrix is ​​calculated, and the patient similarity fusion matrix is ​​filtered according to a graph threshold to generate the adjacency matrix required by the graph neural network.

[0017] Using the unified latent representation of each patient in the patient node feature set as node features, and the adjacency matrix as the graph structure, a graph neural network prognostic inference is performed to output the prognostic result of the target patient.

[0018] The image encoder, gene encoder, pathology encoder, modality weights, and graph neural network prognostic model are all pre-trained and solidified using a training set; the historical patient latent representation disk is solidified after training and used to merge and construct a unified latent representation of new target patients during the online inference phase.

[0019] In some embodiments, the step of acquiring the image modality data of the target patient includes:

[0020] Receive PET / CT DICOM data from the target patient and screen for effective PET and CT sequences;

[0021] The PET sequence is converted into a standard PET image, and the standard PET image is converted into an SUV standard uptake value image based on the injection dose, acquisition time, and patient weight information.

[0022] CT images are resampled into the PET image space to obtain resampled CT images;

[0023] The PET / CT lymphoma lesion segmentation model based on nn-UNet is invoked, and the standard PET image and the resampled CT image are used as dual-channel inputs to perform sliding window depth segmentation inference and output standard lesion mask results.

[0024] Spatial consistency verification is performed between the standard lesion mask and the standard SUV image.

[0025] Three-dimensional connected component analysis was performed on the validated standard lesion mask to obtain the connected components of each lesion. Based on the number of voxels in each lesion's connected component and the physical voxel spacing of the SUV standard uptake value image, the lesion-level metabolic tumor volume was calculated. The total metabolic tumor volume at the patient level was obtained by summing the volumes of each lesion-level metabolic tumor. The SUV radiomics features were extracted using the SUV standard uptake value image and the standard lesion mask as input to obtain image modality data.

[0026] In some embodiments, after summarizing the metabolic tumor volumes of each lesion level to obtain the total metabolic tumor volume at the patient level, at least one of the following patient-level tumor burden indicators is also calculated: total glycolysis at the patient level, maximum SUV, average SUV, peak SUV, maximum lesion volume, and maximum distance between lesions.

[0027] The SUV radiomics features are extracted by using the SUV standard uptake value image and the standard lesion mask as input after obtaining the patient-level total metabolic tumor volume and the patient-level tumor burden index.

[0028] In some embodiments, the lesion-level metabolic tumor volume is calculated as follows:

[0029] MTV i = N i × Δx × Δy × Δz

[0030] Among them, MTV i Let N be the metabolic tumor volume of the i-th lesion. i Let be the number of voxels in the connected region of the i-th lesion, and Δx, Δy, and Δz be the physical voxel spacing of the SUV standard uptake value image in the three spatial directions, respectively.

[0031] The total metabolic tumor volume at the patient level is the MTV of all connected lesions. i sum;

[0032] The total glycolysis at the lesion level is calculated as follows:

[0033] TLG i = MTV i × SUVmean i

[0034] Among them, TLG i SUVmean represents the total glycolysis amount of the i-th lesion. i The average SUV standard uptake value of all voxels within the connected domain of the i-th lesion;

[0035] The total glycolysis volume at the patient level is the total TLG of all lesion connected regions. i sum.

[0036] In some embodiments, before inputting the image modality data, gene modality data, and pathology modality data into the image encoder, gene encoder, and pathology encoder, respectively, the method further includes a step of field alignment and numerical standardization of each modality data:

[0037] Image modal data are concatenated into image feature vectors according to a preset image field order. The image fields include patient-level total metabolic tumor volume, patient-level total glycolysis, maximum SUV value, average SUV value, peak SUV value, maximum distance between lesions, maximum lesion volume, and SUV radiomics characteristics. Missing fields are filled with zero values ​​and missing markers are recorded.

[0038] Gene modality data are mapped according to the preset order of lymphoma-related gene fields. The mutation status, expression level, copy number variation and detection positive and negative of each gene field are numerically encoded. Undetected and missing fields are masked and filled with the numerical values ​​agreed upon by the model. Gene feature vectors are output.

[0039] The pathological modality data is mapped according to the preset pathological field order, and the positive and negative values ​​and scoring levels of the immunohistochemical indicators of each pathological field are converted into numerical values. The pathological feature vector is output by combining the pathological image encoding results and the pathological text encoding results.

[0040] The image feature vector, gene feature vector, and pathological feature vector are standardized using the same normalized statistics as in the training phase, so that the feature distribution of each encoder input is consistent with that in the training phase.

[0041] In some embodiments, the image encoder, gene encoder, and pathology encoder are multilayer fully connected neural networks with the same or different structures, each nonlinearly mapping the input features of its corresponding modality to a potential representation space of the same dimension; the modality weights are trainable scalar parameters, which, after Softmax normalization, satisfy that each modality weight is non-negative and the sum is 1; the image encoder, gene encoder, pathology encoder, and modality weights are jointly optimized during the training phase with prognostic classification supervision loss as the training objective, and are fixed after training.

[0042] In some embodiments, the historical patient potential representation base is constructed in the training phase as follows: the image modality data, gene modality data and pathological modality data of each historical patient in the training set are respectively input into the fixed image encoder, gene encoder and pathological encoder, and fused into a unified potential representation of each historical patient according to the fixed modality weights. The unified potential representation of all historical patients forms the historical patient potential representation base.

[0043] During the online inference phase, the unified latent representation of the target patient is appended to the end of the latent representation base disk of the historical patients according to the sample index, forming an expanded patient node feature set. The historical patient portion of the expanded patient node feature set retains its original index and feature value, while the target patient participates in subsequent graph construction and inference as a new node.

[0044] In some embodiments, the calculation of the inter-patient similarity fusion matrix employs a similarity network fusion algorithm. The similarity network fusion algorithm calculates a multi-kernel similarity matrix based on the similarity between the features of each node in the patient node feature set and performs iterative fusion to obtain the fused inter-patient similarity fusion matrix.

[0045] The step of filtering the patient similarity fusion matrix according to the graph threshold to generate the adjacency matrix required by the graph neural network includes: retaining edges in the patient similarity fusion matrix with similarity values ​​greater than or equal to the graph threshold, and setting edges with similarity values ​​less than the graph threshold to zero, to obtain a weighted adjacency matrix; or, setting edges in the patient similarity fusion matrix with similarity values ​​greater than or equal to the graph threshold to 1, and setting edges with similarity values ​​less than the graph threshold to 0, to obtain a binary adjacency matrix;

[0046] The adjacency matrix, after row normalization, serves as the normalized adjacency matrix required for graph neural network propagation.

[0047] In some embodiments, the prognostic results include the PFS24 prognostic probability; the imaging modal data, genetic modal data and pathological modal data of the target patient are also used for PFS12 prognostic inference after being encoded and fused, and the PFS12 risk grouping results are output.

[0048] The PFS24 prognostic inference and the PFS12 prognostic inference respectively employ time-specific graph neural network model parameters, graph thresholds, inactivation ratios, and risk truncation thresholds; the graph neural network model parameters, graph thresholds, inactivation ratios, and risk truncation thresholds are independently optimized and fixed for the two time periods of PFS24 and PFS12 through a validation set during the training phase;

[0049] The PFS24 prognostic probability is a continuous probability value; the PFS12 risk grouping results divide patients into three groups: low-risk, intermediate-risk, and high-risk, based on the low-risk cutoff threshold and the high-risk cutoff threshold.

[0050] In some embodiments, after performing graph neural network prognostic inference and outputting prognostic results, a quality gating determination step is further included:

[0051] Obtain the set of essential products required under the current prognostic analysis mode. The set of essential products includes: the standard lesion mask, lesion-level metabolic tumor volume, patient-level total metabolic tumor volume, SUV radiomics features, the imaging feature vector, gene feature vector and pathological feature vector, the patient-to-patient similarity fusion matrix, and the prognostic results.

[0052] Check whether all the required products have been generated and read successfully;

[0053] If all products are generated and read successfully, the task status is determined to be successful and the prognosis result is returned; if any required product is missing or reading fails, the task status is determined to be failed and the missing product identifier and failure stage information are returned to prevent incomplete prognosis results from being output.

[0054] In some embodiments, when screening PET and CT sequences in DICOM, if SEG sequences exist in the DICOM data, the SEG sequences are identified and saved. The SEG sequences are used for expert annotation reading during the training phase, comparison of historical segmentation results, or segmentation quality verification, but are not used as necessary input for online automatic segmentation.

[0055] In some embodiments, the PET / CT lymphoma lesion segmentation model uses the nn-UNet three-dimensional full-resolution segmentation process as the baseline model, adopts a three-dimensional multi-channel input structure, uses the standard PET image and the resampled CT image as dual-channel input, and uses the lesion mask reviewed by experts as the supervision label. The segmentation loss function is composed of Dice loss, cross-entropy loss, small lesion constraint term and boundary constraint term weighted. In the training phase, random pruning, scale perturbation, intensity perturbation and flip enhancement are used. In the validation phase, the optimal checkpoint is selected as the online inference model based on the lesion Dice coefficient, lesion recall rate, number of false positive lesions and metabolic tumor volume error.

[0056] In some embodiments, the SUV radiomics features are extracted according to the IBSI Image Biomarker Standardization Initiative standard. The types of the SUV radiomics features include multiple types of first-order statistical features, shape features, gray-level co-occurrence matrix features, gray-level run-length matrix features, gray-level size region matrix features, neighborhood gray-level difference matrix features, and gray-level dependency matrix features, with a total feature dimension of 1694. The SUV radiomics features are extracted using the PyRadiomics toolkit.

[0057] In some embodiments, the resampling of CT images into the PET image space is performed using trilinear interpolation, and the resampled CT images are completely consistent with the PET standard images and SUV standard acquisition value images in terms of matrix size, voxel spacing, and spatial direction.

[0058] In some embodiments, when any of the image modality data, gene modality data, and pathology modality data is temporarily missing, the feature vectors corresponding to the missing modality are all set to zero and a modality missing marker is recorded, keeping the feature dimensions of each encoder unchanged; the image encoder, gene encoder, and pathology encoder are trained using a random modality drop-off strategy during the training phase so that each encoder can still output a stable latent representation when the corresponding modality input is zero.

[0059] In some embodiments, the graph neural network prognostic model adopts a two-layer graph convolutional network structure. The first graph convolutional layer maps node features from a unified latent representation dimension to a hidden layer dimension, and the second graph convolutional layer maps the hidden layer dimension to a prognostic classification dimension. Each graph convolutional layer is followed by a batch normalization layer and a ReLU activation function. The first and second graph convolutional layers are then randomly deactivated at a preset deactivation ratio.

[0060] This embodiment also provides a multimodal prognostic analysis system for lymphoma, including:

[0061] The data acquisition module is used to acquire imaging modality data, genetic modality data, and pathological modality data of the target patient, and generate a unified patient sample identifier;

[0062] The multimodal coding fusion module is used to input the image modal data, gene modal data and pathological modal data into the image encoder, gene encoder and pathological encoder respectively to obtain the potential representation of each modality, and fuse the potential representation of each modality according to the modal weight to form a unified potential representation of the target patient, which serves as the node feature of the graph neural network prognostic model.

[0063] The historical patient potential representation module is used to store the historical patient potential representation base that is solidified after training. The historical patient potential representation base consists of the unified potential representation of each historical patient in the training set. The unified potential representation of each historical patient and the unified potential representation of the target patient are in the same feature space.

[0064] The graph construction module is used to add the unified latent representation of the target patient to the historical patient latent representation base, calculate the patient similarity fusion matrix based on the patient node feature set with the added unified latent representation of the target patient, filter the patient similarity fusion matrix according to the graph threshold, and generate the adjacency matrix required by the graph neural network.

[0065] The graph neural network prognostic reasoning module is used to perform graph neural network prognostic reasoning with the unified latent representation of each patient in the patient node feature set as node features and the adjacency matrix as the graph structure, and output the prognostic result of the target patient.

[0066] The image encoder, gene encoder, pathology encoder, modality weights, and graph neural network prognostic model are all pre-trained and solidified using a training set; the historical patient latent representation disk is solidified after training and used to merge and construct a unified latent representation of new target patients during the online inference phase.

[0067] In some embodiments, the data acquisition module further includes an image preprocessing and segmentation submodule, which is used for:

[0068] Receive PET / CT DICOM data from the target patient and screen for effective PET and CT sequences;

[0069] The PET sequence is converted into a standard PET image, and the standard PET image is converted into an SUV standard uptake value image based on the injection dose, acquisition time, and patient weight information.

[0070] CT images are resampled into the PET image space to obtain resampled CT images;

[0071] The PET / CT lymphoma lesion segmentation model based on nn-UNet is invoked, and the standard PET image and the resampled CT image are used as dual-channel inputs to perform sliding window depth segmentation inference and output standard lesion mask results.

[0072] Spatial consistency verification is performed between the standard lesion mask and the standard SUV image.

[0073] Three-dimensional connected component analysis is performed on the standard lesion mask that has passed the verification to obtain the connected components of each lesion. The volume of metabolic tumor at the lesion level is calculated based on the number of voxels in each connected component and the physical voxel spacing of the SUV standard uptake value image. The total metabolic tumor volume at the patient level is obtained by summing the volumes of metabolic tumor at the lesion level. The SUV radiomics features are extracted using the SUV standard uptake value image and the standard lesion mask as input to obtain image modality data.

[0074] In some embodiments, a quality gating module is further included, which is used to obtain the set of necessary products required under the current prognostic analysis mode after performing graph neural network prognostic inference and outputting prognostic results, and check whether all the necessary products are generated and read successfully; if all are generated and read successfully, the task status is determined to be successful and the prognostic result is returned; if any necessary product is missing or read fails, the task status is determined to be failed and the missing product identifier and failure stage information are returned.

[0075] This application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method as described in any of the preceding claims.

[0076] In some embodiments, the training set is constructed as follows: a single PET / CT examination is used as the basic training sample for segmentation, including PET images, resampled CT images, SUV images consistent with PET space, and expert-annotated lesion masks; patients are used as the basic training sample for prognosis, including patient sample identifiers, standard features of image modalities, standard features of gene modalities, standard features of pathological modalities, PFS24 labels, PFS12 labels, follow-up time, and censoring status;

[0077] The training set, validation set, and test set are divided according to the patient-level stratification principle. All data from the same patient can only be included in one of the three sets: training set, validation set, and test set. Hyperparameter optimization adopts a two-stage strategy from coarse to fine. The first stage performs a grid search within the preset search space. The second stage refines the parameters locally around the optimal combination of the validation set. The final parameters are selected by comprehensively considering indicators such as AUC, C-index, calibration degree, sensitivity, specificity, risk stratification consistency, segmentation Dice coefficient, lesion recall rate, and metabolic tumor volume error.

[0078] In some embodiments, the graph neural network prognostic model uses a hidden layer dimension of 256, the number of nearest neighbors K in the similarity network fusion algorithm is 6, and the fusion parameter μ is 0.25;

[0079] The PFS24 prognostic inference branch uses a graph threshold of 0.65 and an inactivation ratio of 0.75. The edge representation is a threshold-weighted edge, and the PFS24 prognostic inference branch outputs continuous prognostic probability values.

[0080] The PFS12 prognostic inference branch uses a graph threshold of 0.005, an inactivation ratio of 0.5, and binary edges. A low-risk cutoff threshold of 0.1 and a high-risk cutoff threshold of 0.8 are used for PFS12.

[0081] Compared with the prior art, the present invention has the following core beneficial effects:

[0082] I. First successful implementation of effective GNN prognostic inference in a single-patient scenario

[0083] In existing technologies, GNN / GCN models rely on a complete patient similarity graph for reasoning. New patients, as isolated nodes, cannot establish connections with any historical patients in the graph, rendering the GNN's group information propagation mechanism completely ineffective in single-patient prediction scenarios. This invention addresses this by appending a unified latent representation of the target patient to the end of the historical patient latent representation base. It then recalculates the patient similarity fusion matrix containing the new node using the SNF similarity network fusion algorithm, and generates an adjacency matrix based on graph threshold filtering. This allows the previously isolated new node to connect to the historical patient group through similarity edges, transforming the "single-node inference without a graph" problem into a "propagation inference with newly added nodes in the graph" problem. This enables the GNN's group information propagation mechanism to function effectively in clinical single-patient online prediction scenarios.

[0084] II. Constructing an end-to-end traceable closed loop from raw images to prognostic output

[0085] In existing technologies, PET / CT lymphoma lesion segmentation and multimodal prognostic prediction are typically two independent modules, lacking spatial verification and product traceability mechanisms between mask generation and metabolic volume calculation. This invention integrates DICOM screening, SUV conversion and CT trilinear interpolation resampling, nn-UNet segmentation inference and spatial consistency verification, 3D connected component analysis and MTV / TMTV calculation, SUV radiomics extraction, multimodal field alignment and modal-wise encoding fusion, historical base plate incremental mapping and GNN prognostic inference, and quality gating into a complete end-to-end link. Standard lesion masks, lesion-level metabolic tumor volumes, patient-level total metabolic tumor volumes, SUV radiomics features, and prognostic results are all generated under the same patient identifier. Quality gating returns a clear missing product identifier and failure stage information when products are missing, enabling maintenance personnel to accurately pinpoint whether the failure occurred during the segmentation stage, multimodal construction stage, or graph inference stage.

[0086] III. Achieving Deep Semantic Fusion and Adaptive Weight Allocation of Multimodal Heterogeneous Features

[0087] In existing technologies, the three modalities of radiomics (1694 dimensions), genomics (55 dimensions), and pathology (22 dimensions) differ significantly in dimensionality and scale. Directly concatenating these modalities makes it difficult for the model to effectively learn cross-modal complementary information and is prone to overfitting. This invention sets up independent encoders for each of the three modalities, each nonlinearly mapping its input features to a latent representation space of a unified dimension. These are then fused using softmax normalization with trainable modal weights, allowing the model to adaptively allocate the "trust level" of each modality, jointly optimized with a prognostic classification supervision loss. For scenarios where modalities are temporarily unavailable, this invention employs a random modality dropout strategy to train each encoder, ensuring that any encoder can still output a stable latent representation when the corresponding modal input is zero.

[0088] IV. Achieving precise prognostic stratification with dual time-series specificity for PFS24 and PFS12

[0089] In existing technologies, prognostic models typically only model for a single time window, making it difficult to adapt to the different prediction timeframes required for various clinical decision-making scenarios. This invention independently optimizes and solidifies a complete set of GNN model parameters, graph thresholds, inactivation ratios, and risk cutoff thresholds for PFS24 and PFS12. The PFS24 branch uses a graph threshold of 0.65, an inactivation ratio of 0.75, and threshold-weighted edge representation to output continuous prognostic probability values. The PFS12 branch uses a graph threshold of 0.005, an inactivation ratio of 0.5, and binary edge representation, using 0.1 and 0.8 as low-risk and high-risk cutoff thresholds to classify patients into low-risk, intermediate-risk, and high-risk groups, thus optimizing the model for both long-term survival prediction and short-term risk stratification. Attached Figure Description

[0090] Figure 1 Overall module relationship diagram;

[0091] Figure 2 Overall method flowchart;

[0092] Figure 3 : Structural diagram of a segmentation model of lymphoma lesions using PET / CT;

[0093] Figure 4 : Tumor burden characteristics and multimodal sample construction diagram;

[0094] Figure 5 : Structure diagram of a multimodal prognostic prediction model for lymphoma;

[0095] Figure 6 Quality gating and task status determination diagram. Detailed Implementation

[0096] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0097] See Figure 1-6 To address the issue that multimodal prognostic prediction models for lymphoma based on GNNs cannot directly generate an effective patient graph during online inference for a single patient, this application provides a multimodal prognostic analysis method for lymphoma: The method involves acquiring image modality data, gene modality data, and pathological modality data of the target patient and generating a unified patient sample identifier; inputting these three types of modality data into the image encoder, gene encoder, and pathology encoder respectively to obtain the latent representations of each modality; fusing the latent representations of each modality according to modality weights to form a unified latent representation for the target patient, which serves as the node feature of the graph neural network prognostic model; appending the unified latent representation of the target patient to the historical patient latent representation disk, which is composed of the unified latent representations of all historical patients in the training set and resides in the same feature space as the unified latent representation of the target patient; calculating the patient similarity fusion matrix based on the appended patient node feature set, and generating an adjacency matrix according to a graph threshold; using the unified latent representations of each patient in the patient node feature set as node features and the adjacency matrix as the graph structure, performing graph neural network prognostic inference, and outputting the prognostic result.

[0098] The above scheme achieves the following beneficial effects. By appending the unified latent representation of a single patient to the historical patient latent representation base and recalculating the SNF similarity fusion matrix containing the new node, the new patient, originally an isolated node, is connected to the historical patient group through similarity edges. Essentially, this approach transforms the "single-node, graph-less reasoning" problem into a "propagation reasoning problem involving newly added nodes in the graph." During the online reasoning phase, the unified latent representation of the new patient is appended to the end of the historical base according to the sample index. The historical patient portion retains its original index and feature values, while the new patient participates as a new node in subsequent graph construction and GCN propagation. Each encoder, modal weight, and GNN model is pre-trained and solidified on the training set, ensuring consistency between the feature space during the reasoning phase and the training phase. The unified latent representation's expression allows the new patient to activate historical patient nodes with similar prognoses through the similarity network, thereby utilizing group information to assist in individual prognostic prediction and effectively solving the technical obstacle of the baseline GNN model's inability to handle single-patient input without a complete graph structure.

[0099] To obtain the image modality data required for the above technical solutions, and especially to address the spatial inconsistency problem that easily arises in the sequence screening and spatial normalization process of raw PET / CT DICOM data, this application employs the following technical means when acquiring the image modality data of the target patient: receiving the target patient's PET / CT... DICOM data was used to screen for valid PET and CT sequences. PET sequences were converted into standard PET images and then converted into SUV standard uptake value images based on injection dose, acquisition time, and patient weight information. CT images were resampled into the PET image space using trilinear interpolation to obtain resampled CT images. A PET / CT lymphoma lesion segmentation model based on nn-UNet was called, and sliding window depth segmentation inference was performed using standard PET images and resampled CT images as dual-channel inputs to output standard lesion mask results. Spatial consistency verification was performed between the standard lesion mask and the SUV standard uptake value images. Three-dimensional connected component analysis was performed on the verified masks to obtain the connected components of each lesion. The lesion-level metabolic tumor volume was calculated based on the number of voxels and the physical voxel spacing of the SUV images. The total patient-level metabolic tumor volume was obtained by summing the data. SUV radiomics features were extracted using SUV standard uptake value images and standard lesion masks as inputs to obtain image modality data.

[0100] The above scheme achieves the following beneficial effects. By incorporating SUV conversion into the image standardization link, the radioactivity distribution recorded by PET is converted into a physically comparable standard uptake value. By resampling CT to PET space through trilinear interpolation, it ensures that CT anatomical information and PET functional information are aligned in the same spatial reference system. Trilinear interpolation performs a weighted average of the values ​​of the eight neighboring voxels around the target point in three-dimensional space, which can avoid the step-like artifacts that may be generated by nearest-neighbor interpolation while maintaining image continuity, achieving a good balance between computational efficiency and accuracy. The segmentation model inference is performed on spatially unified PET and CT inputs, and the output lesion mask strictly corresponds to the SUV image in the coordinate system. Spatial consistency verification is performed immediately after mask generation, checking the size, orientation, and voxel spacing. Any inconsistency is recorded as a failure stage and subsequent processing is stopped. The above measures upgrade the benchmark nn-UNet model from a "simple segmentation network" to a "service-oriented segmentation link with clinical data access and spatial self-checking capabilities," ensuring that subsequent volume calculation and omics extraction are always performed in the same physical space, eliminating metabolic volume calculation errors caused by coordinate system misalignment. Three-dimensional connected domain analysis organizes discrete voxel-level segmentation results into clinically significant lesion entities. The metabolic tumor volume of each lesion connected domain has a clear physical unit, and the total patient-level metabolic tumor volume obtained by summing can serve as an important input feature for subsequent multimodal prognostic models.

[0101] To enrich the representational dimensions of patient-level tumor burden and enable multimodal prognostic models to acquire more comprehensive quantitative imaging information, this application, after obtaining the total metabolic tumor volume at the patient level, also calculates at least one of the following patient-level tumor burden indicators: total glycolysis at the patient level, maximum SUV, average SUV, peak SUV, maximum lesion volume, and maximum distance between lesions. The SUV radiomics feature is extracted after obtaining the above indicators.

[0102] The above scheme achieves the following beneficial effects. Total metabolic tumor volume reflects the "quantity" of lesions, but it cannot comprehensively reflect the intensity of metabolic activity and spatial distribution characteristics of lesions. By parallelly calculating quantitative indicators across multiple dimensions, such as patient-level total glycolysis, maximum SUV (maximum SUV), average SUV, peak SUV, maximum lesion volume, and maximum distance between lesions, the imaging modal data can characterize the patient's tumor status from different perspectives, including total burden, metabolic intensity, spatial distribution, and lesion heterogeneity. The calculation order of these indicators is as follows: first, the metabolic tumor volume of each lesion is generated from the mask and summarized into the total metabolic tumor volume; then, enhancement indicators such as total glycolysis are calculated based on the total metabolic tumor volume; finally, radiomics features are extracted using SUV images and the mask as input. This order ensures that the extraction of radiomics features shares the same spatially aligned mask and SUV images as the quantitative indicators mentioned above, avoiding inconsistencies caused by using different versions of masks for different calculation paths.

[0103] To define the calculation method for lesion-level metabolic tumor volume with a clear mathematical formula, and to enable those skilled in the art to accurately reproduce the calculation process, this application calculates lesion-level metabolic tumor volume as follows: MTV i = N i ×Δx × Δy × Δz, where MTV i Let N be the metabolic tumor volume of the i-th lesion. i Let be the number of voxels within the connected region of the i-th lesion, and Δx, Δy, and Δz be the physical voxel spacing of the SUV standard uptake value image in the three spatial directions, respectively. The patient-level total metabolic tumor volume is the MTV of all connected regions of the lesions. i The sum. Total glycolysis at the lesion level, calculated in TLG. i = MTV i ×SUVmean i Calculation, where TLG i SUVmean represents the total glycolysis amount of the i-th lesion. i This represents the average SUV standard uptake value of all voxels within the connected region of the i-th lesion. Patient-level total glycolysis is the TLG of all connected regions of the lesions. i sum.

[0104] The above scheme achieves the following beneficial effects. The calculation formula multiplies the voxel count of the 3D discrete mask by the physical voxel spacing of the SUV image in the x, y, and z directions, converting the originally dimensionless pixel count into a metabolic tumor volume with definite physical units (usually cubic centimeters or milliliters). SUVmean i The calculations were performed voxel-by-voxel within the connected domain of the lesion, reflecting the average metabolic activity of that lesion. TLG i Combining both volume and mean metabolic activity, this formula provides a more comprehensive reflection of the metabolic burden of lesions than using MTV or SUVmean alone. The formula is highly practical in clinical practice; any system with DICOM analysis and medical image processing capabilities can reproduce the calculation results, thus meeting the patent law's requirements for clear and complete technical solutions.

[0105] To address the significant differences in dimensionality, units, field order, and missing data patterns among imaging, genetic, and pathological modalities, which make direct concatenation difficult for model training, this application adds a pre-processing step of field alignment and numerical standardization for each modality before inputting it into the encoder. The imaging modality data is concatenated into an image feature vector according to a predefined image field order. This image field includes patient-level total metabolic tumor volume, patient-level total glycolysis, maximum SUV value, average SUV value, peak SUV value, maximum inter-lesion distance, maximum lesion volume, and SUV radiomics characteristics. Missing fields are filled with zero values. The missing markers are recorded; the gene modality data are mapped according to the preset order of lymphoma-related gene fields, and the mutation status, expression level, copy number variation, and detection positivity of each gene field are numerically encoded. Undetected and missing fields are masked and filled with model-defined values, and gene feature vectors are output; the pathology modality data are mapped according to the preset order of pathology fields, and the immunohistochemical indicators positivity and positivity and scoring levels of each pathology field are converted into numerical values, and pathology feature vectors are output by combining the pathology image encoding results and pathology text encoding results; the above three feature vectors are standardized using the same normalized statistics as in the training phase.

[0106] Fixed field order resolves the inconsistency in field arrangement between reports from different patients and testing institutions—by pre-setting the field order, any input test report is rearranged in the same order, ensuring fixed dimensions and interpretable positions. Numerical encoding converts common qualitative descriptions in clinical reports (such as "positive," "negative," "weakly positive," "30%)" into numerical values ​​that the model can process. A missing data marker mechanism distinguishes between "not detected" and "negative" states—in gene testing, not detecting a specific locus has entirely different clinical significance than a negative result; missing data markers allow the model to recognize this difference. The use of normalized statistics ensures consistent input data distribution between the training and inference phases, avoiding performance degradation due to data scaling. These processes convert the three types of heterogeneous data into a unified, fixed-dimensional vector before entering the encoder.

[0107] To further encode the standardized three modal features into a unified-dimensional latent representation, this application sets the image encoder, gene encoder, and pathology encoder as multi-layer fully connected neural networks with identical or different structures, each nonlinearly mapping the input features of its corresponding modality to a unified-dimensional latent representation space. The modality weights are trainable scalar parameters, which, after Softmax normalization, satisfy the condition that each modality weight is non-negative and their sum is 1. During the training phase, the image encoder, gene encoder, pathology encoder, and modality weights are jointly optimized using prognostic classification supervised loss as the training objective, and are then fixed after training.

[0108] The three modalities differ not only in dimensionality but also in information density and noise levels. An independent encoder is set up for each modality, allowing the parameters of each encoder to adapt to the distribution characteristics of its specific modality, rather than forcing all modalities to share the same set of parameters. Softmax normalization automatically adjusts the modality weights in each training step, maintaining a sum of 1. Physically, this means that when fusing information from the three modalities, the model adaptively allocates the "trust level" of different modalities—increasing the weight of a modality when it is information-rich in the current sample and highly correlated with the prognosis, and decreasing its weight when the data quality of a modality is poor. These weights are jointly optimized using the prognosis classification supervision loss as the training objective, ensuring that the unified latent representation after fusion is directly optimized for the prognosis prediction task. After training, the encoders and modality weights are fixed, ensuring that the fusion method in the online inference stage is completely consistent with that in the training stage, and that the prognosis stability is not affected by batch-to-batch drift of modality weights.

[0109] To further clarify the construction method of the historical patient potential representation base and its merging logic with the unified potential representation of the target patient, this application makes the following limitations on the aforementioned base and merging process: The historical patient potential representation base is constructed during the training phase as follows: the image modality data, gene modality data, and pathology modality data of each historical patient in the training set are input into the fixed image encoder, gene encoder, and pathology encoder, respectively, and fused according to the fixed modality weights to form the unified potential representation of each historical patient. The unified potential representations of all historical patients form this base. During the online inference phase, the unified potential representation of the target patient is appended to the end of the historical patient potential representation base according to the sample index, forming an expanded set of patient node features. The historical patient portion retains its original index and feature values, while the target patient participates as a new node in subsequent graph construction and inference.

[0110] The above scheme achieves the following beneficial effects: The latent representation base of historical patients is generated and stored permanently after training, eliminating the need to repeatedly calculate the encoding process of historical patients during online inference, significantly reducing computational latency. The unified latent representation of the target patient is appended to the end of the base according to the sample index, rather than inserted in the middle, ensuring the invariance of the index and feature values ​​of historical patient nodes, making any pre-computed content based on historical patient indices reusable. In the expanded patient node feature set, the features and order of historical patient nodes remain unchanged, with the newly added node located at the end of the set. This organization allows the graph construction module to process the node feature set containing new nodes in a unified way—from the perspective of the graph construction algorithm, this set has the exact same format as the full patient set during the training phase. After the target patient participates in subsequent graph construction as a new node, similarity edges are established between it and historical patients, enabling the new patient to receive prognostic information from similar historical patients through the graph convolutional layer.

[0111] To clarify the specific implementation methods of "calculating the inter-patient similarity fusion matrix" and "generating the adjacency matrix," this application limits the calculation of the inter-patient similarity fusion matrix to using a similarity network fusion algorithm. This algorithm calculates a multi-kernel similarity matrix based on the similarity between the features of each node in the patient node feature set and performs iterative fusion to obtain the fused inter-patient similarity fusion matrix. The graph construction process includes: retaining edges in the inter-patient similarity fusion matrix whose similarity values ​​are greater than or equal to a graph threshold, and setting edges whose similarity values ​​are less than the graph threshold to zero, to obtain a weighted adjacency matrix; or, setting edges whose similarity values ​​are greater than or equal to the graph threshold to 1 and edges whose similarity values ​​are less than the graph threshold to 0, to obtain a binary adjacency matrix. The adjacency matrix, after row normalization, serves as the normalized adjacency matrix required for graph neural network propagation.

[0112] The above scheme achieves the following beneficial effects. The similarity network fusion algorithm, through iterative propagation between different similarity measures (such as Euclidean distance-based similarity, cosine similarity-based correlation, Manhattan distance-based similarity, etc.), can fuse patient relationships from multiple similarity measure perspectives, reflecting the true clinical similarity between patients more robustly than a single similarity measure. The introduction of a graph threshold plays a role in "noise reduction" and "sparseness"—weak connections with similarity below the threshold are deleted, reducing interference information from dissimilar patients during GNN propagation and maintaining the sparsity of the adjacency matrix, thus reducing the computational complexity of graph convolution. The weighted adjacency matrix retains the edge weight information, allowing the GNN to assign different contribution strengths to neighbors with different similarities during propagation; the binary adjacency matrix treats all retained edges equally. The two methods are suitable for different clinical scenarios: the weighted method is more suitable for scenarios with significant differences in patient similarity, while the binary method is more suitable for scenarios requiring suppression of abnormally high-weight edges during propagation. Row normalization makes the sum of the weights of the adjacent edges of each node equal to 1, ensuring the numerical stability of node features during graph convolution propagation.

[0113] To meet the clinical need for two different prognostic prediction timelines, PFS24 and PFS12, and to address the challenge of adapting the same multimodal input to prognostic prediction across different timelines, this application makes the following limitations: the prognostic results include the PFS24 prognostic probability; the same set of multimodal data is also used for PFS12 prognostic inference, outputting PFS12 risk grouping results. PFS24 and PFS12 prognostic inference employ time-specific graph neural network model parameters, graph thresholds, inactivation ratios, and risk cutoff thresholds, respectively. These parameters are independently optimized and solidified for both timelines during the training phase using a validation set. The PFS24 prognostic probability is a continuous probability value; the PFS12 risk grouping results classify patients into low-risk, intermediate-risk, and high-risk groups based on low-risk and high-risk cutoff thresholds.

[0114] The above scheme achieves the following beneficial effects. PFS24 and PFS12 correspond to different clinical decision-making scenarios—PFS24 is used to assess the probability of progression-free survival within two years, suitable for developing long-term treatment strategies; PFS12 is used to assess the risk of progression within one year, suitable for adjusting short-term treatment plans. The prognostic labels of the two time periods are distributed differently in the training set, and the optimal graph threshold, inactivation ratio, and model parameters may also be different. By independently searching for and fixing the optimal parameters for the two time periods, the output of the same multimodal coding fusion module can be connected to two time-specific GNN / GCN configurations to output prognostic results at different time scales. The low-risk (threshold 0.1) and high-risk (threshold 0.8) truncation of PFS12 risk grouping converts continuous probabilities into three-category risk levels, facilitating clinicians to make rapid treatment decisions: the low-risk group can be considered for downgraded treatment or follow-up observation, while the high-risk group needs to be considered for intensive treatment or clinical trial enrollment.

[0115] To address the problem of error propagation at each level between the segmentation model, feature calculation model, and prognostic prediction model, which is difficult to trace and locate, this application adds a quality gating step after executing graph neural network prognostic inference and outputting the prognostic result: It obtains the set of essential products required under the current prognostic analysis mode, including a standard lesion mask, lesion-level metabolic tumor volume, patient-level total metabolic tumor volume, SUV radiomics features, the aforementioned image feature vector, gene feature vector, and pathological feature vector, patient-to-patient similarity fusion matrix, and the prognostic result; it checks whether all the above essential products have been generated and successfully read; if all have been generated and successfully read, the task status is determined to be successful and the prognostic result is returned; if any essential product is missing or reading fails, the task status is determined to be failed, and the missing product identifier and failure stage information are returned.

[0116] By defining a set of required deliverables and enforcing integrity checks at the end of the process, segmentation quality, feature calculation integrity, and graph inference link integrity are uniformly incorporated into quality control. Unlike simple checks that only check the existence of the final output, this solution checks intermediate deliverables one by one, enabling fault location to be precise down to a specific stage—for example, if the standard lesion mask is missing but its subsequent deliverables exist, the system can determine that the segmentation stage has failed; if the mask and volumetric features are complete but the image feature vectors are missing, the system can determine that the multimodal construction stage has failed. This fine-grained location capability is particularly critical for asynchronous services, allowing maintenance personnel to directly locate the faulty module based on the returned missing deliverable identifiers, without needing to debug logs segment by segment. When the deliverables are complete, the prognostic result is returned; when the deliverables are incomplete, the result output is blocked, preventing unreliable prognostic conclusions from being output due to intermediate link breaks at the system level.

[0117] To address the format differences between labeled data during the training phase and input data during the online inference phase, and to support the verification and comparison of historical segmentation results, this application identifies and saves SEG sequences if they exist in the DICOM data when screening PET and CT sequences in DICOM. These SEG sequences are used for expert annotation reading during the training phase, comparison of historical segmentation results, or segmentation quality verification, but are not required input for online automatic segmentation.

[0118] The SEG sequences in the DICOM standard store structured data for delineation or segmentation. By identifying and saving SEG sequences, the system can operate in two modes: during the training phase, SEG sequences can be used as expert-annotated lesion masks for segmentation model training; during the inference phase, if saved SEG sequences exist from previous examinations of the same patient, they can be used as historical comparisons or quality verification references for the current automatic segmentation results. However, SEG sequences are not a necessary input for online automatic segmentation—this means that even if SEG sequences are not present in DICOM or SEG sequence retrieval fails, the segmentation model can still execute normally without interrupting the main process due to a lack of optional input. This approach balances functional completeness and process robustness.

[0119] To define the specific baseline architecture and training configuration of the PET / CT lymphoma lesion segmentation model, this application specifies that the segmentation model is based on the nn-UNet 3D full-resolution segmentation pipeline, employing a 3D multi-channel input structure with standard PET images and resampled CT images as dual-channel inputs, and expert-reviewed lesion masks as supervision labels. The segmentation loss function consists of a weighted sum of Dice loss, cross-entropy loss, small lesion constraint terms, and boundary constraint terms. During training, random pruning, scale perturbation, intensity perturbation, and flip enhancement are employed. During validation, the optimal checkpoint is selected as the online inference model based on lesion Dice coefficient, lesion recall rate, number of false positive lesions, and metabolic tumor volume error.

[0120] nn-UNet, as a self-configurable segmentation benchmark, can automatically adjust network depth, number of convolutional kernels, and upsampling path according to dataset characteristics, exhibiting strong domain adaptability to medical images. Its 3D multi-channel input structure enables the network to simultaneously utilize metabolic information from PET and anatomical information from CT for joint judgment, achieving higher segmentation accuracy than single-modal input. The loss function includes Dice loss focusing on the overall overlap of segmented regions, cross-entropy loss focusing on voxel-level classification accuracy, a small lesion constraint term improving the detection sensitivity for small lesions, and a boundary constraint term optimizing the smoothness of segmentation boundaries. Four enhancement methods during training (cropping, scale perturbation, intensity perturbation, and flipping) expand the diversity of training samples from different dimensions, improving the model's generalization ability to different acquisition devices and patient body types. During the validation phase, five metrics (Dice, recall, false positives, and MTV error) are used for model selection to avoid a single metric dominating the selection while ignoring performance degradation in other dimensions.

[0121] To clarify the extraction criteria and specific feature types for SUV radiomics features, and to enable those skilled in the art to reproduce the feature extraction process based on these limitations, this application limits the extraction of SUV radiomics features to the IBSI (Image Biomarker Standardization Initiative) standard. The feature types include multiple features among first-order statistics, shape features, gray-level co-occurrence matrix features, gray-level run-length matrix features, gray-level size region matrix features, neighborhood gray-level difference matrix features, and gray-level dependence matrix features, with a total feature dimension of 1694. The PyRadiomics toolkit is used for extraction.

[0122] The above scheme achieves the following beneficial effects. IBSI is an internationally recognized standardization initiative in the field of medical radiomics, which specifies the definition, naming, and calculation methods of radiomics features. Adherence to the IBSI standard ensures that the features extracted by this scheme are comparable and reproducible at the feature level to existing research results both domestically and internationally. The seven types of features described above characterize the heterogeneity of lesions from different dimensions: first-order statistics reflect the distribution pattern of voxel values; shape features describe the geometric morphology of lesions; gray-level co-occurrence matrix reflects the joint spatial distribution of voxel values; gray-level run matrix describes the run length distribution of consecutive identical gray values; gray-level size region matrix describes the size distribution of connected regions with the same gray values; neighborhood gray-level difference matrix reflects local gray-level differences; and gray-level dependency matrix describes the spatial dependency relationship of similar gray values. PyRadiomics, as an open-source tool library, has been widely validated, and using this tool library to extract features ensures the reproducibility of this scheme. The specific numerical values ​​of 1694 dimensions provide a clear feature magnitude, enabling system designers to plan storage and computing resources accordingly.

[0123] To address the issue of spatial alignment accuracy discrepancies caused by different interpolation methods when resampling CT images to PET image space, this application limits the interpolation method for resampling CT images to PET image space to trilinear interpolation. The resampled CT images are completely consistent with standard PET images and SUV standard acquisition value images in matrix size, voxel spacing, and spatial orientation.

[0124] Trilinear interpolation performs a weighted average of the values ​​of the eight neighboring voxels around a target point in three-dimensional space, maintaining image continuity while avoiding the staircase artifacts that may occur with nearest-neighbor interpolation. CT values ​​(Henness units) reflect the X-ray attenuation characteristics of tissues, and their spatial gradients exhibit physical continuity, which trilinear interpolation effectively preserves. Trilinear interpolation achieves a good balance between computational efficiency and accuracy: compared to the more complex cubic spline interpolation, trilinear interpolation requires less computation, making it suitable for batch processing in clinical settings; compared to nearest-neighbor interpolation, trilinear interpolation offers higher accuracy without losing spatial information due to integer rounding. Completely consistent spatial alignment (consistent matrix size, voxel spacing, and spatial orientation) eliminates the need for additional coordinate transformations in subsequent voxel-level operations (such as voxel-by-voxel correspondence between masks and SUV images, and dual-channel input from CT and PET).

[0125] To address the issue of system operation even when certain modalities of data are temporarily missing in clinical practice, this application specifies the following handling strategy for modality loss: When any of the imaging, gene, or pathology modalities is temporarily missing, all feature vectors corresponding to the missing modality are set to zero and a modality loss marker is recorded, while maintaining the feature dimensions of the input encoders unchanged. The imaging encoder, gene encoder, and pathology encoder employ a random modality dropout strategy during the training phase to ensure that each encoder outputs a stable latent representation even when the corresponding modality input is zero.

[0126] The above scheme achieves the following beneficial effects. In clinical practice, not every patient completes all three types of tests—some patients may lack gene sequencing results, some may lack digital pathological images, and some patients' PET / CT data may be unreadable due to technical reasons. By setting all feature vectors of the temporarily missing modalities to zero and recording the missing markers, the system allows modality loss without changing the input dimension. However, simply setting them to zero introduces a problem: the encoder has never seen all-zero inputs during training, and encountering all-zero inputs during inference may produce outlier representations. To solve this problem, this application introduces a random modality discarding strategy during the training phase—that is, in each training batch, the input of a certain modality is randomly set to zero with a certain probability. This strategy forces each encoder to learn representations with prognostic value from other modalities even when the corresponding modal input is partially or completely missing. It also enables the encoder to learn the ability to output stable potential representations under all-zero inputs, thus enabling robust handling when encountering modality loss during the inference phase.

[0127] To define the specific structure and connections between layers of the graph neural network prognostic model, enabling those skilled in the art to implement a working GNN model based on this definition, this application specifies the graph neural network prognostic model as employing a two-layer graph convolutional network structure: the first graph convolutional layer maps node features from a unified latent representation dimension to a hidden layer dimension, and the second graph convolutional layer maps the hidden layer dimension to a prognostic classification dimension. Each graph convolutional layer is followed by a batch normalization layer and a ReLU activation function. The first and second graph convolutional layers are then randomly deactivated according to a preset deactivation ratio.

[0128] The above scheme achieves the following beneficial effects. Using two layers of graph convolutions instead of deeper layers is based on the scale of lymphoma prognostic data (hundreds to over a thousand cases). At this scale, two layers of graph convolutions are sufficient to effectively aggregate neighborhood information while avoiding the oversmoothing problem common in deep GNNs (i.e., as the number of layers increases, the representations of each node tend to be consistent, losing node distinguishability). The first layer of graph convolution maps the unified latent representation to the hidden layer dimension, completing the transformation from "node's own features" to "aggregated features of the node and its neighbors." The second layer of graph convolution maps the hidden layer dimension to the prognostic classification dimension, outputting the prognostic class probability of each node. Batch normalization layers standardize the output of each layer of graph convolutions, keeping the distribution of intermediate layer outputs stable during training, accelerating convergence, and reducing sensitivity to the learning rate. The ReLU activation function introduces non-linearity, enabling the network to learn the complex mapping relationship between features and prognosis. Random deactivation randomly sets the output of some neurons to zero at a preset ratio, forcing the network to be independent of any single neuron, thereby enhancing generalization ability and reducing overfitting.

[0129] This application also provides a multimodal prognostic analysis system for lymphoma, comprising: a data acquisition module for acquiring image modality data, gene modality data, and pathological modality data of the target patient and generating a unified patient sample identifier; a multimodal encoding and fusion module for inputting the three types of modality data into the image encoder, gene encoder, and pathological encoder respectively, obtaining the latent representations of each modality, and fusing them into a unified latent representation of the target patient; a historical base disk storage module for storing the historical patient latent representation base disk that has been solidified after training; a graph construction module for appending the unified latent representation of the target patient to the historical patient latent representation base disk, calculating the patient similarity fusion matrix, and generating an adjacency matrix based on a graph threshold; and a graph neural network prognostic inference module for performing GNN prognostic inference using the unified latent representations of each patient in the patient node feature set as node features and the adjacency matrix as the graph structure, and outputting the prognostic result.

[0130] The data acquisition module is responsible for collecting and initially processing heterogeneous data from multiple sources; the multimodal coding fusion module is responsible for the transformation from raw features to deep representations; the historical base storage module writes data during the training phase and reads only during the inference phase, ensuring the stability and consistency of the base; the graph construction module is responsible for similarity calculation and graph threshold filtering; and the graph neural network prognostic inference module is responsible for the final graph propagation and classification output. This modular architecture allows each module to be developed, tested, and upgraded independently—for example, when a new segmentation model emerges, only the segmentation submodule in the data acquisition module needs to be replaced without modifying other modules; when the prognostic task expands from PFS24 to other time windows, only the configuration of the graph neural network prognostic inference module needs to be adjusted without rewriting the entire system.

[0131] This application further configures the data acquisition module with an image preprocessing and segmentation submodule. This submodule is used for: receiving PET / CT DICOM data from the target patient and filtering valid PET and CT sequences; converting PET sequences into standard PET images and converting them into SUV standard uptake value images; resampling CT images into the PET image space using trilinear interpolation; calling a PET / CT lymphoma lesion segmentation model based on nn-UNet, using standard PET images and resampled CT images as dual-channel inputs to perform segmentation inference and output a standard lesion mask; performing spatial consistency verification between the mask and the SUV images; performing three-dimensional connected component analysis on the verified mask and calculating the lesion-level metabolic tumor volume, summarizing the patient-level total metabolic tumor volume, extracting SUV radiomics features, and obtaining image modality data.

[0132] The above scheme achieves the following beneficial effects. This submodule encapsulates the generation of image modality data as an independent functional unit, enabling the system's data acquisition module to possess complete medical image processing capabilities. Internally, the submodule executes the processing flow sequentially: DICOM filtering → SUV conversion → CT trilinear interpolation resampling → nn-UNet segmentation → spatial verification → connected component analysis → MTV / TMTV / radiomics calculation. The data dependencies between each sub-step are clearly defined. The spatial verification result determines whether to continue with subsequent steps, avoiding the waste of ineffective computational resources. The interface between this submodule and the multimodal coding fusion module is standardized to an image modality data format—regardless of the specific sequence of the input DICOM, the submodule's output is uniformly formatted image modality data, freeing the upper-level modules from concern themselves with the internal details of image processing.

[0133] To implement the aforementioned quality gating function at the system level, this application adds a quality gating module to the system. This module is used to, after executing graph neural network prognostic inference and outputting the prognostic result, obtain the set of necessary products required under the current prognostic analysis mode, and check whether all necessary products have been generated and successfully read. If all are successfully generated, the task status is determined to be successful and the prognostic result is returned. If any product is missing or reading fails, the task status is determined to be failed and the missing product identifier and failure stage information are returned.

[0134] The quality gating module, acting as the final quality control checkpoint in the system, operates independently of other functional modules, centrally inspecting the output products of upstream modules. This module does not participate in prognostic calculations but has the authority to halt result output—this separation of "calculation-quality control" ensures the objectivity of quality checks. The set of required products can be dynamically adjusted according to the prognostic analysis mode (PFS24 or PFS12), with different product checklists for different modes, allowing the granularity of quality checks to be adapted to specific clinical tasks. The returned missing product identifiers and failure stage information provide clear troubleshooting clues for system maintenance.

[0135] This application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method as described in any of the above.

[0136] This invention is protected by a computer-readable storage medium, enabling the technical solutions of this invention to be deployed and implemented on various computing devices. The storage medium is not limited to a specific type of memory, but includes, but is not limited to, ROM, RAM, hard disks, solid-state drives, USB flash drives, optical disks, etc. When the program is executed, the processor processes the input data sequentially according to the method steps described above, and finally outputs the pre-result.

[0137] To clarify the construction method of the training set and the principles for dividing training / validation / testing, avoid data leakage, and ensure the reliability of model evaluation, this application constructs the training set as follows: A single PET / CT examination is used as the basic training sample for segmentation, including PET images, resampled CT images, SUV images consistent with the PET space, and expert-annotated lesion masks; patients are used as the basic training samples for prognosis, including patient sample identifiers, standard features of image modalities, standard features of gene modalities, standard features of pathological modalities, PFS24 labels, PFS12 labels, follow-up time, and censoring status. The division of the training set, validation set, and test set adopts a patient-level hierarchical principle, with all data from the same patient entering only one set. Hyperparameter optimization adopts a two-stage strategy from coarse to fine: the first stage performs a grid search within a preset search space; the second stage refines the parameters locally around the optimal combination in the validation set, using AUC (area under the curve), C-index (consistency index), calibration, sensitivity, specificity, risk stratification consistency, segmentation Dice coefficient, lesion recall, and metabolic tumor volume error as indicators to comprehensively select the final parameters.

[0138] The patient-level stratification principle avoids segmenting the imaging, genetic, pathological, and follow-up data of the same patient into different sets. If a patient's PET / CT scans are included in the training set while their genetic data is included in the test set, the model performance evaluated on the test set will be overestimated because the model has indirectly learned the patient's characteristics through their imaging information during training. Patient-level stratification ensures that patients in the training, validation, and test sets do not overlap, and the test set evaluation results reflect the model's generalization ability on novel patients. A coarse-to-fine two-stage hyperparameter search strategy strikes a balance between computational efficiency and optimization accuracy: the first stage uses a grid search to cover a large range of parameters to locate the optimal region, while the second stage performs a finer search near the optimal region to approximate the global optimum. Multiple indicators, including AUC, C-index, calibration, sensitivity, specificity, risk stratification consistency, Dice coefficient, lesion recall, and MTV error, are used for model selection to avoid biases that may arise from a single indicator, ensuring that the final selected model achieves an optimal balance between segmentation accuracy and prognostic prediction accuracy.

[0139] To further clarify the specific fixed parameter values ​​for the two prognostic inference branches, PFS24 and PFS12, and enable those skilled in the art to directly configure and implement a working prognostic prediction model based on these parameters, this application limits the hidden layer dimension of the graph neural network prognostic model to 256, the nearest neighbor number K of the similarity network fusion algorithm to 6, and the fusion parameter μ to 0.25. The PFS24 prognostic inference branch uses a graph threshold of 0.65, an inactivation ratio of 0.75, and threshold-weighted edges, and outputs continuous prognostic probability values. The PFS12 prognostic inference branch uses a graph threshold of 0.005, an inactivation ratio of 0.5, and binary edges, with 0.1 as the low-risk truncation threshold and 0.8 as the high-risk truncation threshold.

[0140] The above parameter values ​​were solidified after independent optimization on a validation set using 1014 PET / CT preprocessing samples, representing the optimal configuration for this data scale and clinical task. The hidden layer dimension of 256 achieves a good balance between model expressiveness and parameter count—too low a dimension results in insufficient model expressiveness, while too high a dimension leads to overfitting with small samples. The SNF nearest neighbor count K=6 means that each node considers its six nearest neighbors when constructing the similarity network; this value is optimal in preserving effective similarity relationships and filtering noise. The graph threshold of the PFS24 branch (0.65) is higher than that of the PFS12 branch (0.005), reflecting the stronger selectivity of PFS24 in predicting prognostic outcomes based on patient similarity relationships—because the two-year prognosis is influenced by more confounding factors, only patient pairs with sufficiently high similarity can provide effective propagation information. The inactivation rate of the PFS24 branch (0.75) is higher than that of the PFS12 branch (0.5), reflecting that the PFS24 branch requires stronger regularization to cope with more complex prognostic patterns. The PFS12 branch uses 0.1 and 0.8 as cutoff thresholds for low and high risk, respectively, to divide patients into three groups, providing clear action guidelines for clinicians.

[0141] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings. The following embodiments are only used to illustrate the present invention and should not be regarded as limiting the scope of the present invention.

[0142] I. System Overall Architecture

[0143] like Figure 1As shown, this invention starts with patient PET / CT DICOM data and sequentially performs image standardization, PET / CT lymphoma lesion segmentation based on nn-UNet, calculation of metabolic indicators and radiomics features, multimodal input construction, multimodal coding fusion potential characterization calculation, SNF patient similarity graph construction, prognostic inference based on GNN / GCN, and key product quality gating. The entire system consists of a data acquisition module, a multimodal coding fusion module, a historical base disk storage module, a graph construction module, a graph neural network prognostic inference module, and a quality gating module working collaboratively.

[0144] The data acquisition module receives DICOM data from a patient's PET / CT scan, height and weight information, gene testing results, and pathological test results. It outputs standard feature vectors for three modalities: image modality data, gene modality data, and pathological modality data, and generates a unified patient sample identifier for the patient. The multimodal encoding and fusion module takes these three modal standard feature vectors as input, encodes them separately by each modality encoder, and then fuses them according to modality weights to output the patient's unified latent representation. The historical baseline storage module stores the unified latent representations of all historical patients solidified during the training phase. The graph construction module appends the target patient's unified latent representation to the end of the historical baseline, calculates the similarity fusion matrix of the expanded patient set, and generates an adjacency matrix. The graph neural network prognostic inference module performs graph convolutional propagation using the unified latent representation as node features and the adjacency matrix as the graph structure, outputting the prognostic result. The quality gating module checks the completeness of all necessary products after prognostic inference is completed, determining the task status as successful or unsuccessful accordingly.

[0145] II. Generation of Image Modal Data

[0146] like Figure 3 As shown in the diagram, the PET / CT lymphoma lesion segmentation model structure diagram details the complete process of generating image modality data. This process takes the PET / CT DICOM set of the target patient as input and goes through the following processing steps in sequence.

[0147] First, DICOM sequence filtering is performed. The system parses the modality label and sequence description field in the DICOM file header, identifies and extracts PET and CT sequences. PET sequences typically contain multiple time frames or bed positions; the system selects static PET sequences covering the whole body based on sequence length and coverage. For CT sequences, plain or diagnostic CT sequences matching the PET acquisition range are selected. If SEG sequences exist in the DICOM set, the system identifies and saves the SEG sequences but does not use them as necessary input for online automatic segmentation.

[0148] Secondly, image standardization is performed. For selected PET sequences, the system reads parameters such as the radioactivity correction factor, acquisition time, and attenuation correction coefficient from the DICOM header information, converting the raw PET count data into standard PET images in Bq / mL units. Based on this, the system reads the injection dose (in MBq), injection time, acquisition time, and patient weight (in kg), and calculates the SUV value for each voxel using the standard SUV conversion formula: SUV = Tissue Concentration (Bq / mL) × Weight (kg) / Injection Dose (MBq) × 1000, obtaining the SUV standard uptake value image. For CT sequences, the system reads their pixel spacing and slice thickness, and resamples the CT images to the PET image space using trilinear interpolation to obtain resampled CT images with the same matrix size, pixel spacing, and spatial orientation. After the above processing, the standard PET image, the SUV standard uptake value image, and the resampled CT image have identical matrix size, pixel spacing, and spatial orientation.

[0149] Next, the nn-UNet segmentation inference is executed. The system reads the pre-trained and fixed nn-UNet 3D full-resolution segmentation model configuration, training plan, and optimal model checkpoints. This segmentation model uses a 3D multi-channel UNet as the network backbone, with two input channels, receiving PET standard images and resampled CT images respectively. During inference, the system crops the PET standard images and resampled CT images into standard-sized 3D blocks, which are then sequentially input into the network in a sliding window manner. The output of each window is processed by softmax to obtain a lesion probability map. The probability maps of all windows are weighted by position to obtain a soft probability map of the entire image range, which is then thresholded and binarized to convert it into a standard lesion mask. The system saves the soft probability map for quality verification.

[0150] Next, spatial consistency verification is performed. The system reads the metadata of the SUV standard acquisition value image (matrix dimensions DimX, DimY, DimZ, voxel spacing Δx, Δy, Δz, spatial orientation) and compares it one by one with the corresponding metadata of the standard lesion mask. If any dimension is inconsistent, the system records the "segmentation-SUV spatial mismatch" failure stage and stops subsequent processing, without passing the segmentation result to downstream modules. If all dimensions are consistent, the verification passes.

[0151] Finally, tumor burden calculation is performed. For example... Figure 3 As shown, the system performs three-dimensional connected component analysis on the validated standard lesion mask, and uses 26-neighborhood connectivity rules to label the independent lesion connected components C1, C2, ..., C6. K For the i-th connected component, the system counts its number of voxels, N. iAnd calculate the metabolic tumor volume of the lesion according to formula (1):

[0152] MTV i = N i × Δx × Δy × Δz (1)

[0153] Where Δx, Δy, and Δz represent the physical voxel spacing of the SUV standard capture value image in the three directions. Simultaneously, the system traverses all voxels within this connected region to calculate the average SUV standard capture value, SUVmean. i And calculate the total glycolysis amount of the lesion according to formula (2):

[0154] TLG i = MTV i × SUVmean i (2)

[0155] The system summarizes the MTVs of all connected domains of the lesions. i The sum of these yields the total metabolic tumor volume (TMTV) at the patient level, which is then used to summarize all TLGs. i The sum of these values ​​yields the total glycolysis at the patient level. Furthermore, the system calculates the maximum SUV, average SUV, and peak SUV for each lesion, and determines the maximum lesion volume and the maximum distance between lesions.

[0156] Based on this, the system uses SUV standard uptake value images and standard lesion masks as input, and extracts SUV radiomics features according to the IBSI standard using the PyRadiomics toolkit. Feature types include first-order statistical features, shape features, gray-level co-occurrence matrix features, gray-level run-length matrix features, gray-level size region matrix features, neighborhood gray-level difference matrix features, and gray-level dependence matrix features, with a total feature dimension of 1694. All of the above numerical features—TMTV, TLG, SUVmax, SUVmean, SUVpeak, maximum distance between lesions, maximum lesion volume, and the 1694-dimensional radiomics features—are output as standard feature vectors of the image modality data.

[0157] III. Field Alignment and Standardization of Multimodal Data

[0158] Figure 4 The system also demonstrates the process of tumor burden characteristics and multimodal sample construction. After obtaining the standard feature vectors of the imaging modality data, the system aligns them with the genetic modality data and pathological modality data using a unified patient sample identifier.

[0159] For gene modality data, the system reads the raw gene expression matrix, mutation detection table, and structured gene report data. Gene fields are mapped according to a pre-defined order of 55 lymphoma-related gene fields—including but not limited to common lymphoma mutation genes such as MYC, BCL2, BCL6, TP53, CD79B, and CARD11. For each gene field, the system encodes its mutation status (wildtype / mutant) as 0 / 1, retains expression levels (such as FPKM and TPM values) as continuous values, encodes copy number variations (amplification / deletion / normal) as +1 / -1 / 0, and encodes positive / negative detection as 1 / 0. Undetected sites and unrecognized fields are masked (recorded as -1 or NaN) and filled with default values ​​agreed upon by the model. The final output is a 55-dimensional gene feature vector.

[0160] For pathological modality data, the system extracts immunohistochemical indicators from the pathology report—including preset pathological fields such as BCL2, BCL6, CD10, CD20, Ki-67, MYC, EBER, and PD-L1. Negative, positive, and weakly positive are encoded as 0, 1, and 0.5, respectively; percentage values ​​are directly used as continuous values; and rating levels (e.g., 1+, 2+, 3+) are encoded as 1, 2, and 3. Simultaneously, the system uses a pre-trained ResNet50 network to extract 2048-dimensional image features from digitized pathological slide images and a pre-trained Clinical-BERT model to extract 768-dimensional text features from pathological text reports. The image and text encoding results are then dimensionality-reduced through their respective fully connected layers and concatenated with the structured pathological indicators to form a pathological feature vector.

[0161] After completing the above field alignment and numericalization, the system performs Z-score standardization on the image feature vector, gene feature vector, and pathological feature vector using the normalized statistics saved during the training phase—that is, subtracting the mean of the training set and dividing by the standard deviation of the training set—to ensure that the feature distribution input to each encoder is consistent with that during the training phase. If data for a certain modality is temporarily missing, the system sets all feature vectors corresponding to that modality to zero and records the modality missing marker.

[0162] IV. Multimodal coding fusion and latent representation generation

[0163] like Figure 5 As shown in the diagram, the multimodal prognostic prediction model structure diagram for lymphoma illustrates the complete processing chain from the standard feature vectors of the three modalities to the final prognostic result.

[0164] Image feature vector input image encoder E radio Gene feature vectors are input into the gene encoder E. gene Pathological feature vectors are input to the pathological encoder E. pathAll three encoders are structured as three-layer fully connected neural networks. The image encoder has an input dimension of 1694-dimensional image features plus several dimensions of tumor burden indicators, a hidden layer dimension of 512 dimensions, and an output dimension of 256 dimensions. The gene encoder has an input dimension of 55 dimensions, a hidden layer dimension of 128 dimensions, and an output dimension of 256 dimensions. The pathology encoder has an input dimension of 22-dimensional structured features plus the sum of image / text encoded features, a hidden layer dimension of 256 dimensions, and an output dimension of 256 dimensions. All three encoders output a latent representation h of the same dimension. radio h gene and h path .

[0165] Modal weights α radio α gene and α path The parameters are trainable scalars, which, after Softmax normalization, satisfy the condition that the sum of the three parameters is 1 and all are non-negative. The system calculates the fused unified latent representation according to equation (3):

[0166] z = α radio × h radio + α gene × h gene + α path × h path (3)

[0167] The unified latent representation z has a dimension of 256 and serves as the GNN node feature for this patient.

[0168] During the training phase, the image encoder, gene encoder, pathology encoder, and modality weights are jointly optimized with prognostic classification supervision loss as the training objective. Specifically, the fused unified latent representation is passed through a fully connected layer for prognostic classification to output the prognostic probability. The cross-entropy loss between this probability and the true prognostic label serves as the supervision loss for the encoding fusion module, and the gradient is backpropagated to the parameters of each encoder and modality weight. After training, all parameters are fixed, and the encoders and modality weights are no longer updated during the inference phase.

[0169] During the training phase, PET / CT examination data from all training set patients were processed through an image modality generation process to obtain image modality data. Combined with their genetic and pathological data, and through the aforementioned coding fusion process, a unified latent representation for each historical patient was obtained. The unified latent representations of all historical patients were arranged in order of patient sample identifier, forming the historical patient latent representation base disk Z. base Dimension N history × 256, where N history This represents the total number of patients in the training set. This base disk is permanently stored in the historical base disk storage module after training is complete.

[0170] V. Dynamic Graph Construction and Single Patient Access

[0171] During the online inference phase, the system uses the unified latent representation z obtained by encoding and fusing the target patient. new Added to the historical patient latent characterization base Z by sample index base At the end, the amplified patient node feature set Z is obtained. all =[Z base ; z new ], dimension (N history + 1) × 256. The node features of historical patients remain unchanged, and the target patient is used as the index N. history +1 new node.

[0172] The system is based on Z all Calculate the patient similarity fusion matrix W SNF Specifically, the system calculates similarity matrices based on Euclidean distance, cosine similarity, and Manhattan distance, respectively. For each similarity metric, each patient selects its K nearest neighbors to construct a nearest-neighbor similarity matrix. These nearest-neighbor similarity matrices are iteratively fused using a similarity network fusion algorithm to obtain the fused patient-to-patient similarity fusion matrix W. SNF Dimension (N) history + 1) × (N history + 1). W SNF (i,j) represents the overall similarity between patient i and patient j, with a value between 0 and 1.

[0173] Subsequently, the system adjusts W according to the graph threshold τ. SNF Edge filtering is performed. In the PFS24 branch, τ=0.65, and a threshold-weighted edge representation method—W—is used. SNF Edges with (i,j) ≥ τ retain their original similarity values ​​as edge weights, W. SNF Edges with (i,j) < τ are set to zero. In the PFS12 branch, τ = 0.005, and a binary edge representation is used—W. SNF Set the edges with (i,j) ≥ τ to 1, W SNF Edges with (i,j) < τ are set to 0. After filtering, the adjacency matrix B (weighted or binary) is obtained, with dimensions (N). history + 1) ×(N history + 1).

[0174] The system performs row normalization on B. Let G be the row degree matrix of B — G(i,i) = Σ j B(i,j). Calculate the normalized adjacency matrix according to equation (4):

[0175] A norm= (G + εI) -1 B (4)

[0176] Where ε is a minimal constant to prevent division by zero, and I is the identity matrix. A norm The sum of each row is 1.

[0177] VI. Prognostic Inference Using Graph Neural Networks

[0178] The system uses the amplified patient node feature set Z all The unified latent representation of each patient in the middle is used as the node feature H (0) = Z all Using the normalized adjacency matrix A norm As a graph structure, it performs propagation through a two-layer graph convolutional network.

[0179] The convolution of the first layer graph is calculated according to equation (5):

[0180] H (1) = ReLU(BN(A norm × H (0) × Θ0)) (5)

[0181] Where Θ0 is the first-layer learnable weight matrix with a dimension of 256 × 256, mapping node features from a unified latent representation dimension of 256 to a hidden layer dimension of 256; BN is batch normalization; and ReLU is the activation function. After the first-layer graph convolution, random deactivation is performed according to a preset deactivation ratio.

[0182] The convolution of the second layer graph is calculated according to equation (6):

[0183] H (2) = BN(A norm × H (1) × Θ1) (6)

[0184] Where Θ1 is the second-layer learnable weight matrix with a dimension of 256 × C, and C is the number of prognostic classification categories (C=2 in binary classification tasks). After the second-layer graph convolution, random deactivation is also performed according to a preset deactivation ratio.

[0185] H (2) The last row is the prognostic category logits of the target patient node, which, after softmax transformation, yields the PFS24 prognostic probability. This probability is a continuous value between 0 and 1, representing the predicted probability of the target patient's progression-free survival within 24 months.

[0186] PFS12 prognostic inference uses the same Z... all And A reconstructed according to PFS12 branch parameters (graph threshold 0.005, binary edge, inactivation ratio 0.5). normThe same forward propagation is performed using time-specific GNN model parameters Θ0' and Θ1'. The PFS12 output is a three-class logits, which are then softmaxed to obtain the probabilities of low-risk, intermediate-risk, and high-risk, respectively. The system classifies patients into the low-risk group based on a low-risk cutoff threshold of 0.1 and a high-risk cutoff threshold of 0.8 (probabilities ≥0.8 are high-risk, ≤0.1 are low-risk, and those in between are intermediate-risk).

[0187] VII. Quality Gating and Task Status Determination

[0188] like Figure 6 As shown, the quality gating and task state decision graph illustrates the definition of the necessary product set and the integrity check logic. After executing the graph neural network prognostic inference and outputting the prognostic result, the system enters the quality gating decision stage.

[0189] The system first obtains the required product set Q as required by the current prognostic analysis mode (PFS24 or PFS12). required For PFS24 mode, Q required Includes: standard lesion mask; lesion-level metabolic tumor volume list; patient-level total metabolic tumor volume; SUV radiomics feature vector; the aforementioned image feature vector, gene feature vector, and pathological feature vector; patient-to-patient similarity fusion matrix W. SNF ; PFS24 prognostic probability. For PFS12 mode, Q required The last item in the table is replaced with the PFS12 risk grouping result.

[0190] The system checks each necessary product to ensure it has been generated and successfully read. Each product's storage path and MD5 checksum are recorded upon generation; the check verifies file existence, readability, and checksum consistency. If Q... required If all products exist and are readable, the system will determine the task status as SUCCESS, and return the prognostic probability (or risk grouping) and result index in a package. If Q required If any product is missing or reading fails, the system will determine the task status as FAILED, recording the identifier of the missing product and the failure stage information—for example, if the standard lesion mask is missing, the failure stage is marked as "segmentation inference failure"; if the image feature vector is missing, the failure stage is marked as "multimodal construction - image modality missing"; if W SNF Missing or failed stages are marked as "graph construction failed". The system returns the missing product identifier and failure stage information, preventing incomplete prognostic results from being output.

[0191] VIII. Model Training and Parameter Fixation

[0192] The technical solution of this invention involves the training of two core models: a PET / CT lymphoma lesion segmentation model and a lymphoma multimodal prognostic prediction model.

[0193] Segmentation Model Training: The segmentation training subset uses a single PET / CT examination as the basic sample, including PET images, resampled CT images, SUV images consistent with PET space, and expert-annotated and reviewed lesion masks. The dataset contains 1014 preprocessed PET / CT examination samples, with volume data sizes ranging from 200×400×400 to 661×400×400, and a median size of 326×400×400. The training patch size is 128×128×128, and the batch size is 2.

[0194] The segmentation model is based on the nn-UNet 3D full-resolution segmentation pipeline, employing a 3D multi-channel input structure. The optimizer is SGD with an initial learning rate of 0.01, momentum of 0.95, and weight decay of 0.00003. The segmentation loss function consists of a weighted average of Dice loss, cross-entropy loss, small lesion constraint terms, and boundary constraint terms. Random pruning, scale perturbation, intensity perturbation, and flipping enhancement are used during training. Validation metrics include lesion Dice coefficient, lesion recall, false positive lesion count, MTV error, and small volume lesion recall. The checkpoint with the best overall performance on the validation set is used as the online inference model.

[0195] Prognostic model training: The prognostic training subset uses patients as the basic sample, including patient sample identifiers, standard features of imaging modality, standard features of genetic modality, standard features of pathological modality, PFS24 labels, PFS12 labels, follow-up time, and censoring status. The standard features of imaging modality have 1694 dimensions, the genetic modality has 55 dimensions, and the pathological modality has 22 dimensions.

[0196] The training, validation, and test sets are stratified according to patient level, with all data from the same patient entering only one set. The division ratio is 70% for the training set, 15% for the validation set, and 15% for the test set. A 50% patient-level cross-validation is used during the model selection phase.

[0197] The prognostic model training consists of two stages. The first stage trains the multimodal coding fusion module: the image encoder, gene encoder, and pathology encoder take the standardized features of their respective modalities as input and output a 256-dimensional latent representation, which is then optimized by the prognostic classification loss function after modality weight fusion. The second stage trains the GNN / GCN: the unified latent representation of the patients in the training set is first calculated according to equation (3) to form a historical base plate, and an SNF patient similarity matrix is ​​constructed on the base plate. An adjacency matrix is ​​generated according to the graph threshold. Then, a two-layer GCN is trained with the unified latent representation as node features and PFS labels as supervision signals. The PFS24 and PFS12 branches are trained separately, and each independently searches for the optimal graph threshold, inactivation ratio, and risk cutoff threshold.

[0198] Hyperparameter optimization employs a two-stage strategy, moving from coarse to fine: the first stage involves a grid search within a predefined search space, while the second stage refines the parameters locally around the optimal combination on the validation set. The final parameters are: hidden layer dimension 256, SNF nearest neighbor count K=6, fusion parameter μ=0.25; PFS24 branch graph threshold 0.65, inactivation ratio 0.75, threshold-weighted edges; PFS12 branch graph threshold 0.005, inactivation ratio 0.5, binary edges, low-risk truncation threshold 0.1, high-risk truncation threshold 0.8.

[0199] After training, the system permanently stores the following parameters: nn-UNet segmentation model parameters, multimodal encoding fusion module parameters (three encoders and modality weights), GNN / GCN model parameters (one set each for PFS24 and PFS12), historical patient latent representation base, feature field order, normalized statistics for each modality feature, SNF parameters (K and μ), graph threshold, and risk stratification threshold. During the online inference phase, the permanently stored parameters and preprocessing strategies are used identically to those used during training, ensuring that a single patient input resides in the same feature space as the historical patient base.

[0200] IX. Complete Implementation Examples

[0201] The following specific clinical scenario example illustrates the complete working process of the present invention.

[0202] A lymphoma patient completed a PET / CT scan. The DICOM data included PET sequences (5 beds) and CT sequences (whole-body plain scan). The patient also had a genetic testing report (covering 55 lymphoma-related gene loci) and a pathology report (including immunohistochemical markers and pathological images). After receiving the above data, the system generated a unified patient sample identifier PT-2026-0001 for the patient.

[0203] The data acquisition module first performs DICOM screening to identify PET and CT sequences. After converting the PET sequences into standard PET images, it calculates SUV standard uptake values ​​based on the injection dose (370 MBq), injection time (08:30), acquisition time (09:15), and patient weight (68 kg). CT images are resampled to PET space using trilinear interpolation. After nn-UNet segmentation and inference, a standard lesion mask is output—in this case, three lesion connected regions are identified. After spatial consistency verification, the MTV for each lesion is calculated to be 12.5 mL, 8.3 mL, and 3.7 mL, the TMTV is 24.5 mL, the total TLG is 156.8, and 1694-dimensional SUV radiomics features are extracted.

[0204] Gene modality data, after field mapping and numerical encoding, forms a 55-dimensional gene feature vector—in this example, TP53 mutation (coded as 1) and BCL2 overexpression (expression level encoded as a continuous value of 5.8) were detected. Pathological modality data, after structured encoding, forms a 22-dimensional pathological feature vector—Ki-67 is 75% (coded as 0.75), MYC is positive (coded as 1), and both pathological image encoding and text encoding results are included.

[0205] The three types of feature vectors, after being standardized by their respective normalized statistics, are input into the multimodal coding fusion module. The image encoder outputs h. radio =256 dimensions, gene encoder output h gene =256 dimensions, pathology encoder output h path =256 dimensions. Modal weights are α. radio =0.55, α gene =0.25, α path =0.20, and after fusion, a unified latent representation z=256 dimensions is obtained.

[0206] The unified potential representation z is added to the historical patient potential representation base Z stored in the historical base disk storage module. base (The dimensions are 812×256, corresponding to 812 patients in the training set), forming the Z-axis at the end. all The dimensions are 813×256. Based on Z... all The similarity fusion matrix among SNF patients was calculated with K=6 and μ=0.25, with a dimension of 813×813.

[0207] For PFS24 prediction: A graph threshold of 0.65, an inactivation rate of 0.75, and threshold-weighted edges were used to select 18 edges connected to the new patient. After the normalized adjacency matrix and node features were propagated through two layers of GCN, the PFS24 prognostic probability of this patient node was 0.72.

[0208] For PFS12 prediction: using a graph threshold of 0.005, an inactivation rate of 0.5, and constructing an adjacency matrix with binary edges, after time-specific GCN propagation, the PFS12 risk probability of this patient node is low risk 0.10, intermediate risk 0.15, and high risk 0.75. Based on the low-risk cutoff threshold of 0.1 and the high-risk cutoff threshold of 0.8, this patient is classified as intermediate risk.

[0209] The quality gating module checks the necessary product set: standard lesion mask, MTV of 3 lesions, TMTV = 24.5 mL, 1694-dimensional radiomics features, the aforementioned image feature vector, gene feature vector, and pathological feature vector, SNF matrix, PFS24 probability 0.72, and PFS12 grouping result "intermediate"—all products are complete and readable, and the task status is judged as SUCCESS. The system returns the above prognostic results and result index, and the entire end-to-end analysis process is completed.

[0210] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for multimodal prognostic analysis of lymphoma, characterized in that, Includes the following steps: Acquire imaging modality data, genetic modality data, and pathological modality data of the target patient, and generate a unified patient sample identifier; The image modality data, gene modality data, and pathology modality data are respectively input into the image encoder, gene encoder, and pathology encoder to obtain the latent representation of each modality. The latent representations of each modality are fused according to modality weights to form a unified latent representation of the target patient, which serves as the node feature of the graph neural network prognostic model. The unified potential representation of the target patient is added to the potential representation disk of historical patients. The potential representation disk of historical patients is composed of the unified potential representations of each historical patient in the training set. The unified potential representations of each historical patient and the unified potential representation of the target patient are in the same feature space. Based on the patient node feature set with the added unified potential representation of the target patient, a patient similarity fusion matrix is ​​calculated, and the patient similarity fusion matrix is ​​filtered according to a graph threshold to generate the adjacency matrix required by the graph neural network. Using the unified latent representation of each patient in the patient node feature set as node features, and the adjacency matrix as the graph structure, a graph neural network prognostic inference is performed to output the prognostic result of the target patient. The image encoder, gene encoder, pathology encoder, modality weights, and graph neural network prognostic model are all pre-trained and solidified using a training set; the historical patient latent representation disk is solidified after training and used to merge and construct a unified latent representation of new target patients during the online inference phase.

2. The method according to claim 1, characterized in that, The steps for acquiring the image modality data of the target patient include: Receive PET / CT DICOM data from the target patient and screen for effective PET and CT sequences; The PET sequence is converted into a standard PET image, and the standard PET image is converted into an SUV standard uptake value image based on the injection dose, acquisition time, and patient weight information. CT images are resampled into the PET image space to obtain resampled CT images; The PET / CT lymphoma lesion segmentation model based on nn-UNet is invoked, and the standard PET image and the resampled CT image are used as dual-channel inputs to perform sliding window depth segmentation inference and output standard lesion mask results. Spatial consistency verification is performed between the standard lesion mask and the standard SUV image. Three-dimensional connected component analysis was performed on the validated standard lesion mask to obtain the connected components of each lesion. Based on the number of voxels in each lesion's connected component and the physical voxel spacing of the SUV standard uptake value image, the lesion-level metabolic tumor volume was calculated. The total metabolic tumor volume at the patient level was obtained by summing the volumes of each lesion-level metabolic tumor. The SUV radiomics features were extracted using the SUV standard uptake value image and the standard lesion mask as input to obtain image modality data.

3. The method according to claim 2, characterized in that, After summarizing the metabolic tumor volumes of each lesion level to obtain the total metabolic tumor volume at the patient level, at least one of the following patient-level tumor burden indicators is also calculated: total glycolysis at the patient level, maximum SUV, average SUV, peak SUV, maximum lesion volume, and maximum distance between lesions. The SUV radiomics features are extracted by using the SUV standard uptake value image and the standard lesion mask as input after obtaining the patient-level total metabolic tumor volume and the patient-level tumor burden index.

4. The method according to claim 3, characterized in that, The volume of the lesion-level metabolic tumor is calculated as follows: MTV i = N i × Δx × Δy × Δz Among them, MTV i Let N be the metabolic tumor volume of the i-th lesion. i Let be the number of voxels in the connected region of the i-th lesion, and Δx, Δy, and Δz be the physical voxel spacing of the SUV standard uptake value image in the three spatial directions, respectively. The patient-level total metabolic tumor volume is the MTV of all connected lesions. i sum; Total glycolysis at the lesion level is calculated as follows: TLG i = MTV i × SUVmean i Among them, TLG i SUVmean represents the total glycolysis amount of the i-th lesion. i The average SUV standard uptake value of all voxels within the connected domain of the i-th lesion; The total glycolysis volume at the patient level is the total TLG of all lesion connected regions. i sum.

5. The method according to claim 1, characterized in that, Before inputting the image modality data, gene modality data, and pathology modality data into the image encoder, gene encoder, and pathology encoder, respectively, the method includes steps of field alignment and numerical standardization for each modality data. Image modal data are concatenated into image feature vectors according to a preset image field order. The image fields include patient-level total metabolic tumor volume, patient-level total glycolysis, maximum SUV value, average SUV value, peak SUV value, maximum distance between lesions, maximum lesion volume, and SUV radiomics characteristics. Missing fields are filled with zero values ​​and missing markers are recorded. Gene modality data are mapped according to the preset order of lymphoma-related gene fields. The mutation status, expression level, copy number variation and detection positive and negative of each gene field are numerically encoded. Undetected and missing fields are masked and filled with the numerical values ​​agreed upon by the model. Gene feature vectors are output. The pathological modality data is mapped according to the preset pathological field order, and the positive and negative values ​​and scoring levels of the immunohistochemical indicators of each pathological field are converted into numerical values. The pathological feature vector is output by combining the pathological image encoding results and the pathological text encoding results. The image feature vector, gene feature vector, and pathological feature vector are standardized using the same normalized statistics as in the training phase, so that the feature distribution of each encoder input is consistent with that in the training phase.

6. The method according to claim 1, characterized in that, The image encoder, gene encoder, and pathology encoder are multilayer fully connected neural networks with the same or different structures, each nonlinearly mapping the input features of its corresponding modality to a potential representation space of the same dimension. The modality weights are trainable scalar parameters, which, after Softmax normalization, satisfy the condition that each modality weight is non-negative and the sum of the weights is 1. The image encoder, gene encoder, pathology encoder, and modality weights are jointly optimized during the training phase with the prognostic classification supervision loss as the training objective, and are fixed after training.

7. The method according to claim 1, characterized in that, The potential representation base of historical patients is constructed in the training phase as follows: the image modality data, gene modality data and pathological modality data of each historical patient in the training set are respectively input into the solidified image encoder, gene encoder and pathological encoder, and fused into a unified potential representation of each historical patient according to the solidified modality weights. The unified potential representation of all historical patients forms the potential representation base of historical patients. During the online inference phase, the unified latent representation of the target patient is appended to the end of the latent representation base disk of the historical patients according to the sample index, forming an expanded patient node feature set. The historical patient portion of the expanded patient node feature set retains its original index and feature value, while the target patient participates in subsequent graph construction and inference as a new node.

8. The method according to claim 1, characterized in that, The calculation of the patient similarity fusion matrix adopts a similarity network fusion algorithm. The similarity network fusion algorithm calculates a multi-kernel similarity matrix based on the similarity between the features of each node in the patient node feature set and performs iterative fusion to obtain the fused patient similarity fusion matrix. The step of filtering the patient similarity fusion matrix according to the graph threshold to generate the adjacency matrix required by the graph neural network includes: retaining edges in the patient similarity fusion matrix with similarity values ​​greater than or equal to the graph threshold, and setting edges with similarity values ​​less than the graph threshold to zero, to obtain a weighted adjacency matrix; or, setting edges in the patient similarity fusion matrix with similarity values ​​greater than or equal to the graph threshold to 1, and setting edges with similarity values ​​less than the graph threshold to 0, to obtain a binary adjacency matrix; The adjacency matrix, after row normalization, serves as the normalized adjacency matrix required for graph neural network propagation.

9. The method according to claim 1, characterized in that, The prognostic results include the PFS24 prognostic probability; the imaging modal data, genetic modal data and pathological modal data of the target patients are encoded and fused and used for PFS12 prognostic inference, outputting PFS12 risk grouping results; The PFS24 prognostic inference and the PFS12 prognostic inference respectively employ time-specific graph neural network model parameters, graph thresholds, inactivation ratios, and risk truncation thresholds; the graph neural network model parameters, graph thresholds, inactivation ratios, and risk truncation thresholds are independently optimized and fixed for the two time periods of PFS24 and PFS12 through a validation set during the training phase; The PFS24 prognostic probability is a continuous probability value; the PFS12 risk grouping results divide patients into three groups: low-risk, intermediate-risk, and high-risk, based on low-risk cutoff thresholds and high-risk cutoff thresholds.

10. A multimodal prognostic analysis system for lymphoma, characterized in that, include: The data acquisition module is used to acquire imaging modality data, genetic modality data, and pathological modality data of the target patient, and generate a unified patient sample identifier; The multimodal coding fusion module is used to input the image modal data, gene modal data and pathological modal data into the image encoder, gene encoder and pathological encoder respectively to obtain the potential representation of each modality, and fuse the potential representation of each modality according to the modal weight to form a unified potential representation of the target patient, which serves as the node feature of the graph neural network prognostic model. The historical patient potential representation module is used to store the historical patient potential representation base that is solidified after training. The historical patient potential representation base consists of the unified potential representation of each historical patient in the training set. The unified potential representation of each historical patient and the unified potential representation of the target patient are in the same feature space. The graph construction module is used to add the unified latent representation of the target patient to the historical patient latent representation base, calculate the patient similarity fusion matrix based on the patient node feature set with the added unified latent representation of the target patient, filter the patient similarity fusion matrix according to the graph threshold, and generate the adjacency matrix required by the graph neural network. The graph neural network prognostic reasoning module is used to perform graph neural network prognostic reasoning with the unified latent representation of each patient in the patient node feature set as node features and the adjacency matrix as the graph structure, and output the prognostic result of the target patient. The image encoder, gene encoder, pathology encoder, modality weights, and graph neural network prognostic model are all pre-trained and solidified using a training set; the historical patient latent representation disk is solidified after training and used to merge and construct a unified latent representation of new target patients during the online inference phase.

Images