Glioma molecular typing prediction method and device based on three-dimensional MRI and clinical text

Abstract

This invention discloses a method and device for predicting molecular subtyping of gliomas based on 3D MRI and clinical text. The method includes: a 3D visual encoder that divides standardized 3D volume data into patches, which are then linearly projected and positionally encoded before being input into the encoder, outputting an image feature sequence; a clinical text encoder that encodes clinical information into clinical semantic vectors; a cross-modal fusion module that uses the image feature sequence as the query and the clinical semantic vector as the key and value, calculating fusion features through cross-attention; and a multi-task prediction head containing three MLP classifiers that predict IDH mutation, 1p / 19q co-deletion, and MGMT methylation status, respectively. The cross-modal fusion module further includes a task-specific gating network and a modality quality assessment unit. The former generates task-specific fusion weights for each prediction task, while the latter dynamically adjusts the weights based on data quality and cross-modal divergence, achieving task-case dual adaptive fusion and improving prediction accuracy and robustness.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing and artificial intelligence, specifically involving a multimodal medical image analysis method based on three-dimensional vision Transformer and cross-modal attention mechanism, which is particularly suitable for non-invasive prediction of molecular subtyping of gliomas. Background Technology

[0002] 1. Limitations of existing detection methods for molecular markers of glioma

[0003] IDH gene mutation is an early driving event in the development and progression of gliomas, and patients with IDH-mutant gliomas have a significantly better median survival than those with IDH wild-type gliomas. 1p / 19q co-deletion is a molecular diagnostic marker for oligodendrogliomas, and patients with oligodendrogliomas carrying this co-deletion have higher sensitivity to combined chemotherapy and radiotherapy. MGMT promoter methylation status is a key biomarker for assessing the sensitivity to temozolomide (TMZ) chemotherapy; patients with positive MGMT methylation have a significantly improved prognosis after receiving TMZ chemotherapy.

[0004] Currently, the acquisition of these molecular biomarkers mainly relies on gene testing or immunohistochemical analysis of postoperative tissue samples. These methods have the following inherent limitations: first, the invasive procedures introduce additional surgical risks and potential complications for patients; second, the test results depend on the quality and heterogeneity of the tissue sample, and spatial heterogeneity within the tumor may lead to sampling bias; third, the testing cycle is long (usually requiring several days to several weeks), delaying the timely development of individualized postoperative treatment plans; and fourth, the testing costs are high, limiting their widespread application in areas with limited medical resources. Therefore, non-invasive predictive methods based on preoperative multimodal MRI images have significant clinical value.

[0005] 2. Existing deep learning-based prediction methods and their limitations

[0006] In recent years, deep learning-based radiomics methods have made progress in the prediction of molecular markers for gliomas. However, existing methods generally suffer from the following technical limitations:

[0007] (1) Two-dimensional slice input lacks the three-dimensional spatial continuity and overall topological structure of the tumor.

[0008] When 3D MRI volume data is decomposed into independent 2D slices for processing, the model cannot obtain spatial continuity information along the Z-axis (between slices). Gliomas, as invasive tumors that grow irregularly in 3D space, contain important molecular phenotypic information in their spatial morphology, edema extent, and 3D topological relationship with surrounding brain tissue. The 2D slice processing method disrupts this 3D spatial continuity, causing the model to lose overall tumor morphological descriptors (such as 3D shape features, spatial lateralization, spatial distribution of multicentric lesions, etc.), limiting further improvement in predictive performance.

[0009] (2) The limitation of local receptive field in traditional CNNs leads to the loss of global contextual information.

[0010] Traditional CNNs expand the receptive field by stacking convolutional and pooling layers, but due to the sliding operation characteristics of local convolutional kernels, a single neuron on a deep feature map can only perceive a limited region in the input space. For glioma molecular subtyping prediction tasks, the mapping relationship between the tumor's molecular phenotype and its macroscopic imaging manifestations involves multi-scale, long-range spatial dependencies: for example, IDH-mutant gliomas typically present as well-defined, low-grade tumors, but their fluorodeoxyglucose uptake patterns, calcification distribution, and cortical involvement may be located in distant regions of the tumor body. Traditional CNNs struggle to effectively model such long-range spatial dependencies, leading to the loss of global contextual information.

[0011] (3) Single molecular marker prediction models do not utilize the biological associations between molecular markers.

[0012] Existing studies often model the prediction of IDH mutation status, 1p / 19q co-deletion status, and MGMT methylation status as three separate binary tasks, neglecting the intrinsic biological associations among these three molecular markers. In fact, according to the WHO Classification of Tumors of the Central Nervous System (2021), IDH mutation status and 1p / 19q co-deletion status jointly define the molecular diagnosis of oligodendroglioma (IDH mutation and 1p / 19q co-deletion); IDH mutation status and MGMT methylation interact in the prognostic assessment of glioblastoma. Separating multi-marker prediction into independent tasks not only fails to leverage the correlation constraints between markers to improve the accuracy of individual predictions but also increases the complexity of model deployment and clinical application.

[0013] (4) The simple splicing and fusion of imaging and clinical data lacks a deep semantic alignment mechanism.

[0014] Existing methods for fusing image features and patient clinical features often employ simple vector concatenation or weighted summation operations. This shallow fusion approach assumes that image features and clinical features have comparable semantic scales and distribution characteristics within the same latent representation space. However, in reality, they are heterogeneous data types with significantly different data distribution structures and semantic granularities. Fusion strategies lacking deep semantic alignment mechanisms struggle to achieve effective interaction between image spatial features and clinical semantic features, limiting the full utilization of cross-modal complementary information.

[0015] (5) The fixed fusion strategy did not take into account the differences in prediction tasks and differences in case data quality.

[0016] Furthermore, existing multimodal fusion methods employ the same fixed fusion strategy for all prediction tasks and all cases. On the one hand, there are significant differences in the association strength between the three molecular markers—IDH mutation, 1p / 19q co-deletion, and MGMT promoter methylation—and imaging phenotypes and clinical features: IDH mutation is associated with factors such as overall tumor metabolism and age; 1p / 19q co-deletion is mainly associated with image texture in specific regions; and MGMT promoter methylation is more strongly associated with clinical factors such as treatment history and age. A uniform fusion weight cannot match the differentiated image-clinical dependence characteristics of each marker. On the other hand, the image data quality (such as motion artifacts, signal-to-noise ratio, and tumor visibility) and the completeness of clinical information vary significantly among different cases. A fixed fusion strategy cannot adaptively adjust modal dependence when the input data quality deteriorates, resulting in a significant decrease in the predictive reliability of the model on low-quality data, and lacking quantitative assessment and early warning methods for the credibility of prediction results.

[0017] In summary, existing deep learning-based methods for predicting molecular biomarkers of gliomas have shortcomings in four aspects: 3D spatial modeling capability, global context awareness capability, multi-task collaboration capability, and cross-modal deep fusion capability. Summary of the Invention

[0018] In view of the above-mentioned defects and shortcomings of the existing technology, the present invention aims to provide a molecular subtyping prediction method for gliomas based on three-dimensional MRI and clinical text, so as to solve the following technical problems:

[0019] (1) Problem of loss of three-dimensional spatial information: Existing methods decompose three-dimensional MRI volume data into two-dimensional slices, which breaks the three-dimensional spatial continuity and overall topological structure of the tumor and loses spatial morphological information along the Z-axis.

[0020] (2) Global context missing problem: Traditional CNNs are limited by local receptive fields and have difficulty modeling the long-distance spatial dependence between the molecular phenotype of glioma and macroscopic imaging manifestations;

[0021] (3) Multi-marker prediction of the fragmentation problem: Existing studies model IDH mutation, 1p / 19q co-deletion and MGMT methylation as independent tasks, without utilizing the biological association between the three markers for synergistic optimization;

[0022] (4) Shallow problem of cross-modal fusion: The fusion of imaging features and clinical features often adopts simple splicing or weighted summation, lacking a deep semantic alignment mechanism, and cannot achieve effective interaction of heterogeneous data.

[0023] (5) Problems of fixed fusion strategy and quality oversight: Existing methods use a fixed fusion strategy for all prediction tasks and all cases. They do not consider the differences in the dependence of different molecular markers on imaging modalities and clinical modalities, nor do they consider the differences in the quality of input data for different cases. They lack an adaptive fusion mechanism based on data quality assessment and prediction uncertainty, as well as a prediction reliability warning mechanism.

[0024] This invention relates to a molecular subtyping prediction method for gliomas based on 3D MRI and clinical text, characterized by the following steps:

[0025] ( The system acquires preoperative MRI images and clinical structured information of the patient to be predicted, and inputs the preoperative MRI images and clinical structured information into the prediction system; the prediction system includes: a data preprocessing module, a three-dimensional visual encoder, a clinical text encoder, a cross-modal fusion module, and a multi-task prediction head;

[0026] The data preprocessing module is used to perform rigid registration, N4 bias field correction, Z-Score normalization, and 3D nnU-Net-based tumor region segmentation on the preoperative MRI images. Based on the tumor region segmentation results output by 3D nnU-Net, it calculates the minimum bounding box surrounding the tumor region and expands it outward by a predetermined number of voxels along three spatial dimensions to obtain standardized three-dimensional volume data. It also converts the clinical structured information into a text sequence using a natural language template.

[0027] A 3D visual encoder, including a Transformer encoder, is used to uniformly divide the standardized 3D volume data into non-overlapping 3D patches along three spatial dimensions. Each patch is linearly projected onto an embedding space and then element-wise added to a learnable position encoding vector to obtain a patch embedding vector sequence. The index of the learnable position encoding vector is determined by the patch's coordinates in the 3D mesh. Through formula The coordinates are uniquely determined, where x, y, and z are the grid coordinates of the patch in the width, height, and depth directions, respectively. , The total number of patches in the width and height directions are respectively; the patch embedding vector sequence is input into the Transformer encoder, and the image feature sequence is output;

[0028] A clinical text encoder, employing a pre-trained language model, is used to encode the text sequence and output a clinical semantic vector.

[0029] The cross-modal fusion module is used to calculate cross-attention using the result of linear projection of the image feature sequence as the query and the clinical semantic vector as the key and value, to obtain the modulated feature sequence; the modulated feature sequence is residually connected with the linearly projected image feature sequence, and after layer normalization and feedforward network processing, residual connection and layer normalization are performed again, and then the fused feature vector is obtained through global average pooling;

[0030] The multi-task prediction head contains three parallel prediction branches, which are used to synchronously output the prediction results of IDH mutation state, 1p / 19q co-deletion state and MGMT promoter methylation state based on the fusion feature vector.

[0031] The prediction system is obtained through the following training method: The prediction system is trained using a multi-task loss function, which is a weighted sum of the losses of each task, where the weights of each task are... Calculate, where α=1.0, β=0.5, γ=0.3, For inverse frequency weights based on sample imbalance, To validate the dynamic weights based on the trend of loss decay, Clinical prior weights;

[0032] ( The prediction system synchronously outputs the prediction results of IDH mutation status, 1p / 19q co-deletion status and MGMT promoter methylation status.

[0033] Furthermore, the aforementioned clinical prior weights Based on the WHO 2021 classification criteria for tumors of the central nervous system, the prediction scores for IDH mutation status were set to 0.40, 1p / 19q co-deletion status to 0.35, and MGMT promoter methylation status to 0.25.

[0034] Furthermore, the Transformer encoder consists of 12 sequentially stacked coding layers;

[0035] Each coding layer includes: a first normalized sublayer, a multi-head self-attention sublayer, a first residual adder, a second normalized sublayer, a feedforward network sublayer, and a second residual adder;

[0036] The connection relationships within each coding layer are as follows:

[0037] The input of the current layer is processed sequentially by the first normalization sublayer and the multi-head self-attention sublayer to obtain the attention output;

[0038] The first residual adder adds the attention output to the input of the current layer to obtain a first intermediate result;

[0039] The first intermediate result is processed sequentially by the second normalization sub-layer and the feedforward network sub-layer to obtain the feedforward output;

[0040] The second residual adder adds the feedforward output to the first intermediate result to obtain the output of the current layer;

[0041] The embedding dimension is 768, the number of attention heads in the multi-head self-attention sub-layer is 12, the dimension of each head is 64, and the hidden layer dimension of the feedforward network sub-layer is 3072, with GELU as the activation function.

[0042] The output of the 12th coding layer is the image feature sequence, with a dimension of 512×768.

[0043] Furthermore, the cross-modal fusion module includes: a query projection sublayer, a key-value projection sublayer, a single-head cross-attention sublayer, a first residual adder, a first normalization sublayer, a feedforward network sublayer, a second residual adder, a second normalization sublayer, a global average pooling sublayer, and a linear projection sublayer.

[0044] The connection relationships between the sub-layers are as follows:

[0045] (a) The image feature sequence F img Learnable projection matrix W proj and bias b proj Linear projection yields the projected image features F. img ′;

[0046] (b) The clinical semantic vector f clin It is expanded into a two-dimensional tensor of length 1, which serves as the input for the key and value.

[0047] (c) The single-headed cross-attention sublayer uses the projected image feature F img After querying the projection matrix W Q and bias b Q The result after linear transformation is the query, which is the clinical semantic vector projected onto the key matrix W. K and bias b K The result of the linear transformation is the key and the longitude projection matrix W.V and bias b V The result after the linear transformation is the value, Value. Cross-attention is then calculated, with a scaling factor of [value missing]. With d=768, we obtain the cross-attention output. ;

[0048] (d) The first residual adder outputs the cross-attention statement. With the projected image feature F img The residual fusion features are obtained by adding them together. This is used to prevent modality collapse caused by excessive dominance of clinical information;

[0049] (e) The residual fusion feature After being processed sequentially by the first normalization sublayer and the feedforward network sublayer, a feedforward output is obtained. The second residual adder adds the feedforward output to the input of the first normalization sublayer, and then processes it again by the second normalization sublayer to obtain the normalized output. ;

[0050] (f) The global average pooling sublayer averages the normalized output Fout along the sequence dimension to obtain the fused feature vector. ;

[0051] (g) The linear projection sublayer will fuse the feature vector via projection matrix and bias vector Linear projection yields a fused feature vector z, which serves as the unified input to the multi-task prediction head;

[0052] The hidden layer of the feedforward network sub-layer has a dimension of 3072 and the activation function is GELU.

[0053] Furthermore, the cross-modal fusion module also includes a task-specific gated fusion submodule, which comprises a task embedding layer and three parallel task gating networks. The task embedding layer generates learnable task embedding vectors for the IDH mutation state prediction task, the 1p / 19q co-deletion state prediction task, and the MGMT promoter methylation state prediction task, respectively. Each task gating network takes the concatenation result of the fusion feature vector and the corresponding task embedding vector as input, outputs the two-dimensional modal weights of the corresponding task through a gated perceptron, and obtains the task-specific modal weights through softmax normalization. The image modality weights and clinical modality weights are calculated. The projected image features and the clinical semantic vectors are weighted and combined according to the modality weights specific to each task to obtain three task-specific fusion feature vectors, which are then input into the corresponding prediction branches. The initial weights of the task-gated network are set differently according to the biological characteristics of each molecular marker: the IDH mutation state prediction task is initialized in a balanced manner, the image modality weight is higher than the clinical modality weight for the 1p / 19q co-deletion state prediction task, and the clinical modality weight is higher than the image modality weight for the MGMT promoter methylation state prediction task. Moreover, the gating weights can be learned end-to-end during training.

[0054] Furthermore, the task-specific gating fusion submodule also includes a modality quality assessment unit and an uncertainty weighting unit. The modality quality assessment unit calculates the image modality quality score and the clinical modality quality score respectively, and calculates the symmetric KL divergence between the image branch prediction probability and the clinical branch prediction probability as the cross-modality divergence degree. The uncertainty weighting unit dynamically corrects the modality weights output by each task gating network on a case-by-case basis in a manner that is positively correlated with the modality quality score and negatively correlated with the prediction uncertainty. When the cross-modality divergence degree exceeds a preset divergence threshold and both modality quality scores are lower than the preset quality threshold, a prompt message indicating that the prediction is unreliable and suggesting supplementary examinations is output.

[0055] Furthermore, the three parallel prediction branches are the IDH mutation state prediction branch, the 1p / 19q co-deletion state prediction branch, and the MGMT promoter methylation state prediction branch, respectively. Each branch is an independent MLP classifier. The MLP classifier is a four-layer fully connected network, with dimensions of 512-dimensional → 256-dimensional → 128-dimensional → 64-dimensional → 1-dimensional. The activation function of the first three layers is ReLU, the output layer is Sigmoid, and the dropout rates of each layer are 0.3, 0.2, and 0.2, respectively.

[0056] Furthermore, the loss for each task adopts focus loss, with focus parameters... And introduce a reweighting method based on the number of valid samples for each category, with reweighting coefficients. ,in β=0.9999 is the number of original samples in class c.

[0057] Furthermore, the tumor region segmentation based on 3D nnU-Net employs a dynamic network topology adaptation mechanism, with the network depth L determined by the voxel spacing Δ using the formula... Automatic inference, among which The loss function of the 3D nnU-Net is:

[0058] Where Dice loss is defined as:

[0059]

[0060] Cross-entropy loss is defined as:

[0061]

[0062] in: This is a probability graph for network prediction. The actual segmentation mask is manually labeled. and The first True label and predicted probability of category The number of segmentation categories; As a smoothing factor, to avoid the denominator being zero; These are the weighting coefficients for the cross-entropy loss. It is the set of all voxels within the image domain.

[0063] Furthermore, the clinical text encoder uses the BERT model, whose pre-trained weights are frozen during training, and the parameters are efficiently fine-tuned through low-rank adaptation LoRA, with LoRA rank r=8.

[0064] Furthermore, the multi-task loss function is adjusted according to the following parameters after each training iteration: The weights of each task are subject to normalization constraints, where Let be the weight of the t-th task in the e-th round.

[0065] Furthermore, the prediction system is trained using a five-fold patient-level hierarchical cross-validation, with the training set, validation set, and test set allocated in a 60%:20%:20% ratio, and all data from the same patient grouped into the same fold. During training, online data augmentation is performed using random horizontal flipping, random vertical flipping, random depth flipping, random rotation, random intensity scaling, random intensity shifting, and Gaussian noise.

[0066] The present invention also discloses an electronic device for predicting molecular subtyping of gliomas based on three-dimensional MRI and clinical text, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method for predicting molecular subtyping of gliomas based on three-dimensional MRI and clinical text.

[0067] The beneficial effects of this invention are:

[0068] (1) Complete preservation of three-dimensional spatial topological information

[0069] Existing methods segment 3D MRI volume data along the axial direction into two-dimensional slices for independent processing, losing information about the spatial continuity of the tumor along the Z-axis (between slices). This invention employs a 3D visual encoder (including a Transformer encoder) to directly divide standardized 3D volume data into 3D patches, and assigns a unique 3D spatial location identifier to each patch through 3D learnable positional coding. This mechanism enables the model to perceive the complete spatial topology of the tumor in the three dimensions of width, height, and depth, including the three-dimensional distribution of edema zones, the 3D positional relationships of multicentric lesions, and the infiltration morphology along white matter fiber bundles, thereby significantly improving the modeling ability for 3D morphological features such as IDH-mutant gliomas.

[0070] (2) Cross-modal deep semantic alignment and personalized feature modulation

[0071] Existing technologies, when fusing image and clinical features, employ simple vector concatenation or weighted summation operations, failing to achieve deep interaction between image features and clinical semantics. This results in clinical information participating in the fusion only as a global bias, unable to achieve point-by-point differentiated modulation of image spatial locations. This invention, through a directed cross-attention mechanism, explicitly uses the image feature sequence as the query and the clinical semantic vector as the key and value, enabling the clinical semantic vector to independently weight the response to the query at each image spatial location. This mechanism achieves personalized feature modulation with "different strategies for the same image": for patients with similar imaging presentations but different clinical backgrounds, the model can assign different attention weights to the same image region based on clinical semantic differences, thereby generating differentiated fused features. This capability is not available in existing simple concatenation or weighted summation methods.

[0072] (3) Joint prediction and synergistic optimization of multi-molecular biomarkers

[0073] Existing studies often model the prediction of IDH mutation status, 1p / 19q co-deletion status, and MGMT promoter methylation status as three independent tasks, neglecting the intrinsic biological connections between the three biomarkers. This invention integrates the prediction of these three biomarkers into a unified multi-task learning framework, sharing the joint representation of a 3D visual encoder, a clinical text encoder, and a cross-modal fusion module. Through shared network parameters and gradient backpropagation, it achieves information complementarity and collaborative optimization among the biomarkers.

[0074] Experimental results show that, compared with three independent task models (3D ViT single task image only), the multi-task learning framework can improve performance by utilizing the synergistic gain effect of biological associations between molecular markers, thus verifying the effectiveness of multi-task joint training.

[0075] (4) Adaptive weighted loss based on WHO 2021 clinical prior

[0076] Existing multi-task loss weighting strategies do not consider the differences in diagnostic decision-making hierarchy among tasks in the WHO 2021 classification of central nervous system tumors. This invention designs a three-factor adaptive weighted loss function. The model employs a three-factor joint regulation mechanism, combining the inverse frequency weight of sample imbalance, the dynamic weight of validation loss decay trend, and the WHO 2021 clinical prior weight. This mechanism alleviates sample imbalance while introducing prior knowledge of clinical diagnostic decisions, enabling the model to prioritize learning the features most discriminative for clinical subtyping during training, thus improving the clinical usability of the prediction results.

[0077] (5) End-to-end fully automated processing

[0078] This invention starts with raw multi-sequence MRI images and clinical structured information, and through data preprocessing, 3D visual encoding, cross-modal fusion, and multi-task prediction, achieves end-to-end fully automated processing without the need for manual design of radiomics features or manual selection of regions of interest. Compared to traditional gene detection methods that rely on postoperative tissue samples, this invention can simultaneously predict three molecular markers in a short time before surgery, significantly reducing waiting time.

[0079] (6) Improved accuracy and robustness brought about by task-case dual adaptive fusion

[0080] Existing methods use the same fusion strategy for all prediction tasks and all cases. This invention proposes a task-case dual adaptive fusion mechanism: a task-specific gating network selects differentiated fusion paradigms based on the biological characteristics of different molecular markers, achieving "gene-specific" adaptive fusion strategy; a modality quality assessment and uncertainty weighting unit dynamically adjusts modality weights on a case-by-case basis based on input data quality, achieving "case-specific" adaptive fusion weights. When motion artifacts are present in the images or the tumor volume is too small, the weight of clinical information is automatically increased; when clinical information is missing or logically contradictory, the weight of images is automatically increased; when the two modal predictions are severely divergent and both data quality are poor, a warning message of "prediction uncertain, supplementary examination recommended" is output, rather than forcibly outputting a potentially incorrect classification result. This mechanism enables the prediction system to act like a senior radiologist, clearly defining when to rely on images, when to rely on medical history, and when to give an uncertain conclusion, significantly improving the accuracy of molecular subtyping prediction, robustness under low-quality data, and interpretability of clinical decisions. Attached Figure Description

[0081] Figure 1 This is a diagram of the overall architecture of the glioma molecular subtyping prediction system. Detailed Implementation

[0082] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0083] The molecular subtyping prediction method for gliomas based on 3D MRI and clinical text in this embodiment includes the following steps:

[0084] ( The system acquires preoperative MRI images and clinical structured information of the patient to be predicted, and inputs the preoperative MRI images and clinical structured information into the prediction system.

[0085] Preoperative MRI imaging data is typically obtained from the hospital's PACS (Picture Archiving and Communication System). The specific MRI sequences acquired include four sequences: T1-weighted images (T1), T1-contrast-enhanced images (T1ce), T2-weighted images (T2), and Fluid Attenuation Inversion Recovery (FLAIR) sequences. The image data format is standardized three-dimensional volumetric data. To ensure the integrity of the three-dimensional spatial context information, no slicing-level processing (i.e., dimensionality reduction operation of decomposing three-dimensional volumetric data into two-dimensional slice sequences) is performed on the original volumetric data; instead, the complete spatial structure is preserved.

[0086] The patient's structured clinical information includes the patient's age, gender, lesion location, clinical symptoms, disease course, Karnofsky functional status score, and treatment history.

[0087] The prediction system includes: a data preprocessing module, a 3D visual encoder, a clinical text encoder, a cross-modal fusion module, and a multi-task prediction head.

[0088] (i) Data preprocessing module, used to perform rigid registration, N4 bias field correction, Z-Score normalization, and 3D nnU-Net-based tumor region segmentation on the preoperative MRI images; calculate the minimum bounding box surrounding the tumor region based on the tumor region segmentation results output by 3D nnU-Net, and expand the bounding box outward by a predetermined number of voxels along three spatial dimensions to obtain standardized three-dimensional volume data; and used to convert the clinical structured information into a text sequence using a natural language template.

[0089] 1.1) Image Preprocessing Branch

[0090] (a) Rigid registration

[0091] Input: The input is the raw multi-sequence MRI images, including T1 sequence I T1 (X), T1CE sequence I T1CE (X), T2 sequence I T2 (X) and FLAIR sequence I FLAIR (X), where X=(x,y,z) represents the three-dimensional spatial coordinates, and x, y, z are the voxel coordinates in the width, height, and depth directions, respectively. Due to differences in scanning parameters, different sequences may exhibit spatial position offsets and resolution inconsistencies.

[0092] Processing procedure: Using the T1 contrast-enhanced image (T1CE) sequence as the reference modality, the other three sequences (T1, T2, and FLAIR) are registered to the unified spatial coordinate system of the T1CE sequence. The reason for choosing the T1CE sequence as the reference is that the T1CE sequence enhances the contrast of the tumor boundary and the blood-brain barrier disruption area through contrast agent injection, providing the clearest display of tumor structure and offering the most accurate spatial benchmark for subsequent segmentation and feature extraction.

[0093] For the sequence to be registered Find the optimal transformation To maximize the statistical similarity between the transformed image and the reference mode:

[0094]

[0095] in: Indicates the number to be registered A sequence of images, Represents the reference mode (T1CE) image. It is a rigid transformation, involving a three-dimensional rotation matrix. Translation vector , This represents the image similarity measurement function.

[0096] In multimodal registration scenarios (where there are significant differences in grayscale distribution between different MRI sequences), Normalized Mutual Information (NMI) is used as a similarity measure. Two random variables... and The normalized mutual information between them is defined as:

[0097]

[0098] in For mutual information, and Images and marginal entropy The joint entropy is given. NMI is invariant to nonlinear transformations of image grayscale distribution and is suitable for multimodal medical image registration.

[0099] Registration optimization employs gradient descent, iteratively updating the transformation parameter μ=(θ) x ,θ y ,θ z ,t x ,t y ,t z ), where θ x ,θ y ,θ z Let t be the rotation angle about the three coordinate axes. x ,t y ,t z These represent the translation amounts in three directions. The iteration termination condition is: the change in NMI between two consecutive iterations is less than a preset threshold (e.g., ...). (or reach the maximum number of iterations (e.g., 200 times)).

[0100] After registration, the floating image is transformed by the optimal transform T. m ∗ The image is mapped to a fixed image space, and the gray values ​​of each voxel after transformation are calculated using trilinear interpolation to obtain a registered image aligned with the T1CE space.

[0101] Output: Registered four-sequence images , , , All of them are located in the same spatial coordinate system of the T1CE sequence and have the same spatial resolution and image size.

[0102] b) N4 bias field correction

[0103] N4 bias field correction was performed on each sequence of MRI images after rigid registration. This was used to eliminate low-frequency intensity distortion (bias field effect) caused by the non-uniformity of the radio frequency coil transmission / reception field during MRI imaging, so that the gray values ​​of the same tissue tend to be consistent in different spatial locations, providing a more uniform image basis for subsequent normalization and segmentation operations.

[0104] The input for N4 bias field correction is a rigidly registered four-sequence image: , , , ,use This refers to any one of the four image sequences mentioned above, with all four sequences undergoing the same correction process independently and in parallel. The observed signal intensity of the input image is denoted as... The mathematical model for the bias field is defined as follows:

[0105]

[0106] in, The signal intensity observed at voxel position x, i.e., the input registered image. ; The actual signal strength of the organization (to be recovered); For a slowly varying bias field (multiplicative distortion factor, B(x)>0); It is additive Gaussian noise. .

[0107] To estimate the bias field First, a logarithmic transformation is performed on the observation model to convert the multiplicative model into an additive model:

[0108]

[0109] in This is the transformed noise term. The N4 algorithm will... Modeled as a linear combination of B-spline basis functions:

[0110]

[0111] in, For cubic B-spline basis functions, For the location of control points, The control coefficients to be optimized are: To control the spacing between points, This represents the total number of control points.

[0112] The N4 algorithm estimates the bias field through the following iterative process:

[0113] Step 1: Initialize bias field estimation .

[0114] Step 2: In the t-th iteration, correct the image based on the current bias field estimate:

[0115]

[0116] Step 3: Perform soft-thresholding segmentation on the corrected image and estimate the tissue category probability distribution. Assume there are C tissue categories, and calculate the posterior probability. , .

[0117] Step 4: Update the B-spline control coefficients using weighted least squares based on the organization category probability. This maximizes the expected log-likelihood function:

[0118]

[0119] in is the logarithmic gray mean of the c-th type of organization.

[0120] Step 5: Reconstruct the bias field estimate based on the updated control coefficients. If the change in the bias field estimate is less than the convergence threshold... Or reach the maximum number of iterations If the iteration fails, stop; otherwise, return to step 2.

[0121] Output: Image sequences after bias field correction , , , Low-frequency intensity distortion has been effectively suppressed, and the gray values ​​of the same tissue tend to be consistent.

[0122] (c) Z-Score normalization

[0123] Z-Score normalization was performed on each sequence of MRI images after N4 bias field correction to eliminate differences in image grayscale distribution caused by different scanning equipment, scanning parameters and individual patient factors.

[0124] The input for Z-Score normalization is the four-sequence image after N4 bias field correction: , , , ,use This refers to any one of the four sequences mentioned above, with the four sequences undergoing the same normalization process independently and in parallel.

[0125] Unlike global normalization, this embodiment adopts a local normalization strategy based on the brain parenchyma mask region, that is, only the gray-scale statistics of the brain parenchyma (gray matter + white matter) region are calculated to exclude the interference of cerebrospinal fluid (CSF), skull and background regions.

[0126] First, brain parenchyma masks are obtained through thresholding and morphological manipulation. ,in Voxel representation It belongs to the brain parenchyma region.

[0127] Then, the gray-scale mean is calculated within the brain parenchyma mask region. and standard deviation :

[0128]

[0129]

[0130] in, A collection of voxels representing brain parenchyma regions. This represents the total number of brain parenchyma voxels.

[0131] Finally, Z-Score normalization is performed on each voxel:

[0132]

[0133] This standardization method ensures that the output image meets the requirements. , This means that the gray values ​​in the brain parenchyma region follow a standard normal distribution with zero mean and unit variance.

[0134] The reasons for choosing the brain parenchyma region as the statistical reference region are: (1) The brain parenchyma is the anatomical carrier of tumor development, and its gray-scale statistical characteristics are more comparable to those of the tumor region; (2) Excluding CSF and background regions can avoid the downward shift of low gray-scale regions to the statistics, ensuring that the normalized gray-scale values ​​of the tumor region have a stable dynamic range; (3) Standardized input distribution helps the training stability and convergence speed of the deep learning model.

[0135] Output: Z-Score normalized images of each sequence , , , The mean gray value in the brain parenchyma region is 0, and the standard deviation is 1.

[0136] (d) Tumor region segmentation based on 3D nnU-Net

[0137] Automatic tumor region segmentation based on 3D nnU-Net was performed on four-sequence MRI images after Z-Score normalization to obtain the precise spatial localization of the tumor region, providing a basis for subsequent bounding box calculation and cropping.

[0138] The input for 3D nnU-Net segmentation is four-sequence images normalized by Z-Score: , , , ,use This refers to any one of the four sequences mentioned above, and all four sequences are fed into the 3D nnU-Net network as multi-channel inputs.

[0139] Automatic tumor region segmentation is achieved using the 3D nnU-Net (no-new-Net) framework. The core innovation of nnU-Net lies in its Dynamic Network Topology Adaptation mechanism, which automatically infers the optimal network architecture and training hyperparameters based on the characteristics of the input data, eliminating the need for manual parameter tuning. The Dynamic Network Topology Adaptation mechanism comprises the following three core strategies:

[0140] Strategy 1: Automatic network depth inference based on voxel spacing. nnU-Net first analyzes the voxel spacing of the input data. The network depth (encoder-decoder layers) is automatically selected based on the spacing value. For isotropic or near-isotropic high-resolution data ( mm), using a deeper network ( The network depth is increased by adding layers to capture fine microstructures; for anisotropic or low-resolution data, the network depth is appropriately reduced to avoid oversampling in lower-resolution dimensions. The formula for choosing the network depth is:

[0141]

[0142] in The minimum sampling dimension (in voxels) is divided by 8 to ensure that the deepest feature map retains at least 8 voxels in each spatial dimension.

[0143] Strategy 2: Effective matching of receptive field and target structure scale. nnU-Net uses the maximum scale of tumor structures in the training data. and minimum scale Calculate the network's effective receptive field (ERF) to ensure that the ERF can cover the largest target structure:

[0144]

[0145] in The effective kernel size (considering the expansion of dilated convolutions). The goal of automatic network architecture adjustment is to meet... At the same time, it maintains a high spatial resolution to preserve details of small targets, provided that the data allows.

[0146] Strategy 3: Automatic hyperparameter optimization based on GPU memory constraints. nnU-Net optimizes hyperparameters based on available GPU memory capacity. By balancing batch size and network depth, the training hyperparameters are automatically optimized. The formula for determining it is:

[0147]

[0148] in For the framework overhead of video memory, Forward / backward propagation memory consumption per sample (as opposed to network depth) Input block size and basic channel number (Related).

[0149] Loss function design: using a joint loss function For network optimization, the loss function is a weighted average of the Dice loss and the cross-entropy loss:

[0150]

[0151] Where Dice loss is defined as:

[0152]

[0153] Cross-entropy loss is defined as:

[0154]

[0155] in: This is a probability graph for network prediction. The actual segmentation mask is manually labeled. and The first True label and predicted probability of category The number of segmentation categories (in this embodiment) (i.e., tumor / background binary classification). As a smoothing factor, to avoid the denominator being zero; These are the weighting coefficients for the cross-entropy loss. It is the set of all voxels within the image domain.

[0156] The key parameter settings for 3D nnU-Net are as follows:

[0157]

[0158] Output: Binary segmentation mask for the tumor region ,in Voxel representation It was classified as a tumor area.

[0159] (e) Calculation of 3D bounding box of tumor region and voxel-level expansion and clipping

[0160] Based on the tumor region segmentation mask output by 3D nnU-Net, the minimum bounding rectangle of the tumor region is calculated, and the predetermined voxel amount is extended outward along the three spatial dimensions to obtain standardized three-dimensional volume data.

[0161] The input for calculating and clipping the tumor region bounding box is: Binary segmentation mask for tumor region output by 3D nnU-Net . Z-Score-normalized four-sequence images , , , .

[0162] The process includes the following four steps:

[0163] Step 1: Calculate the set of voxel coordinates for the tumor region. Extract the voxel coordinates of all voxels labeled as tumors:

[0164]

[0165] Step 2: Calculate the minimum and maximum coordinates in each dimension to obtain the minimum bounding box. Calculate the minimum and maximum values ​​of the tumor voxel coordinates in the width, height, and depth directions respectively:

[0166]

[0167]

[0168]

[0169] This yields the minimum bounding rectangle of the tumor. .

[0170] Step 3: Isotropic voxel-level expansion. Based on the minimum bounding rectangle, expand outwards along each of the three spatial dimensions. Individual factors:

[0171]

[0172]

[0173]

[0174] The expanded bounding box is Extended parameters The range of values ​​is The specific selection criteria for voxels are as follows:

[0175]

[0176] The technical basis for choosing a 20-30 voxel extension range is as follows: (1) The microscopic infiltration range of high-grade gliomas such as glioblastoma (GBM) is usually within 10-20 mm outside the tumor margin. An extension of 20 voxels (assuming 1 mm isotropic resolution, about 20 mm) can cover the pathological change area in most cases; (2) An extension of 30 voxels provides an additional safety margin for diffuse growth patterns, avoiding omission of tumor microenvironment information that is of great significance to prognosis; (3) Excessive extension will introduce too much normal tissue background, increase the computational burden of subsequent models and may dilute tumor-related features.

[0177] Step 4: Boundary Cropping and Size Constraints. The expanded bounding box coordinates are cropped to within the effective image area. If the expanded bounding box is smaller than the target size of 128 in a certain dimension, it is padded by symmetrical expansion; if it is larger than 128, it is trimmed to 128 based on the tumor center.

[0178] Output: Coordinates of the expanded 3D bounding box This defines the minimum sufficient volume encompassing the tumor and its surrounding key pathological regions. The Z-Score-normalized four-sequence images are then analyzed based on this bounding box. , , , After cropping, standardized 3D volumetric data with spatial dimensions of 128×128×128 voxels and 4 channels (corresponding to the four sequences T1, T1CE, T2, and FLAIR) is obtained. This standardized 3D volumetric data will be used as input for the next step of the 3D vision encoder.

[0179] 1.2) Clinical text preprocessing branch

[0180] The clinical text preprocessing branch is used to convert clinical structured information into a text sequence using natural language templates, which is then processed by the clinical text encoder.

[0181] Clinical information sources are highly structured and heterogeneous, containing discrete variables (such as gender and lesion location), continuous variables (such as age and KPS score), and free text (such as symptom descriptions and treatment history). This embodiment designs a natural language template using standardized medical terminology to convert structured clinical data into coherent text descriptions that conform to the rules of natural language syntax. The design of the natural language template follows the principles of clinical information integrity and semantic coherence. The template structure is as follows:

[0182] The patient's age is Age: 18, Gender: Preoperative MRI showed an intracranial space-occupying lesion located in The main clinical symptoms include Course of disease Months. Karnofsky Functional Status Rating Points. Previous medical history: .

[0183] The definitions and value specifications for each placeholder are shown in the table below:

[0184]

[0185] This template transforms discrete, structured clinical information into a coherent natural language text sequence T, facilitating subsequent encoding processing by the BERT pre-trained language model.

[0186] Output: Text sequence T, which will be used as input to the clinical text encoder.

[0187] (ii) A 3D visual encoder, including a Transformer encoder, is used to uniformly divide the standardized 3D volume data into non-overlapping 3D patches along three spatial dimensions. Each patch is linearly projected onto the embedding space and then element-wise added to a learnable position encoding vector to obtain a patch embedding vector sequence. The index of the learnable position encoding vector is determined by the coordinates of the patch in the 3D mesh. Through formula The coordinates are uniquely determined, where x, y, and z are the grid coordinates of the patch in the width, height, and depth directions, respectively. , The total number of patches in the width and height directions are respectively; the patch embedding vector sequence is input into the Transformer encoder, and the image feature sequence is output.

[0188] The 3D vision encoder consists of four steps: patch partitioning, linear projection, 3D learnable position encoding, and Transformer encoder processing, as detailed below:

[0189] Step 1: 3D Patch Division—Dividing the 3D volume data into non-overlapping cubic patches

[0190] Input: The input to the 3D vision encoder is the standardized 3D volume data output by the data preprocessing module, whose dimensions are represented as follows: The specific meanings and values ​​of each dimension are as follows:

[0191]

[0192] Therefore, the complete dimension of the input data V is .

[0193] Processing procedure: The input 3D volume data is processed along... Axis (width W) Axis (height H) and The three spatial dimensions of axis (depth Z) are evenly divided into Each cube patch consists of three non-overlapping cube patches (3D patches). The spatial dimensions of each cube patch are given by... ,in That is, each patch is A cube.

[0194] The number of patches divided along each dimension is:

[0195]

[0196]

[0197]

[0198] in This represents the total number of patches in the width direction. This represents the total number of patches in the height direction. This represents the total number of patches in the depth direction.

[0199] The total number of patches (i.e., sequence length) obtained after partitioning the 3D volume data is:

[0200]

[0201] That is, each input data is serialized into A 3D cube patch.

[0202] The partitioned patch sequence is then transformed into a patch tensor through a tensor reshaping operation. Its dimensions are This tensor contains Each patch contains [number] patches. There are [number] channels, each with a spatial dimension of [size]. .

[0203] Output: Patch tensor .

[0204] Step 2: Linear Projection—Map each patch to a D-dimensional feature space using a learnable projection matrix. Flattening and linear projection mapping to the embedding space.

[0205] Input: The patch tensor output from step one .

[0206] Processing procedure:

[0207] Flatten each patch's channel and spatial dimensions into a one-dimensional vector, and you get... ,in Flattening dimensions for each patch:

[0208]

[0209] Therefore, the complete patch matrix after input serialization is ,Right now .

[0210] Then, through a learnable linear projection matrix Each flattened patch vector is mapped to a D-dimensional embedding space. In this embodiment, the embedding dimension D is 768. The dimensions of the projection matrix are:

[0211]

[0212] The projection matrix During model training, end-to-end optimization is performed using the backpropagation algorithm. The initial values ​​are randomly initialized using a truncated normal distribution with a standard deviation of [value missing]. .

[0213] The linear projection of the i-th patch is calculated as follows:

[0214]

[0215] in, For the first The flattened vector of each patch, For the projection matrix, For the first The projected embedding vector of each patch.

[0216] After linear projection, a learnable bias vector is added. :

[0217]

[0218] in The dimension is The dimension is consistent with the dimension of the projected embedding vector.

[0219] For all The matrix form for simultaneously linearly projecting each patch is:

[0220]

[0221] in, For the flattened matrix of all patches, For the projection embedding matrix of all patches, It is a column vector consisting entirely of 1s. The output dimension is... .

[0222] Output: Projection embedding matrix .

[0223] Step 3: 3D Learnable Position Encoding—Mapping (x,y,z) 3D coordinates to a single index and generating patch embeddings

[0224] Input: The projection embedding matrix output from step two

[0225] Processing procedure: In order to preserve the position information of the patch in three-dimensional space, this embodiment proposes a three-dimensional learnable position encoding mechanism, which maps the three-dimensional coordinates of each patch to a single index and obtains the corresponding position encoding vector through a lookup table operation.

[0226] Analysis of the limitations of two-dimensional position coding

[0227] The original ViT model is used to process two-dimensional images, and its positional encoding only needs to encode two spatial dimensions (horizontal and vertical). Let the two-dimensional image be divided into... For a two-dimensional grid, the two-dimensional position encoding matrix is: Index of each position From two-dimensional coordinates Through formula The mapping yields, where , .

[0228] However, when processing three-dimensional volume data, two-dimensional position encoding has fundamental shortcomings:

[0229] Dimensional Missing: Two-dimensional positional encoding cannot represent The spatial location information of the axis (depth direction) makes it impossible for the model to distinguish features at different depth levels;

[0230] Disruption of spatial relationships: In medical imaging, tumor tissue along... The axial direction has significant morphological continuity and anatomical hierarchy. Ignoring the positional information of the depth dimension will lead to incomplete modeling of the topological relationship in three-dimensional space.

[0231] Equivalent position ambiguity: In a two-dimensional coding scheme, two objects located at different depth levels but having the same positional ambiguity. The two patches for the coordinates will be assigned the exact same location code, causing confusion in spatial location information.

[0232] Mapping from 3D coordinates to a single index

[0233] This embodiment proposes a three-dimensional learnable position encoding mechanism, which encodes the three-dimensional coordinates of each patch. Mapped to a single index ,in , , The mapping formula is:

[0234]

[0235] Under the specific parameter configuration of this embodiment, ,therefore:

[0236]

[0237] in, ,common A different location index.

[0238] Position encoding matrix definition

[0239] This embodiment employs Learnable Absolute Positional Encoding. A positional encoding matrix is ​​defined. Its dimensions are Each row of this matrix Corresponding position index The positional encoding vector at that location.

[0240] Position encoding matrix During model training, the parameters are optimized end-to-end using the backpropagation algorithm along with other network parameters. Initial values ​​are randomly initialized using the same truncated normal distribution as the projection matrix, with a standard deviation of [value missing]. .

[0241] Location code lookup table acquisition

[0242] For the The patch has three-dimensional mesh coordinates of... First, calculate its position index. Then, its position encoding vector is obtained through a table lookup operation:

[0243]

[0244] in .

[0245] The design of the 3D positional encoding ensures that spatially adjacent patches have similar positional encoding vectors, while patches that are spatially distant have significantly different positional encoding vectors. Specifically, for any two position indices... and Their corresponding three-dimensional coordinates are as follows: and The Euclidean distance is:

[0246]

[0247] During the training of the position encoding matrix, the model implicitly learns that... The two smaller positions have more similar position encoding vectors, i.e. The value of is relatively small; conversely, two locations that are spatially far apart have significantly different location encoding vectors. This characteristic enables the model to effectively perceive relative positional relationships in three-dimensional space, thereby more accurately understanding the morphological distribution of tumors in three-dimensional space.

[0248] Patch embedding generated by element-wise addition

[0249] For each patch (in ), its final patch embedding vector The linear projection vector of the patch Its three-dimensional position encoding vector Adding each element together, we get:

[0250]

[0251] in, For the first The final embedding vector of each patch, with superscript... This indicates that the embedding is a feature representation of layer 0 (i.e., the input layer) of the Transformer encoder.

[0252] All The embedding vectors of each patch are stacked into a matrix to obtain the input feature matrix of the Transformer encoder:

[0253]

[0254] in, This is the complete patch embedding matrix.

[0255] Dimension confirmation and data flow tracing

[0256] During the patch embedding and generation phase, the dimensional changes of the data flow are shown in the table below:

[0257]

[0258] Output: Patch embedding vector sequence .

[0259] Step 4: Transformer Encoder Processing – Layer Stacking of L-Layer Encoders

[0260] Input: Patch embedding vector sequence

[0261] Processing procedure:

[0262] Transformer encoder overall architecture

[0263] The Transformer encoder consists of L identical encoder layers stacked sequentially. In this embodiment, the number of encoder layers... Each floor (in ) Receive the output feature matrix of the previous layer As input, after processing through a multi-head self-attention mechanism, a position feedforward network, residual connections, and layer normalization, the updated feature matrix is ​​output. .

[0264] A single-layer encoder consists of sub-modules.

[0265] Each Transformer encoder layer It consists of the following two main sub-modules in sequence:

[0266] Submodule A: Multi-Head Self-Attention (MSA) mechanism, which includes the following computational steps: 1) Generating query matrices through three independent linear projections. Key matrix Sum matrix ; 2) will , , Divided into 3) Calculate the scaled dot product attention independently for each attention head; 4) Concatenate the outputs of all attention heads and perform linear projection; 5) Obtain intermediate feature representations through residual connections and layer normalization.

[0267] Submodule B: Position-wise Feed-Forward Network (FFN), which includes the following computational steps: 1) Converting the dimension from... Expand to 2) Introduce a nonlinear transformation using the GELU activation function; 3) Reduce the dimension from... Restore to ; 4) The final output of the layer is obtained through residual connections and layer normalization.

[0268] The complete calculation formula for the encoder layer.

[0269] No. The complete forward propagation process of the layer encoder is described by the following formula:

[0270]

[0271]

[0272] in, This indicates the layer normalization operation. This is the intermediate output of the multi-head self-attention submodule. For the first The final output of the layer.

[0273] Summary of key encoder parameters.

[0274]

[0275] Multi-head self-attention mechanism: generation of query, key, and value matrices, and scaling dot product attention calculation formula.

[0276] 1) The input to the multi-head self-attention mechanism is the feature matrix after layer normalization in the ℓth layer encoder: ,in , .

[0277] 2) Generation of query, key, and value matrices

[0278] The query matrix is ​​generated using three independent learnable linear projection matrices. Key matrix Sum matrix :

[0279]

[0280]

[0281]

[0282] in, , , There are three independent learnable projection matrices, each with dimension 1. The dimension of the output matrix is... , , That is, the dimension of each matrix is ​​1. .

[0283] 3) Multi-head splitting

[0284] Will , , The three matrices are divided along the feature dimension into Each attention head. (in Extract the corresponding submatrix:

[0285]

[0286]

[0287]

[0288] in, The submatrix dimension of each attention head is... .

[0289] 4) Calculate the attention of the scaled dot product.

[0290] For each attention head Calculate its scaled dot product attention output :

[0291]

[0292] The calculation process can be broken down into the following sub-steps:

[0293] Sub-step 4.1: Calculate the attention score matrix.

[0294]

[0295] in, The attention score matrix has the following dimensions: Each element in the matrix Indicates the first The first query patch and the first Attention score between key patches.

[0296] Sub-step 4.2: Scaling operation.

[0297] To prevent the dot product result from being too large and causing the softmax function to enter the gradient saturation region, the attention score matrix is ​​scaled:

[0298]

[0299] scaling factor This ensures that the numerical range of the dot product result remains within a reasonable range.

[0300] Sub-step 4.3: Softmax normalization.

[0301] The scaled attention score matrix is ​​applied along the row direction (i.e., for each query position). Applying the softmax function, we obtain the attention weight matrix:

[0302]

[0303] The softmax function is defined as follows:

[0304]

[0305] Attention weight matrix The sum of the elements in each row is 1, that is... For all This is true. The dimension of this matrix is... .

[0306] Sub-step 4.4: Weighted summation.

[0307] attention weight matrix AND-value matrix Multiply to get the first... Output of each attention head:

[0308]

[0309] Multi-head attention stitching and output projection: Parallel computation and information integration of 12 attention heads

[0310] 1) Multi-head output splicing.

[0311] Will The output of each attention head Concatenate along the feature dimension to obtain the concatenated multi-head attention output:

[0312]

[0313] The dimensions after splicing are:

[0314]

[0315] therefore, .

[0316] 2) Output linear projection.

[0317] Linear projection is performed on the concatenated multi-head attention output, using a learnable output projection matrix. Perform the transformation:

[0318]

[0319] in, To output the projection matrix, end-to-end optimization is performed via backpropagation during training. The output dimension of the multi-head self-attention mechanism is... .

[0320] 3) Residual connection and layer normalization (first sub-module).

[0321] The output of the multi-head self-attention function is residually concatenated with the input, and then normalized at the application layer.

[0322]

[0323] in, This is the output of the first submodule and also the input of the position feedforward network.

[0324] 4) Summary of complete multi-head attention data stream.

[0325]

[0326] Location-based feedforward network: two-layer fully connected structure, GELU activation function, dimensionality expansion strategy

[0327] 1) Overall architecture of the location feedforward network.

[0328] The Position-wise Feed-Forward Network (FFN) applies the same fully connected feed-forward transform independently to each location (i.e., each patch). The FFN consists of two linear transform layers and a nonlinear activation function, and its structure is a three-stage design of "expansion-transformation-shrinkage".

[0329] 2) First layer fully connected and dimensional expansion.

[0330] The first fully connected layer of FFN transfers the input features from the embedding dimension. Extend to hidden layer dimension In this embodiment, .

[0331] The transformation formula for the first fully connected layer is:

[0332]

[0333] in, For the input of FFN, This is the weight matrix of the first fully connected layer. This is the bias vector for the first fully connected layer.

[0334] The output dimension of the first fully connected layer is:

[0335]

[0336] 3) GELU activation function.

[0337] After the first fully connected layer, a nonlinear transformation is introduced by applying the GELU (Gaussian Error Linear Unit) activation function to each element. The GELU function is defined as follows:

[0338]

[0339] in, The cumulative distribution function of the standard normal distribution. Let be the Gaussian error function. The approximate calculation form of the GELU activation function is:

[0340]

[0341] The GELU function has the following technical advantages over the ReLU function: Smoothness: The GELU function is continuously differentiable throughout its entire domain, unlike the ReLU function. The point where it is not differentiable; the property of random regularization: the GELU function is probabilistically... Retain the input values, in probability Setting the input to zero provides randomness that contributes to the training stability of deep networks; it is also better suited for Transformer architectures: in a wide range of natural language processing and computer vision tasks, the GELU function has been shown to be better suited than the ReLU function for training deep networks with Transformer architectures.

[0342] The intermediate representation after applying the GELU activation function is as follows:

[0343]

[0344] 4) Second-layer fully connected and dimension restoration.

[0345] The second fully connected layer of FFN reduces the hidden layer dimension from Restore to the embedded dimension :

[0346]

[0347] in, This is the weight matrix for the second fully connected layer. This is the bias vector for the second fully connected layer.

[0348] The final output dimension of FFN is:

[0349]

[0350] 5) Combining the above steps, the complete calculation formula for the location feedforward network is:

[0351]

[0352] The changes in the data flow dimensions of FFN are shown in the table below:

[0353]

[0354] Residual connections and layer normalization: a stable training mechanism for deep networks

[0355] 1) The principle and implementation of residual connection.

[0356] Residual connections are a technique used in this embodiment to achieve deep, stable training of the Transformer encoder. By introducing skip connections with identity mappings between the output and input of each submodule (multi-head self-attention mechanism and position feedforward network), the network can learn the residual function. Instead of directly learning the underlying mapping .

[0357] The formula for calculating residual connectivity is:

[0358]

[0359] in, For the input of the submodule, The residual mapping learned by the submodule This is the output of the residual connection.

[0360] The technical advantages of residual connections include: Gradient propagation guarantee: Skip connections provide a shortcut path for gradient propagation, effectively alleviating the gradient vanishing problem in deep networks, enabling networks with 12 layers or even deeper to be trained effectively; Identity mapping initialization: In the early stages of training, the network can easily learn the identity mapping (i.e., ... This ensures that deep networks will at least not perform worse than shallow networks; Feature reuse: Residual connections allow each layer to directly access the feature representations of all previous layers, promoting the reuse and propagation of hierarchical features.

[0361] 2) The principle and implementation of layer normalization.

[0362] Layer Normalization (LN) normalizes each sample across all feature dimensions. Its calculation formula is as follows:

[0363]

[0364] in, For the input feature vector, and These are the mean and variance of the sample across all feature dimensions:

[0365]

[0366]

[0367] and For learnable scaling and translation parameters, To prevent the numerical stability constant from being divided by zero, This indicates element-wise multiplication.

[0368] For batch feature matrix Layer normalization performs a normalization operation independently on each row vector (i.e., the feature representation of each patch).

[0369] 3) Complete forward propagation of the encoder layer

[0370] Combining residual connectivity and layer normalization, the first The complete forward propagation formula for the layer encoder is as follows:

[0371] Submodule A (Multi-head self-attention):

[0372]

[0373] Submodule B (Location Feedforward Network):

[0374]

[0375] In the above formula, each submodule follows the sequence of "first layer normalization—then submodule computation—then residual connection—final layer normalization," which is called a variant of the "Post-LN" architecture. In this embodiment, a "Pre-LN" architecture of "first residual connection—then layer normalization" can also be used, with the following formula:

[0376]

[0377]

[0378] This embodiment uses a Pre-LN architecture because in deep networks ( In the (layer), the Pre-LN architecture has been proven to have more stable training dynamics and faster convergence speed.

[0379] Hierarchical feature abstraction: shallow layer responds to local intensity patterns, mid-layer integrates tumor substructures, and deep layer captures global morphological features.

[0380] 1) The overall mechanism of hierarchical feature abstraction.

[0381] 3D vision encoder through The sequential processing of the layered Transformer encoder progressively abstracts low-level pixel-level features into high-level semantic features. Each layer of the encoder aggregates information globally through a self-attention mechanism and performs non-linear transformations through a feedforward network, thereby progressively advancing the feature representation from low to high levels. This hierarchical feature abstraction mechanism enables the model to learn image features of different granularities at different depths, ultimately forming a multi-scale comprehensive representation of brain tumors.

[0382] 2) Shallow features (encoder layers 1 to 4): Local intensity mode response.

[0383] In the shallow layer of the encoder ( In self-attention mechanisms, the receptive field is relatively localized, with each patch primarily focusing on other patches within its 3D spatial neighborhood. At this stage, the model learns the following low-level visual features:

[0384] Local intensity statistics: images of each modality in Statistics such as gray-level mean, variance, and gradient magnitude within a local neighborhood;

[0385] Edge and texture patterns: boundary gradient, texture direction, and roughness between tumor regions and normal tissue;

[0386] Intermodal contrast: the difference in signal intensity among four MRI modalities (FLAIR, T1, T1ce, T2) within the same local region;

[0387] Preliminary identification of tissue type: Based on local gray distribution, cerebrospinal fluid, gray matter, white matter and abnormal areas are preliminarily distinguished.

[0388] Shallow features It primarily encodes the local structural information of the volume data, similar to the low-level features extracted by shallow convolutional kernels in traditional convolutional neural networks.

[0389] 3) Mid-layer features (encoder layers 5 to 8): Tumor substructure integration.

[0390] In the middle layer of the encoder ( As the number of layers increases, the receptive field of the self-attention mechanism gradually expands, allowing each patch to attend to other patches within a larger spatial range. At this stage, the model learns the following intermediate-level semantic features:

[0391] Tumor subregion segmentation: distinguishing the substructural features of enhancing tumors, tumor cores, and edema regions;

[0392] Spatial context: The spatial location and topological structure between different subregions of a tumor, for example, enhancing tumors are usually located in the central region of the tumor core;

[0393] Three-dimensional morphological description: The morphological characteristics of a tumor in three-dimensional space, including volume, surface area, aspect ratio, and irregularity;

[0394] Peripheral tissue infiltration: The pattern of tumor infiltration into surrounding normal brain tissue and areas with blurred boundaries.

[0395] Mid-layer features It has already acquired a certain semantic level, which can effectively characterize the substructural composition and spatial organization of tumors.

[0396] 4) Deep features (encoder layers 9 to 12): global morphological feature capture.

[0397] In the deep layers of the encoder ( The self-attention mechanism enables global information interaction, allowing each patch to pay attention to all other patches in the volume data. At this stage, the model learns the following high-level semantic features:

[0398] Global morphological description: The complete morphological features of the entire tumor region in three-dimensional space, including symmetry, multifocality, and degree of necrosis;

[0399] Tumor grading indicators: high-level imaging features associated with WHO grading, such as tumor heterogeneity, necrosis rate, and enhancement pattern.

[0400] Cross-modal fusion representation: Deeply fuses information from four MRI modalities into a unified semantic representation, eliminating redundancy between modalities and preserving complementary information;

[0401] Clinically relevant features: biomarkers in deep imaging that are associated with patient prognosis and treatment response.

[0402] Deep features This constitutes a high-level semantic embedding of the input data, which serves as the input for subsequent task-specific heads (such as segmentation heads, classification heads, or survival prediction heads).

[0403] 5) Summary of data flow based on hierarchical feature abstraction.

[0404]

[0405] 6) The final output of the 3D vision encoder.

[0406] go through After sequential processing by the layer Transformer encoder, the 3D vision encoder outputs the final feature matrix:

[0407]

[0408] The feature matrix contains The feature representation of each patch, with each patch having a feature dimension of . These features represent global semantic information fused from the original four-channel MRI volume data, and can be directly used for downstream tasks, including three-dimensional tumor segmentation, tumor grading and classification, and patient survival prediction.

[0409] Furthermore, to support downstream tasks that require globally aggregated feature representations (such as overall lifetime prediction), this embodiment provides... Global average pooling is performed along the sequence dimension to obtain the global feature vector:

[0410]

[0411] This global feature vector It encodes global semantic information of the entire 3D volume data, making it suitable for tasks requiring holistic-level prediction.

[0412] (iii) Clinical text encoder, which uses a pre-trained language model to encode the text sequence and output clinical semantic vectors.

[0413] The clinical text encoder uses a BERT (Bidirectional Encoder Representations from Transformers) pre-trained language model to encode the text sequence output by the data preprocessing module into clinical semantic vectors, providing clinical context guidance for subsequent cross-modal fusion.

[0414] 3.1) WordPiece word segmentation processing

[0415] The input to the clinical text encoder is a text sequence T output by the data preprocessing module (this text sequence has been converted from structured clinical information into a coherent natural language description using a natural language template). This text sequence is first segmented into subwords by the WordPiece word segmenter. The WordPiece algorithm employs a greedy strategy to decompose out-of-vocabulary words into the most frequent subword units, effectively addressing the lexical sparsity problem of medical terminology.

[0416] The token sequence after word segmentation is:

[0417]

[0418] in A special label for classification is placed at the beginning of the sequence; This marks the end of the sequence. If the sequence length is less than the maximum sequence length... Then fill in the end. Mark to a fixed length The input after word segmentation is represented as follows:

[0419]

[0420] At the same time, segment embedding is constructed. and Position Embedding The final input embedding is obtained by adding three layers of embeddings:

[0421]

[0422] in The hidden layer dimension of the BERT-base model. Embed a lookup table for the token.

[0423] 3.2) BERT Deep Bidirectional Encoding

[0424] Input embedding It is fed into the BERT encoder for deep bidirectional encoding. The BERT encoder consists of... It consists of stacked Transformer coding layers, each containing two sub-layers: Multi-Head Self-Attention (MHA) and Feed-Forward Network (FFN).

[0425] For the layer( The calculation process is as follows:

[0426] Step 1: Multi-head self-attention calculation

[0427]

[0428]

[0429]

[0430] in For the number of attention heads, For each head dimension, and These are learnable parameters.

[0431] Step 2: Residual Connectivity and Layer Normalization

[0432]

[0433] Step 3: Feedforward network processing

[0434]

[0435] in , GELU is the activation function of the Gaussian error linear unit.

[0436] Step 4: Secondary residual connection and layer normalization

[0437]

[0438] After 12 layers of stacked encoding, the final hidden state matrix is ​​obtained:

[0439]

[0440] 3.3) Extracting the hidden state of the [CLS] token

[0441] BERT model will The token is designed as a special label that aggregates the semantic information of the entire sequence. This label is located in the 0th row of the output hidden state matrix, and its 768-dimensional vector represents the comprehensive semantics of all clinical variables.

[0442]

[0443] This clinical semantic vector It will be used as a source of key and value in the cross-modal attention fusion module, providing rich clinical context guidance for image features.

[0444] 3.4) Weight Freezing Strategy and LoRA Low-Rank Adaptation Fine-Tuning Mechanism

[0445] This embodiment employs a hybrid fine-tuning strategy to balance general semantic preservation and domain adaptability. By default, BERT pre-trained weights are frozen during model training and used only as fixed feature extractors. This strategy effectively avoids the catastrophic forgetting problem caused by insufficient data in downstream tasks.

[0446] In an optional configuration, this embodiment supports the LoRA (Low-Rank Adaptation) fine-tuning mechanism. LoRA achieves efficient parameter fine-tuning by injecting a low-rank decomposition matrix as a bypass into the original weight matrix.

[0447]

[0448] in For frozen pre-trained weights, , For a learnable low-rank matrix, the rank... (Typical settings) LoRA only trains a small number of parameters. Compared to full parameter fine-tuning The parameter reduction ratio, in Only about 2.08% of the parameters need to be trained at that time.

[0449] Output: The output of the clinical text encoder is a clinical semantic vector. This vector will serve as the key and value source for the directional cross-attention mechanism in the cross-modal fusion module, along with the image feature sequence output by the 3D visual encoder. Perform cross-modal fusion.

[0450] (iv) Cross-modal fusion module, used to calculate cross attention with the result of linear projection of the image feature sequence as query, and the clinical semantic vector as key and value, to obtain the modulated feature sequence; the modulated feature sequence and the linearly projected image feature sequence are residually connected, and after layer normalization and feedforward network processing, residual connection and layer normalization are performed again, and then the fused feature vector is obtained through global average pooling.

[0451] 4.1) Module Input

[0452] The input to the cross-modal attention fusion module consists of two parts:

[0453] Image feature sequence The output from the 3D vision encoder contains a high-level semantic representation of N 3D patches.

[0454] Clinical semantic vector The hidden state from the Clinical Text Encoder [CLS] token encodes the comprehensive semantics of all clinical variables.

[0455] 4.2) Semantic Space Alignment (Linear Projection Transformation)

[0456] Image feature sequence With clinical semantic vector While both feature dimensions are compatible (768-dimensional), their semantic spaces and representations differ: image features are in sequence form (512 spatial locations), while clinical features are in aggregated vector form (a single global representation). To achieve effective cross-attention computation, the image features undergo the following projection transformation:

[0457] Through a learnable projection matrix and bias Mapping image features to a fused semantic space:

[0458]

[0459] In this embodiment, spatial dimensions are fused. Therefore, the dimension of the projection matrix is The projected feature dimensions remain unchanged. .

[0460] 4.3) Design of cross-attention mechanism

[0461] The cross-attention mechanism designed in this embodiment is a core innovation for achieving deep integration of image and clinical semantics. Unlike self-attention, where the query, key, and value all come from the same feature source, cross-attention enables bidirectional interaction of cross-modal information.

[0462] Explicit definitions of Query, Key, and Value:

[0463]

[0464] For unified processing, clinical semantic vectors It is expanded (unsqueeze) into a two-dimensional tensor of sequence length 1. , which serves as the input for both Key and Value.

[0465] Linear transformation of Query, Key, and Value:

[0466]

[0467]

[0468]

[0469] in and The projection parameters are learned independently by the cross-attention module.

[0470] 4.4) Attention Weight Calculation

[0471] The core computational process of cross-attention is explained step by step as follows:

[0472] Step 1: Calculate the similarity score matrix

[0473]

[0474] Each element of the similarity matrix The image was measured. Spatial location ( The feature representation of a function is the degree of semantic relevance between itself and the clinical semantic vector. Scaling factor. ( This prevents the softmax gradient from disappearing due to an excessively large dot product magnitude.

[0475] Step 2: Obtain attention weights using Softmax normalization

[0476]

[0477] Since the Key sequence length is 1 in this scenario, the softmax operation degenerates into an identity mapping (single-element softmax is always 1), and the attention weights... For all locations This approach is valid. However, it retains the module's versatility, eliminating the need to modify the computational framework when expanding to multi-clinical token scenarios.

[0478] Step 3: Weighted aggregation of Values ​​to obtain fused features

[0479]

[0480] This step enables point-by-point modulation of clinical semantic information at each spatial location of the image: each spatial location Fusion characteristics All are clinical semantic vectors A scaled version, the scaling factor is determined by the semantic relevance of the location to the clinical context.

[0481] 4.5) Dynamic calibration mechanism

[0482] The dynamic calibration mechanism is one of the core innovations of this cross-modal fusion module. Its physical significance lies in the fact that the same image region should produce different attentional responses for patients with different clinical characteristics, thereby achieving personalized feature modulation of "same image, different strategies".

[0483] The mathematical essence of dynamic calibration lies in attention weights. For each spatial location The value is differentiated in response. Although the key is a single vector in this configuration, the query has 512 row vectors. The spatial heterogeneity of image features leads to differences in similarity scores. The distribution is differentiated across 512 locations:

[0484]

[0485] Clinical case illustration:

[0486] Suppose that the preoperative MRI images of the two patients show similar space-occupying lesion features in the frontal lobe region:

[0487] Patient A: A 32-year-old female with a 3-month history of illness, a Karnofsky Performance Status (KPS) score of 90, and mild headache as her clinical symptom. Her clinical semantic vector... It encodes information about "young age, good functional status, and early-stage lesions." The cross-attention mechanism guides the model to focus on well-defined cortical areas with low enhancement—these imaging features are more likely to correspond to low-grade lesions in young, early-stage patients.

[0488] Patient B: A 68-year-old male with a 12-month disease duration, a Karnofsky Performance Status (KPS) score of 60, and clinical symptoms of severe hemiplegia with cognitive impairment. His clinical semantic vector... It encodes information about "elderly, poor functional status, and advanced disease". Despite the same imaging region, cross-attention will guide the model to focus on central necrosis and ring enhancement areas—these imaging features are more likely to correspond to high-grade malignant lesions in elderly and advanced patients.

[0489] This dynamic calibration mechanism breaks through the limitation of traditional fusion methods that "clinical information is only used as a global bias", and realizes the adaptive and spatially differentiated fine modulation of image representation by clinical context.

[0490] 4.6) Point-by-point modulation of imaging representation by clinical information

[0491] Output of cross attention The entire spatial structure (512 locations) of the original image features was preserved, but the feature descriptors at each location were deeply modulated with clinical semantic information. This point-wise modulation mechanism ensures that:

[0492] Spatial heterogeneity preservation: Image features from different spatial locations retain their original anatomical location information;

[0493] Clinical context injection: The feature descriptor at each location incorporates the patient's individualized clinical semantics;

[0494] Gradient transferability: The modulated features still maintain gradient connectivity with respect to the image input, supporting end-to-end training.

[0495] 4.7) Residual connectivity prevents clinical information from becoming overly dominant.

[0496] To prevent clinical information from over-dominantly dominating image feature representation during the fusion process, this embodiment introduces residual connections to achieve information balance:

[0497]

[0498] The physical significance of this residual connection lies in the fact that the fused feature consists of two parts—the cross-attention output. (A branch dominated by clinical information) and post-projection image features (The branch dominated by original image information). Residual connections ensure that even if the output of the cross-attention branch approaches zero (such as when clinical information is irrelevant to image features), the fused features still degenerate into the image features themselves, avoiding excessive suppression of clinical information. This design effectively mitigates the risk of "modal collapse" in cross-modal fusion.

[0499] 4.8) Layer normalization and feedforward network to enhance nonlinear representation

[0500] The features after residual fusion are stably distributed after layer normalization:

[0501]

[0502] Layer normalization independently calculates the mean and variance of the feature vector at each spatial location, and then standardizes it along the feature dimension (768 dimensions):

[0503]

[0504] in , , For learnable scaling and translation parameters, is the numerical stability constant.

[0505] The nonlinear expressive power is then enhanced using a feed-forward network (FFN):

[0506]

[0507] in , ,accomplish The dimensional transformation is achieved through the GELU activation function, which introduces nonlinearity, enabling the fusion module to model higher-order interactions between images and clinical features.

[0508] After the FFN output, it passes through a residual connection again:

[0509]

[0510] 4.9) Global average pooling to obtain fixed-length fused feature vectors

[0511] To convert the two-dimensional feature matrix into a fixed-length fused feature vector for subsequent classifier use, this invention employs Global Average Pooling (GAP):

[0512]

[0513] Right now:

[0514]

[0515] Global average pooling has three advantages:

[0516] Dimensionality determinism: Regardless of the size of the input image, the output is always a fixed 768-dimensional vector, ensuring the consistency of the input dimension of the classifier;

[0517] Spatial robustness: It has natural invariance to changes in lesion location, avoiding overfitting that may be caused by max pooling;

[0518] Semantic aggregation: Each dimension of the fused feature vector is the average response of all spatial locations in that dimension, aggregating complete cross-modal fusion information.

[0519] The final result The output of the cross-modal attention fusion module will be fed into the downstream prediction head for pathological grading or survival prognosis prediction.

[0520] 4.10) Dimensional projection of fused features onto the prediction head

[0521] To adapt to the input dimensionality requirements of downstream multi-task prediction heads, features are fused. Mapped to a 512-dimensional feature space via a learnable linear projection layer:

[0522]

[0523] in For the projection matrix, This is the bias vector. During training, this projection layer is jointly optimized with the parameters of other modules to achieve a smooth transition from the fused representation to the prediction space.

[0524] Output: The output of the cross-modal attention fusion module is a 512-dimensional fused feature vector. fusion of feature vectors The input is fed into the task-specific gated fusion submodule, and after being processed by the gated network, three task-specific fusion feature vectors are generated, which are used as inputs for the subsequent prediction branches of three molecular markers: IDH mutation, 1p / 19q co-deletion, and MGMT methylation.

[0525] 4.11) Task-Specific Gating Fusion Mechanism

[0526] This embodiment, building upon cross-modal attention fusion, further introduces a task-specific gated multimodal fusion mechanism. The core idea is that the three molecular markers—IDH mutation, 1p / 19q co-deletion, and MGMT promoter methylation—have fundamentally different degrees of dependence on imaging and clinical modalities. Therefore, task-specific fusion strategies should be employed, rather than sharing the same fusion method across all tasks. The overall data flow is: globally shared encoding features → task-gated network → three task-specific fusion weights → three task-specific fusion features → three prediction heads.

[0527] The biological motivations for task gating are as follows: IDH mutations are associated with the overall metabolic characteristics of tumors, and imaging phenotypes and clinical factors such as age are all indicative, so an image-clinical balanced fusion strategy is adopted; 1p / 19q co-deletion is highly correlated with the imaging texture features of specific tumor regions, so an image-led fusion strategy is adopted; MGMT promoter methylation is more strongly associated with clinical factors such as treatment history and age, so a clinical-led fusion strategy is adopted.

[0528] In its specific implementation, the task-specific gating fusion submodule includes a task embedding layer and three parallel task gating networks. The task embedding layer defines learnable task embedding vectors for the three prediction tasks, with a dimension of 128. The t-th task gating network takes the concatenation result of the fused feature vector z and the corresponding task embedding vector as input, passes through two layers of gated perceptron (hidden layer dimension 256, ReLU activation) to output two-dimensional logits, and then passes through softmax normalization to obtain the image modality weights and clinical modality weights specific to task t, the sum of which is always 1.

[0529] The task-specific fusion feature vector is calculated as follows: the image-side representation obtained by global average pooling of the projected image features and the clinical-side representation obtained by linear projection of the clinical semantic vector are used as the basis, and then weighted and combined according to the modality weights specific to task t. The result is then mapped to a 512-dimensional feature space by linear projection to obtain the task-specific fusion feature vector. The three task-specific fusion feature vectors are respectively input into the prediction branches corresponding to IDH mutation, 1p / 19q co-deletion, and MGMT promoter methylation.

[0530] The biases of the task-gated network output layers are initialized differentially based on biological priors: the IDH task is initialized with zero bias, corresponding to balanced fusion; the 1p / 19q task is initialized with a bias biased towards the imaging modality, with initial weights of approximately 0.7:0.3; and the MGMT task is initialized with a bias biased towards the clinical modality, with initial weights of approximately 0.3:0.7. The gating weights are jointly optimized end-to-end with other network parameters during training, with the biological prior serving only as the starting point for optimization. This accelerates model convergence while preserving data-driven adaptive capabilities.

[0531] 4.12) Modal quality assessment and uncertainty weighted fusion mechanism

[0532] This embodiment further introduces a modal quality assessment and uncertainty-weighted fusion mechanism (Uncertainty-Aware Fusion), which dynamically adjusts the modal weights of each case on a case-by-case basis based on the quality of the input data within the same prediction task. This mechanism includes a modal quality assessment unit and an uncertainty-weighted unit. Its design principle is that the dynamic adjustment of weights must have quantifiable and interpretable quality assessment criteria to avoid adaptive weighting degenerating into an uninterpretable black box. The final fused weights are driven by three factors: modal quality score, prediction uncertainty, and cross-modal divergence.

[0533] Modal quality assessment unit:

[0534] The image modality quality score is calculated by weighting the quality dimensions shown in the table below. It should be noted that the quality dimension indicators do not need to be absolutely precise; it is only necessary to establish a reliable relative ranking among cases in the same batch, and the modality with the worst quality will be downweighted accordingly.

[0535]

[0536] The clinical modality quality score is calculated by weighting the following dimensions: information completeness, i.e., the proportion of missing fields such as age, gender, KPS score, clinical symptoms, and treatment history; the more missing fields, the lower the weight. Logical consistency, such as marking contradictory information when "KPS score of 90" and "bedridden" coexist. Information timeliness, image-clinical pairings with clinical data collection time more than 6 months after surgery are downweighted. Numerical reasonableness, outliers such as age greater than 100 or less than 0, and KPS score greater than 100 are marked as abnormal.

[0537] Uncertainty-weighted unit:

[0538] For each of the three prediction tasks—IDH mutation status, 1p / 19q co-deletion status, and MGMT promoter methylation status—the weights of the clinical modality in the task-specific gated fusion submodule are reset to zero, and weighted fusion is performed using only the image-side representation. The resulting temporary fusion feature vector is input into the multi-task prediction head to obtain the image branch prediction probability. Similarly, the weights of the image modality are reset to zero, and weighted fusion is performed using only the clinical-side representation. The resulting temporary fusion feature vector is input into the multi-task prediction head to obtain the clinical branch prediction probability. The uncertainty weighting unit obtains the image branch prediction probability and the clinical branch prediction probability, respectively, and the symmetric KL divergence between the two is used as the cross-modal divergence degree of the task.

[0539] The modality weights output by the task gating network are dynamically adjusted on a case-by-case basis, in a manner that is positively correlated with modality quality score and negatively correlated with prediction uncertainty. Prediction uncertainty is measured by performing 20 random forward propagations using Monte Carlo Dropout, with the variance of the prediction probability as the metric. That is, modalities with higher modality quality scores and lower prediction uncertainty have higher weights; conversely, modalities with lower modality quality scores and higher prediction uncertainty have lower weights.

[0540] The fusion decision logic is as follows: When the cross-modal divergence is lower than the preset divergence threshold, the original modality weights output by the task gating network are maintained; when the cross-modal divergence is higher than the preset divergence threshold and the image modality quality score is higher than the clinical modality quality score, image-led fusion is adopted; when the cross-modal divergence is higher than the preset divergence threshold and the clinical modality quality score is higher than the image modality quality score, clinical-led fusion is adopted; when the divergence is high and both modality quality scores are lower than the preset quality threshold, a warning message of "prediction uncertain, supplementary examination recommended" is output, rather than forcibly giving a potentially incorrect classification result.

[0541] Clinical scenario adaptive example: When the image is blurry (significant motion artifacts or the tumor volume is too small), the weight of clinical information is automatically increased; when clinical information is missing (such as no medical history in the emergency department), the weight of the image is automatically increased.

[0542] Therefore, this embodiment achieves task-case dual adaptive fusion: the task-specific gating network selects the fusion paradigm based on the biological characteristics of molecular markers, and the uncertainty weighting unit dynamically adjusts the modality weights based on the quality of input data, achieving precise fusion that is "different for genes and adjusted for cases", improving prediction accuracy while ensuring robustness under low-quality data and interpretability of the fusion process.

[0543] (v) Multi-task prediction head, which contains three parallel prediction branches, used to synchronously output prediction results of IDH mutation state, 1p / 19q co-deletion state and MGMT promoter methylation state based on task-specific fusion feature vectors.

[0544] In this embodiment, the multi-task prediction head includes three structurally identical but parameter-independent multi-layer perceptron (MLP) classifiers, corresponding to the prediction tasks of three key molecular markers of glioma:

[0545] MLP1: IDH gene mutation status classification (wild type vs. mutant)

[0546] MLP2: Classification of 1p / 19q Joint Deletion Status (Co-deletion vs. Non-co-deletion)

[0547] MLP3: MGMT promoter methylation status classification (methylation positive vs. methylation negative)

[0548] Each MLP classifier receives the corresponding task-specific fusion feature vector from the task-specific gated fusion submodule as input, and performs nonlinear transformation and probabilistic prediction through a four-layer fully connected network. The three classifiers do not share any parameters, ensuring that the prediction boundaries for each task evolve independently in the parameter space. The three task-specific fusion feature vectors respectively characterize the optimal image-clinical information ratio for each molecular marker, avoiding the averaging of inter-task differences caused by a single shared fusion representation, and further improving the task adaptability of each prediction branch.

[0549] (vi) Training and Validation of the Prediction System in this Embodiment

[0550] 6.1) The multi-task loss function used in the training phase is composed of three weighted cross-entropy losses:

[0551]

[0552] Among them, single-task cross-entropy loss Defined as:

[0553]

[0554] Weighting coefficient A three-factor adaptive update strategy is adopted:

[0555]

[0556] Among them, experience settings (Weight of factors related to sample imbalance). (Convergence speed factor weight). (Clinical prior factor weights), updated after each epoch. Apply normalization constraints.

[0557] The specific calculation methods for each factor are as follows:

[0558] (1) Imbalanced weights Inverse frequency weighting based on the ratio of positive to negative samples for each task:

[0559]

[0560] (2) Convergence speed weight : Dynamically adjust based on the decay trend of the validation set loss. If the validation loss of a task decreases less than a threshold within 5 consecutive epochs. If so, its weight will be increased by 10%.

[0561] (3) Clinical prior weights Fixed baseline values ​​were assigned based on the relative importance of clinical diagnosis: IDH mutation was 0.40, 1p / 19q co-deletion was 0.35, and MGMT methylation was 0.25.

[0562] The weighting is based on the influence of each biomarker on diagnostic decisions in the World Health Organization (WHO) Classification of Tumors of the Central Nervous System (2021 edition): IDH mutation, as a primary biomarker for glioma subtyping (the primary determinant), is assigned the highest weight of 0.40; 1p / 19q co-deletion, as a diagnostic biomarker for oligodendroglioma, is assigned a weight of 0.35; and MGMT methylation, as a predictive biomarker for treatment response, is assigned a weight of 0.25.

[0563] Normalization constraints: This ensures the stability of the total gradient magnitude.

[0564] 6.2) Imbalanced Sample Handling: Minority Class Gradient Contribution Enhancement Mechanism

[0565] To address the sample imbalance caused by uneven positivity rates among various molecular markers (approximately 80% for IDH mutation, approximately 50% for 1p / 19q co-deletion, and approximately 40% for MGMT methylation), this framework implements a minority class gradient contribution enhancement mechanism:

[0566] Focal Loss Correction: A modulation factor is introduced on top of the cross-entropy to reduce the gradient weights of easily separable samples and focus on difficult-to-separate samples.

[0567]

[0568] in To focus parameters, For the sample The balance coefficient of the class. Misclassification of minority class samples will generate larger gradient magnitudes, driving the model to optimize towards the decision boundary.

[0569] Reweighting of effective sample count: Introducing the effective number of samples for each category. ,in The original sample size. The reweighting coefficient is This allows the minority class to receive a larger proportion of the loss contribution.

[0570] 6.3) Multi-task collaborative optimization: Joint training of shared encoder and task-specific prediction head

[0571] The complete multi-task training process follows the encoder-prediction head joint optimization paradigm:

[0572] (1) Forward propagation: The input sample extracts hierarchical features through a 3D visual encoder, and generates a 512-dimensional fused feature vector through a cross-modal attention fusion module. . Simultaneously, three independent MLP classifiers are fed in to generate the predicted probabilities of three molecular markers in parallel.

[0573] (2) Backpropagation: Losses of each task Weighted The weighted average is then aggregated to obtain the total loss. The gradients are backpropagated from each prediction head to the shared encoder, whose parameters are updated by receiving the vector sum of the gradients from the three tasks:

[0574]

[0575] (3) Gradient clipping and conflict resolution: The gradient normalization (GradNorm) algorithm is used to monitor the gradient magnitude of each task. When the gradient norm of a task exceeds the threshold, projection clipping is performed to prevent a single task from dominating the update direction of shared parameters.

[0576] (4) Alternating training strategy: After each complete epoch, joint training and single-task fine-tuning are performed alternately in a 3:1 ratio to ensure that the shared representation takes into account both generality and task specificity.

[0577] (5) Five-fold cross-validation strategy: patient-level data partitioning

[0578] Model evaluation employed 5-fold stratified cross-validation, with all data partitioning performed at the patient level. This meant all image slices from the same patient had to fall within the same fold, strictly prohibiting data leakage. Within each fold, the training, validation, and test sets were allocated in a 60%:20%:20% ratio. Stratified sampling ensured that the positive rate distribution of the three molecular markers within each fold was consistent with the overall population.

[0579] ( The prediction system synchronously outputs the prediction results of IDH mutation status, 1p / 19q co-deletion status and MGMT promoter methylation status.

[0580] During the inference phase, a deterministic forward propagation is performed. The complete computational path from the original input to the prediction result is as follows:

[0581] Step 1: Input the preprocessed multimodal MRI image block ( Voxels (4 sequence channels) and clinical feature vectors .

[0582] Step 2: The image blocks are processed layer by layer by a 3D visual encoder to extract hierarchical features. .

[0583] Step 3: The cross-modal attention fusion module calculates the bidirectional attention response of global image features and clinical features, and outputs a 512-dimensional fused feature vector. .

[0584] Step 4: The data is processed through MLP1, MLP2, and MLP3, and after four fully connected layers, the Sigmoid function outputs three predicted probabilities: , , .

[0585] Step 5: Binarize the probabilities into final classification labels using a threshold of 0.5.

[0586] Example 2: An electronic device for predicting molecular subtyping of gliomas based on 3D MRI and clinical text, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the prediction method described in Example 1.

[0587] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for predicting the molecular subtype of glioma based on 3D MRI and clinical text, characterized by: Includes the following steps: ( The system acquires preoperative MRI images and clinical structured information of the patient to be predicted, and inputs the preoperative MRI images and clinical structured information into the prediction system; the prediction system includes: a data preprocessing module, a three-dimensional visual encoder, a clinical text encoder, a cross-modal fusion module, and a multi-task prediction head; The data preprocessing module is used to perform rigid registration, N4 bias field correction, Z-Score normalization, and 3D nnU-Net-based tumor region segmentation on the preoperative MRI images. Based on the tumor region segmentation results output by 3D nnU-Net, it calculates the minimum bounding box surrounding the tumor region and expands it outward by a predetermined number of voxels along three spatial dimensions to obtain standardized three-dimensional volume data. It also converts the clinical structured information into a text sequence using a natural language template. A 3D visual encoder, including a Transformer encoder, is used to uniformly divide the standardized 3D volume data into non-overlapping 3D patches along three spatial dimensions. Each patch is linearly projected onto an embedding space and then element-wise added to a learnable position encoding vector to obtain a patch embedding vector sequence. The index of the learnable position encoding vector is determined by the patch's coordinates in the 3D mesh. Through formula The coordinates are uniquely determined, where x, y, and z are the grid coordinates of the patch in the width, height, and depth directions, respectively. , The total number of patches in the width and height directions are respectively; the patch embedding vector sequence is input into the Transformer encoder, and the image feature sequence is output; A clinical text encoder, employing a pre-trained language model, is used to encode the text sequence and output a clinical semantic vector. The cross-modal fusion module is used to calculate cross-attention using the result of linear projection of the image feature sequence as the query and the clinical semantic vector as the key and value, to obtain the modulated feature sequence; the modulated feature sequence is residually connected with the linearly projected image feature sequence, and after layer normalization and feedforward network processing, residual connection and layer normalization are performed again, and then the fused feature vector is obtained through global average pooling; The multi-task prediction head contains three parallel prediction branches, which are used to synchronously output the prediction results of IDH mutation state, 1p / 19q co-deletion state and MGMT promoter methylation state based on the fusion feature vector. The prediction system is obtained through the following training method: The prediction system is trained using a multi-task loss function, which is a weighted sum of the losses of each task, where the weights of each task are... Calculate, where α=1.0, β=0.5, γ=0.3, For inverse frequency weights based on sample imbalance, To validate the dynamic weights based on the trend of loss decay, Clinical prior weights; ( The prediction system synchronously outputs the prediction results of IDH mutation status, 1p / 19q co-deletion status and MGMT promoter methylation status.

2. The method for predicting the molecular subtype of glioma based on three-dimensional MRI and clinical text according to claim 1, characterized in that: The Transformer encoder consists of 12 sequentially stacked coding layers; Each coding layer includes: a first normalized sublayer, a multi-head self-attention sublayer, a first residual adder, a second normalized sublayer, a feedforward network sublayer, and a second residual adder; The connection relationships within each coding layer are as follows: The input of the current layer is processed sequentially by the first normalization sublayer and the multi-head self-attention sublayer to obtain the attention output; The first residual adder adds the attention output to the input of the current layer to obtain a first intermediate result; The first intermediate result is processed sequentially by the second normalization sub-layer and the feedforward network sub-layer to obtain the feedforward output; The second residual adder adds the feedforward output to the first intermediate result to obtain the output of the current layer; The embedding dimension is 768, the number of attention heads in the multi-head self-attention sub-layer is 12, the dimension of each head is 64, and the hidden layer dimension of the feedforward network sub-layer is 3072, with GELU as the activation function. The output of the 12th coding layer is the image feature sequence, with a dimension of 512×768.

3. The method for predicting the molecular subtype of glioma based on three-dimensional MRI and clinical text according to claim 1, characterized in that: The cross-modal fusion module includes: a query projection sublayer, a key-value projection sublayer, a single-head cross-attention sublayer, a first residual adder, a first normalization sublayer, a feedforward network sublayer, a second residual adder, a second normalization sublayer, a global average pooling sublayer, and a linear projection sublayer. The connection relationships between the sub-layers are as follows: (a) The image feature sequence F img Learnable projection matrix W proj and bias b proj Linear projection yields the projected image features F. img ′; (b) The clinical semantic vector f clin It is expanded into a two-dimensional tensor of length 1, which serves as the input for the key and value. (c) The single-headed cross-attention sublayer uses the projected image feature F img After querying the projection matrix W Q and bias b Q The result after linear transformation is the query, which is the clinical semantic vector projected onto the key matrix W. K and bias b K The result of the linear transformation is the key and the longitude projection matrix W. V and bias b V The result after the linear transformation is the value, Value. Cross-attention is then calculated, with a scaling factor of [value missing]. With d=768, we obtain the cross-attention output. ; (d) The first residual adder outputs the cross-attention statement. With the projected image feature F img The residual fusion features are obtained by adding them together. This is used to prevent modality collapse caused by excessive dominance of clinical information; (e) The residual fusion feature After being processed sequentially by the first normalization sublayer and the feedforward network sublayer, a feedforward output is obtained. The second residual adder adds the feedforward output to the input of the first normalization sublayer, and then processes it again by the second normalization sublayer to obtain the normalized output. ; (f) The global average pooling sublayer averages the normalized output Fout along the sequence dimension to obtain the fused feature vector. ; (g) The linear projection sublayer will fuse the feature vector via projection matrix and bias vector Linear projection yields a fused feature vector z, which serves as the unified input to the multi-task prediction head; The hidden layer of the feedforward network sub-layer has a dimension of 3072 and the activation function is GELU.

4. The method for predicting the molecular subtype of glioma based on three-dimensional MRI and clinical text according to claim 1, characterized in that: The three parallel prediction branches are the IDH mutation state prediction branch, the 1p / 19q co-deletion state prediction branch, and the MGMT promoter methylation state prediction branch, respectively. Each branch is an independent MLP classifier. The MLP classifier is a four-layer fully connected network, with dimensions of 512-dimensional → 256-dimensional → 128-dimensional → 64-dimensional → 1-dimensional. The activation function of the first three layers is ReLU, the output layer is Sigmoid, and the dropout rates of each layer are 0.3, 0.2, and 0.2, respectively.

5. The method for predicting the molecular subtype of glioma based on three-dimensional MRI and clinical text according to claim 1, characterized in that: The tumor region segmentation based on 3D nnU-Net employs a dynamic network topology adaptive mechanism, with the network depth L determined by the voxel spacing Δ using the formula... Automatic inference, among which The loss function of the 3D nnU-Net is: Where Dice loss is defined as: Cross-entropy loss is defined as: in: This is a probability graph for network prediction. The actual segmentation mask is manually labeled. and The first True label and predicted probability of category The number of segmentation categories; As a smoothing factor, to avoid the denominator being zero; These are the weighting coefficients for the cross-entropy loss. It is the set of all voxels within the image domain.

6. The method for predicting the molecular subtype of glioma based on three-dimensional MRI and clinical text according to claim 1, characterized in that: The clinical text encoder uses the BERT model, whose pre-trained weights are frozen during training and whose parameters are efficiently fine-tuned using low-rank adaptation (LoRA), with LoRA rank r=8.

7. The method for predicting the molecular subtype of glioma based on three-dimensional MRI and clinical text according to claim 1, characterized in that: The multi-task loss function is applied after each training iteration according to... The weights of each task are subject to normalization constraints, where Let be the weight of the t-th task in the e-th round.

8. The method for predicting the molecular subtype of glioma based on three-dimensional MRI and clinical text according to claim 1, characterized in that: The cross-modal fusion module further includes a task-specific gated fusion submodule, which comprises a task embedding layer and three parallel task gating networks. The task embedding layer generates learnable task embedding vectors for the IDH mutation state prediction task, the 1p / 19q co-deletion state prediction task, and the MGMT promoter methylation state prediction task, respectively. Each task gating network takes the concatenation result of the fused feature vector and the corresponding task embedding vector as input, outputs the two-dimensional modal weights of the corresponding task through a gated perceptron, and obtains the task-specific shadow through softmax normalization. Modal weights and clinical modal weights are used; the projected image features and the clinical semantic vectors are weighted and combined according to the modal weights specific to each task to obtain three task-specific fusion feature vectors, which are then input into the corresponding prediction branches; the initial weights of the task-gated network are set differently according to the biological characteristics of each molecular marker: the image modal weights and clinical modal weights of the IDH mutation state prediction task are initialized in a balanced manner, the image modal weights of the 1p / 19q co-deletion state prediction task are higher than the clinical modal weights, and the clinical modal weights of the MGMT promoter methylation state prediction task are higher than the image modal weights.

9. The method for predicting the molecular subtype of glioma based on three-dimensional MRI and clinical text according to claim 8, characterized in that: The task-specific gating fusion submodule further includes a modality quality assessment unit and an uncertainty weighting unit. The modality quality assessment unit calculates the image modality quality score and the clinical modality quality score. The image modality quality score is obtained by weighting at least two of the following: inter-slice motion artifact index, signal-to-noise ratio, tissue contrast, multi-sequence registration accuracy, tumor segmentation confidence, and tumor visibility. The clinical modality quality score is obtained by weighting at least two of the following: clinical field completeness, logical consistency, information timeliness, and numerical reasonableness. The uncertainty weighting unit calculates the symmetric KL divergence between the image branch prediction probability and the clinical branch prediction probability as the cross-modality divergence degree, and dynamically adjusts the modality weights output by the task gating network on a case-by-case basis in a manner that is positively correlated with the modality quality score and negatively correlated with the prediction uncertainty. When the cross-modality divergence degree exceeds a preset divergence threshold and both modality quality scores are lower than the preset quality threshold, a prompt message indicating unreliable prediction and suggesting supplementary examination is output.

10. An electronic device for predicting the molecular subtyping of gliomas based on three-dimensional MRI and clinical text, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the prediction method according to any one of claims 1 to 10.

Images