An alignment-guided dual-layer reliable gated ct-wsi fusion prediction method

Abstract

The present application belongs to the technical field of medical artificial intelligence, and in particular to a double-layer reliability gated CT-WSI fusion prediction method guided by alignment, comprising the following steps: obtaining the pre-treatment three-dimensional CT volume data of a subject and its corresponding preoperative biopsy whole slide image WSI; extracting CT features from the CT volume data, the CT features including at least one CT token and a CT global feature; extracting WSI features from the WSI, the WSI features including at least one WSI token and a WSI global feature, calculating the cross-modal mismatch degree based on the CT global feature and the WSI global feature, and generating a pre-gated interaction coefficient; the present application generates a pre-gated interaction coefficient by dynamically calculating the cross-modal mismatch degree, and combines an adaptive fusion weight mechanism, effectively coordinating the reliability difference between CT and WSI modalities, solving the prediction instability problem caused by data distribution offset and mismatching of supervision signals, and having significantly improved low false positive rate robustness and clinical applicability.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence technology, specifically to an alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method. Background Technology

[0002] Esophageal cancer is a major malignant tumor worldwide, and the assessment of pathological response after neoadjuvant therapy is crucial for clinical decision-making. Accurate determination of the tumor regression grade TRG0 (i.e., no residual tumor) directly impacts the necessity of surgery and follow-up strategies. Misdiagnosing non-TRG0 patients as TRG0 significantly increases the risk of undertreatment, leading to serious consequences such as tumor recurrence and metastasis. Therefore, achieving highly robust TRG0 prediction with a low false positive rate is an urgent need in clinical practice.

[0003] Existing technical solutions mainly fall into two categories: three-dimensional feature extraction models based on CT images, which predict tumors by analyzing morphological information; and weakly supervised multi-instance learning frameworks based on preoperative biopsy whole-section images, which combine pathological models to mine histological features. Some studies have attempted to integrate CT and WSI modal data through post-fusion or static weighting strategies, but these methods face three core defects in practical applications. First, the gold standard for TRG grading relies on pathological evaluation of en bloc resection specimens after treatment, while preoperative CT only provides three-dimensional structural information of the tumor, and biopsy WSI only covers local tissue areas. This structural mismatch between the supervisory signal and the input evidence makes it difficult for the model to capture subtle differences between TRG0 and non-TRG0, especially in TRG0 / 1 borderline cases, where small residual lesions are easily missed, resulting in significant cross-center fluctuations in the model's performance at the low false positive rate operating point.

[0004] Secondly, differences in image acquisition protocols, pathology slide preparation procedures, and equipment parameters among different medical centers cause significant data distribution shifts. Existing fusion methods lack a mechanism for perceiving dynamic relationships between modalities. When the tumor boundaries shown by CT and the histological features presented by WSI contradict each other due to sampling bias or technical factors, fixed-weight fusion strategies often amplify spurious correlations, further weakening the stability of prediction results at critical operating points. Furthermore, the area under the curve (AUC), a commonly used evaluation metric, cannot fully reflect the clinical priority of low false positive rates, making it difficult to guarantee the reliability of existing models in real-world scenarios. These issues collectively hinder the synergistic value of preoperative CT and WSI data in TRG0 prediction tasks, especially in multi-center validation and modality-deficient scenarios where they are more vulnerable.

[0005] Therefore, this application proposes an alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method to solve the above problems. Summary of the Invention

[0006] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides an alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method, which effectively alleviates the cross-modal mismatch problem, significantly improves prediction robustness and stability under low false positive rate constraints, and is particularly suitable for multi-center clinical validation scenarios, thus solving the problems mentioned in the background technology.

[0007] (II) Technical Solution To achieve the above objectives, the present invention specifically adopts the following technical solution: An alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method includes the following steps: Acquire pre-treatment three-dimensional CT volume data of the subjects and their corresponding preoperative biopsy whole slice images (WSI); Extract CT representations from the CT volume data, the CT representations including at least one CT token and a global CT representation; Extract WSI representations from the WSI, the WSI representations including at least one WSI token and a WSI global representation; Based on the CT global representation and the WSI global representation, the degree of cross-modal mismatch is calculated, and pre-gated interaction coefficients are generated, wherein the pre-gated interaction coefficients are negatively correlated with the degree of mismatch. Based on the pre-gated interaction coefficients, cross-modal interaction and fusion are performed on the CT token and the WSI token to generate a fused representation. Based on the CT representation, the WSI representation, and the degree of mismatch, adaptive fusion weights for CT experts, WSI experts, and fusion experts are generated. Based on the adaptive fusion weights, residual fusion is performed on the CT-only prediction results, WSI-only prediction results, and fusion prediction results based on the fusion representation to obtain the final prediction result. The final prediction result is used to output the prediction probability value representing the possibility of TRG0, which is an intermediate result for medical personnel to refer to.

[0008] Furthermore, the extraction of CT characterization from the CT volume data includes: The CT scan data is segmented to obtain the region of interest (ROI). Based on the tumor ROI, multiple habitat masks representing different spatial ranges are constructed. The habitat mask includes a core mask representing the interior of the tumor, a rim mask representing the tumor boundary, and a ring mask representing the area surrounding the tumor. Voxel-level feature maps of the CT volume data are extracted using a pre-trained 3D visual encoder. Pooling is performed on the voxel-level feature map within each habitat mask to obtain the corresponding CT token.

[0009] Further, the extraction of WSI characterization from the WSI includes: The WSI is segmented and divided into regions to obtain multiple image patches; Each patch is encoded as a semantic token using a pre-trained pathology-based model; Each semantic token is scored for importance, and the top K tokens with the highest scores are retained; The state-space sequence model MambaMIL is used to perform context modeling on the K retained tokens, and the global representation of WSI and the WSI-only prediction results are obtained through multi-instance learning MIL pooling.

[0010] Furthermore, the calculation of cross-modal mismatch degree and the generation of pre-gated interaction coefficients further include: The global representation of CT and the global representation of WSI are concatenated, the difference is calculated, and the element-wise product is calculated. The result is input into the difference embedding network to obtain the difference vector. The difference vector is input into the mismatch scoring network and mapped by the sigmoid function to obtain the scalar mismatch score. Set the pre-gated interaction coefficient to 1 minus the mismatch score, so that the higher the mismatch score, the lower the pre-gated interaction coefficient.

[0011] Furthermore, the cross-modal interaction and fusion of the CT token and the WSI token to generate a fused representation includes: Pre-gating is achieved by weighting the results of cross-modal projection using the pre-gated interaction coefficients. Input the weighted CT token and WSI token into the Cross-Mamba module to update their respective context information; The updated CT token and WSI token are concatenated, layer normalized, and aggregated to obtain the fused representation.

[0012] Furthermore, the adaptive fusion weights for generating CT experts, WSI experts, and fusion experts include: A CT quality vector and a WSI quality vector are constructed. The CT quality vector is obtained based on the confidence statistics of tumor segmentation, and the WSI quality vector is obtained based on image quality control statistics and attention distribution concentration. The CT quality vector, the WSI quality vector, the mismatch score, and the difference vector are concatenated and input into a routing network composed of a softmax function. The output is three weight values ​​corresponding to the fusion expert, the CT expert, and the WSI expert, respectively.

[0013] Furthermore, the training phase of the method is optimized using a combined loss function, which includes: weighted binary cross-entropy loss, partial AUC (pAUC) loss for regions with low false positive rates, Align-Guard regularization loss for aligning cross-modal representations, and Co-teaching loss for resisting label noise; and, during the inference phase, the Align-Guard regularization term, the Co-teaching mechanism, and the pAUC loss are removed.

[0014] Furthermore, the difference vector and the mismatch score are calculated as follows: Difference vector Calculated using the following formula: ; in, For differential embedding networks, For global CT characterization, For WSI global representation; Mismatched scores Calculated using the following formula ;in, For output The scoring head, The sigmoid function has the following pre-gated interaction coefficients: .

[0015] Furthermore, the adaptive fusion weights of the fusion expert, CT expert, and WSI expert are calculated through the following routing network:

[0016] in, For CT quality vector, For WSI quality vector, For mismatched scores, For the difference vector, For routing networks.

[0017] Furthermore, the residual logit fusion formula is used for calculation: ; ; in, To integrate the prediction results, For CT-only prediction results, For WSI-only prediction results, , , These are adaptive fusion weights for fusion experts, CT experts, and WSI experts, respectively. For the sigmoid function, To integrate with Logit, To predict the probability of TRG0 in the final calculation; The combined loss function is specifically expressed as follows: ; in, For weighted binary cross-entropy loss, False positive rate Partial AUC proxy loss over the interval For Align-Guard alignment regularization loss, To compensate for the losses in co-teaching, , , These are the corresponding equilibrium hyperparameters.

[0018] (III) Beneficial Effects Compared with existing technologies, this invention provides an alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method, which has the following advantages: This invention generates pre-gated interaction coefficients by dynamically calculating the degree of cross-modal mismatch and combines them with an adaptive fusion weighting mechanism. This effectively coordinates the reliability differences between CT and WSI modalities, solves the prediction instability problem caused by data distribution offset and supervision signal mismatch, and has significantly improved robustness and clinical applicability with a low false positive rate. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall framework of the DRGFuse of the present invention; Figure 2 This is a schematic diagram comparing the single-modal baseline of the present invention with the method of the present invention across all indicators; Figure 3 This is a schematic diagram of the diagnostic analysis for low-FPR deployment according to the present invention; Figure 4 This is a schematic diagram showing a quantitative comparison between the present invention and existing multimodal methods on three datasets; Figure 5 This is a schematic representation of the ablation study of the key components of the DRGFuse of the present invention; Figure 6 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example Traditional stratification methods for response after neoadjuvant therapy in esophageal cancer suffer from unreliability in predicting TRG0 based on preoperative CT data and whole-slice biopsy images (WSI). Furthermore, structural mismatches between supervision and preoperative evidence make it difficult to maintain a low false positive rate operating point stably across centers. Additionally, mainstream multimodal fusion methods may amplify spurious correlations when modal evidence is inconsistent, leading to unstable performance with low false positive rates.

[0022] In this regard, such as Figure 1-6 As shown, this application proposes an alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method, comprising the following steps: Acquire pre-treatment three-dimensional CT volume data of the subjects and their corresponding preoperative biopsy whole slice images (WSI); Extract CT representations from the CT volume data, which include at least one CT token and a global CT representation. Extract the WSI representation from the WSI, which includes at least one WSI token and a global WSI representation; Based on the CT global representation and the WSI global representation, the degree of cross-modal mismatch is calculated, and a pre-gated interaction coefficient is generated, wherein the pre-gated interaction coefficient is negatively correlated with the degree of mismatch. Based on the pre-gated interaction coefficient, cross-modal interaction and fusion of the CT token and the WSI token are performed to generate a fused representation; Based on the CT representation, the WSI representation, and the degree of mismatch, adaptive fusion weights for CT experts, WSI experts, and fusion experts are generated. Based on the adaptive fusion weight, residual fusion is performed on the CT-only prediction results, WSI-only prediction results, and fusion prediction results based on the fusion representation to obtain the final prediction result. This final prediction result is used to output the prediction probability value representing the possibility of TRG0, which is an intermediate result for medical personnel to refer to.

[0023] This embodiment provides an alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method. The method first acquires the subject's pre-treatment three-dimensional CT volume data and its corresponding preoperative biopsy whole-slice image (WSI). For example, the CT volume data can be acquired through scanning with medical imaging equipment, while the WSI can be obtained through a digital slide scanner in a pathology department. After acquisition, these data typically require quality checks and format conversion to ensure compatibility with subsequent processing.

[0024] Subsequently, CT representations are extracted from the CT volume data. These CT representations include at least one CT token and a global CT representation. In one implementation, the extraction of CT representations can be performed by manually selecting specific regions in the CT volume data and calculating the statistical characteristics of these regions as CT tokens, while simultaneously using the global statistical characteristics of the entire CT volume data as the global CT representation. This manual extraction method may rely on the operator's experience and is difficult to capture complex spatial patterns.

[0025] Simultaneously, a WSI representation is extracted from the WSI, which includes at least one WSI token and a global WSI representation. For example, WSI representation extraction could involve uniformly dividing the WSI into blocks, calculating the color histogram or texture features of each image block as a WSI token, and then using the average of the features from all image blocks as the global WSI representation. This method of uniform block division and feature extraction may not effectively distinguish pathological areas from normal tissue and may also overlook local pathological information.

[0026] Furthermore, based on the CT global representation and the WSI global representation, the degree of cross-modal mismatch is calculated, and a pre-gated interaction coefficient is generated. This pre-gated interaction coefficient is negatively correlated with the degree of mismatch. Specifically, the degree of mismatch can be measured by comparing the Euclidean distance or cosine similarity between the CT global representation and the WSI global representation, and then the reciprocal of this distance or similarity is used as the pre-gated interaction coefficient. This distance metric may not capture complex semantic mismatches between different modalities.

[0027] Based on this, cross-modal interaction and fusion are performed on the CT token and the WSI token using the pre-gated interaction coefficients to generate a fused representation. For example, a weighted average approach can be used for interaction and fusion, i.e., a linear combination of the CT token and WSI token based on the pre-gated interaction coefficients is performed to obtain the fused representation. However, this linear combination may not be able to uncover the deep nonlinear complementary information between the two modalities.

[0028] Furthermore, based on the CT representation, the WSI representation, and the degree of mismatch, adaptive fusion weights are generated for the CT expert, WSI expert, and fusion expert. One implementation involves assigning weight combinations to the CT expert, WSI expert, and fusion expert according to preset rules for the global CT representation, global WSI representation, and degree of mismatch. However, such preset rules may not be flexible enough to adapt to differences between different patients or different data batches.

[0029] Finally, based on the adaptive fusion weights, residual fusion is performed on the CT-only prediction results, WSI-only prediction results, and the fusion prediction results based on this fusion representation to obtain the final prediction result. This final prediction result is used to output a predicted probability value representing the likelihood of TRG0, serving as an intermediate result for medical personnel's reference. For example, the CT-only prediction results, WSI-only prediction results, and fusion prediction results can be directly weighted and summed to obtain the final prediction result. However, this direct weighted summation may not take advantage of the strengths of different experts in different contexts and may also be sensitive to outliers.

[0030] The method proposed in this embodiment acquires CT volume data and WSI (Warranty-Side Interface) and extracts multi-level representations from them. It then calculates the degree of cross-modal mismatch to generate pre-gated interaction coefficients, achieving reliable gated fusion of CT tokens and WSI tokens. Furthermore, this method adaptively generates expert fusion weights based on multi-modal representations and the degree of mismatch, performing residual fusion of prediction results from different experts. Therefore, this method can predict the TRG0 status after neoadjuvant therapy for esophageal cancer preoperatively, reducing the risk of undertreatment or undermonitoring due to misjudgment of TRG0, and exhibits robustness in cross-center shift and missing modality scenarios.

[0031] In some embodiments described above in this application, CT representations are proposed for extraction from CT volume data, wherein the CT representations include at least one CT token and a global CT representation. However, in CT volume data, the heterogeneity of the tumor region and its surrounding environment is of great significance for disease diagnosis and prognosis. If only general feature extraction methods are used, these key local spatial information may not be fully captured, thereby affecting the accuracy and reliability of subsequent cross-modal fusion and prediction.

[0032] To address this, this application further proposes a specific method for extracting CT representations from CT volume data, comprising the following steps: First, tumor region segmentation is performed on the CT volume data to obtain the region of interest (ROI). This step aims to accurately identify and locate tumor lesions in the CT image, laying the foundation for subsequent refined feature extraction. Tumor region segmentation can be achieved through various image processing and machine learning techniques. For example, deep learning-based segmentation models (such as U-Net, V-Net, etc.) can be used. These models, trained on a large amount of labeled data, can automatically identify and delineate the boundaries of the tumor. Alternatively, semi-automatic or manual segmentation can be performed in conjunction with the expertise of radiologists to ensure the accuracy of the tumor ROI.

[0033] Secondly, based on the tumor ROI, multiple habit masks representing different spatial extents are constructed. These habit masks include a core mask representing the tumor interior, a rim mask representing the tumor boundary, and a ring mask representing the region surrounding the tumor. This step is one of the core innovations of this method, recognizing that tumors and their microenvironment are not homogeneous, and different regions may contain different biological information. The core mask is typically generated by morphological erosion of the tumor ROI or by selecting its central region, representing the main part of the tumor. The rim mask is defined by the difference between the tumor ROI after expansion and the original ROI, representing the interface between the tumor and surrounding tissues, reflecting the invasiveness of the tumor. The ring mask is further generated by the difference between the rim mask or the tumor ROI after a larger expansion operation and the rim mask, representing the normal or diseased tissue region surrounding the tumor, reflecting the characteristics of the tumor microenvironment. These habit masks can subdivide the tumor region into subregions with clear spatial semantics, thereby capturing the heterogeneity of the tumor interior and surrounding area.

[0034] Next, a pre-trained 3D visual encoder is used to extract voxel-level feature maps from the CT volume data. This 3D visual encoder is typically a deep neural network, such as a 3D convolutional neural network (3D CNN) or a 3D VisionTransformer, capable of learning and extracting rich, multi-scale voxel-level features from the raw 3D CT volume data. The pre-trained encoder, trained on a large amount of medical image data, has learned general image feature representation capabilities, which helps improve the efficiency and generalization of feature extraction. The voxel-level feature maps preserve the spatial information of the CT volume data, providing a foundation for subsequent local feature aggregation.

[0035] Finally, pooling is performed on the voxel-level feature maps within each habitat mask to obtain the corresponding CT token. The pooling operation aims to aggregate the voxel-level features within the region covered by each habitat mask into a compact vector, i.e., the CT token. Commonly used pooling methods include average pooling, max pooling, or attention pooling. In this way, each CT token can highly summarize the feature information of the specific spatial region represented by its corresponding habitat mask. For example, the pooling result within the core mask forms the core token, representing the features inside the tumor; the pooling result within the rim mask forms the rim token, representing the features of the tumor boundary; and the pooling result within the ring mask forms the ring token, representing the features of the region surrounding the tumor.

[0036] By employing the aforementioned technical solution, this application overcomes the limitations of traditional CT feature extraction methods, namely, their inability to fully capture the fine-grained spatial heterogeneity of tumors and their microenvironments. Through precise tumor region segmentation and the construction of multiple habitat masks, this method can extract CT tokens with clear spatial semantics from CT volume data. These tokens represent key information about the tumor's interior, boundaries, and surrounding areas. This refined feature extraction approach enables subsequent cross-modal interaction and fusion to utilize more clinically meaningful and biologically relevant CT representations, thereby significantly improving the richness and accuracy of the fused representations. This, in turn, enhances the reliability and clinical guidance value of the final prediction results, providing a more solid data foundation for predicting the tumor regression grade (TRG0) of subjects.

[0037] This application further proposes a specific method for extracting WSI representations from the WSI, including: segmenting and dividing the WSI into tissue regions to obtain multiple image patch patches; encoding each patch into a semantic token using a pre-trained pathological baseline model; scoring the importance of each semantic token and retaining the top K tokens with the highest scores; using the state-space sequence model MambaMIL to perform contextual modeling on the retained K tokens, and obtaining the global representation of the WSI and the WSI-only prediction results through multi-instance learning MIL pooling.

[0038] Specifically, the WSI (Wide Image Sensor) is segmented and partitioned into tissue regions to efficiently process the data. Due to the enormous size of WSI images, direct processing is computationally intensive, and they contain numerous non-tissue areas (such as background and ink blots). Tissue region segmentation uses image processing techniques (e.g., color thresholding, morphological manipulation, or deep learning semantic segmentation) to identify and extract effective regions containing biological tissue. Subsequently, within these effective regions, the WSI is partitioned into blocks of a preset size (e.g., 256x256 pixels or 512x512 pixels) and resolution (e.g., 20x or 40x), generating a series of manageable image patch blocks. This step effectively narrows the processing scope, focusing on pathology-related information.

[0039] Building upon this, each patch is encoded into a semantic token using a pre-trained pathology baseline model. The pathology baseline model is typically a deep learning model pre-trained on a large-scale pathology image dataset, such as the Vision Transformer (ViT) based on the Transformer architecture or an advanced convolutional neural network (CNN). These models are capable of extracting high-dimensional vector representations with rich semantic information—i.e., semantic tokens—from each image patch. By leveraging the powerful feature extraction capabilities of the pre-trained model, key information such as cell morphology, tissue structure, and pathological patterns can be effectively captured, avoiding the resource consumption of training from scratch and improving the generalization ability of features.

[0040] To further optimize information processing, each semantic token is scored for importance, and the top K tokens with the highest scores are retained. In WSI, not all image patches are equally important to the final prediction result; some patches may contain normal tissue, while others contain key tumor regions or microenvironment features. The importance scoring mechanism evaluates each semantic token through a lightweight network (e.g., an attention module or a simple fully connected layer), outputting a scalar score that reflects the token's potential contribution to the prediction task. By selecting the top K tokens with the highest scores, redundant or irrelevant background information can be effectively filtered out, allowing the model to focus more on the most discriminative regions, thereby reducing computational complexity and improving model efficiency and accuracy. The value of K is an adjustable parameter that can be optimized according to the specific task and dataset.

[0041] Subsequently, the state-space sequence model MambaMIL is used to perform contextual modeling on the retained K tokens, and the global representation of the WSI and the WSI-only prediction results are obtained through multi-instance learning (MIL) pooling. MambaMIL is an efficient state-space sequence model that can effectively capture long-distance dependencies and complex spatial contextual information among the retained K semantic tokens, overcoming the computational bottleneck of traditional self-attention mechanisms when processing long sequences. Through contextual modeling, the model can understand the interactions between different important regions, thus forming a more comprehensive pathological understanding. After MambaMIL processing, these context-enhanced tokens are aggregated into a single global representation of the WSI through multi-instance learning (MIL) pooling. MIL pooling methods can employ attention-based MIL, max-pooling MIL, or gated attention pooling, aiming to identify and aggregate the most discriminative information from multiple instances, ultimately generating a global feature vector representing the entire WSI. Based on this global representation, a WSI-only prediction result, i.e., the probability that the WSI is predicted as TRG0, can be obtained through a classification head (e.g., a fully connected layer and a Sigmoid activation function).

[0042] This application further proposes a specific method for calculating the degree of cross-modal mismatch and generating pre-gated interaction coefficients, which includes: concatenating the CT global representation and the WSI global representation, calculating the difference and the element-wise product, inputting the result into a difference embedding network to obtain a difference vector; inputting the difference vector into a mismatch scoring network, and mapping it through the sigmoid function to obtain a scalar mismatch score; setting the pre-gated interaction coefficient to be equal to 1 minus the mismatch score, so that the higher the mismatch score, the lower the pre-gated interaction coefficient.

[0043] Specifically, to comprehensively capture the relationship between the global representations of CT and WSI, they are first subjected to multi-faceted feature processing. The "concatenation" operation directly joins the global representations of the two modalities to form a joint vector containing all original information, providing a comprehensive input view for subsequent networks. "Difference calculation" directly quantifies the degree and direction of their differences in the feature space by calculating the vector difference between the two global representations, highlighting inconsistencies. "Element-wise product calculation" multiplies the two global representations element-wise, aiming to capture common activation patterns or synergistic effects between them, highlighting consistency or complementarity. These operations collectively provide rich and multi-dimensional intermodal relationship information for subsequent differential embedding networks, including original features, difference features, and consistency features, thus laying the foundation for a comprehensive assessment of cross-modal mismatch.

[0044] Subsequently, the composite result obtained from the above concatenation, difference calculation, and element-wise product calculation is input into the differential embedding network. The differential embedding network is a neural network module whose main function is to perform nonlinear transformation and dimensionality reduction on these multi-dimensional relational information, thereby extracting core features that can effectively represent cross-modal differences and generating a compact and discriminative difference vector. This network typically consists of one or more fully connected layers, capable of learning and compressing these complex intermodal relationships into a more semantically meaningful vector representation. This difference vector can capture deep mismatch patterns in semantics and information content between the global representations of CT and WSI, providing high-quality input for subsequent mismatch scoring.

[0045] Next, the difference vector is input into the mismatch scoring network. The mismatch scoring network is a specially designed neural network module that receives the difference vector as input and outputs a scalar value representing the degree of mismatch between the CT global representation and the WSI global representation. This network typically contains one or more fully connected layers, ultimately activated by a sigmoid function. The sigmoid function maps the network's output value to a probability range between 0 and 1, thus quantifying the degree of mismatch into a standardized mismatch score. Here, 0 represents a perfect match, 1 represents a complete mismatch, and intermediate values ​​represent varying degrees of mismatch. This standardization process makes the mismatch score highly interpretable and comparable.

[0046] Finally, the pre-gated interaction coefficient is generated by subtracting the mismatch score from 1. This negative correlation ensures that the higher the mismatch score between the CT global representation and the WSI global representation, the lower the pre-gated interaction coefficient. Conversely, the lower the mismatch score (i.e., the more matched the modalities), the higher the pre-gated interaction coefficient. This coefficient acts as a gate in subsequent cross-modal interaction and fusion processes, dynamically adjusting the information interaction strength between different modal tokens.

[0047] This application further proposes a method for cross-modal interaction and fusion of CT token and WSI token to generate a fused representation, which includes: weighting the cross-modal projection result using the pre-gated interaction coefficient to achieve pre-gating; inputting the weighted CT token and WSI token into the Cross-Mamba module to update their respective context information; and concatenating, layer normalizing and aggregating the updated CT token and WSI token to obtain the fused representation.

[0048] Specifically, the pre-gated interaction coefficients are used to weight the results of cross-modal projection. The pre-gating step aims to dynamically adjust the weights of the CT and WSI modalities during cross-modal information interaction based on the degree of mismatch between them. The pre-gated interaction coefficients (whose values ​​are negatively correlated with the degree of mismatch between modalities) are applied to the cross-modal projection results of the CT token and WSI token. For example, after projecting the CT token and WSI token into a shared feature space, these coefficients can be used as multiplication factors, applied element-wise to the projected feature vectors. When the degree of mismatch between modalities is high, the pre-gated interaction coefficients are low, thereby reducing the influence of mismatched modal information in the interaction and minimizing the interference of potential noise or misleading information on the fusion process. Conversely, when the degree of matching between modalities is high, the pre-gated interaction coefficients are high, promoting the full exchange and fusion of information between modalities. This weighting mechanism ensures that subsequent interaction and fusion processes are more reliable and targeted.

[0049] Building upon this, the weighted CT token and WSI token are input into the Cross-Mamba module to update their respective context information. The Cross-Mamba module is an advanced State-Space Model (SSM) architecture whose core advantage lies in its ability to efficiently process sequential data and capture long-range dependencies. Here, the pre-gated weighted CT token and WSI token are input into the Cross-Mamba module to achieve deep and context-aware cross-modal interaction. This module can be designed to contain two Mamba branches, processing the CT token and WSI token respectively, and exchanging information through a cross-attention mechanism or a shared state space mechanism. For example, when the CT token is processed through its Mamba branch, the WSI token can be introduced as the key and value of the cross-attention mechanism, allowing the CT token to learn and update its context information from the WSI token, and vice versa. The selective scanning mechanism unique to the Mamba model enables it to dynamically adjust its state updates based on the input content, thereby efficiently capturing long-range dependencies and rich context information, ensuring that the complementary advantages of the two modalities are fully utilized in cross-modal interaction.

[0050] Subsequently, the updated CT token and WSI token are concatenated, layer-normalized, and aggregated to obtain the fused representation. After the Cross-Mamba module completes the context information update of the CT and WSI tokens, these updated tokens already contain rich cross-modal complementary information. First, the updated CT and WSI tokens are concatenated along the feature dimension to form a longer feature vector that integrates local features and contextual information from both modalities. Next, layer-normalization is performed on the concatenated feature vector, which helps stabilize the model training process, accelerate convergence, and improve the model's generalization ability. Layer-normalization involves independently normalizing the features of each sample to achieve a mean of 0 and a variance of 1. Finally, the normalized feature vectors are integrated into a single, compact fused representation through an aggregation operation. Aggregation methods can include, but are not limited to, average pooling, max pooling, or mapping the concatenated feature vectors to a fixed-dimensional fused representation using a small fully connected network. This fused representation is comprehensive and of high quality, providing a powerful input for subsequent tumor regression grading prediction tasks.

[0051] This application further proposes a method for generating adaptive fusion weights for CT experts, WSI experts, and fusion experts. This method includes constructing CT quality vectors and WSI quality vectors. The CT quality vector aims to quantify the reliability of tumor region segmentation in CT volumetric data. In CT image analysis, accurate segmentation of tumor regions is fundamental for subsequent feature extraction and diagnosis. The CT quality vector can be obtained based on various confidence statistics methods. For example, during tumor segmentation, the segmentation probability map output by a deep learning model can be used to calculate the average probability or entropy value of the probability distribution of pixels within the tumor region; higher probability or lower entropy generally indicates higher segmentation confidence. Alternatively, quality can be indirectly reflected by evaluating the consistency between the segmentation results and expert annotations (e.g., the Dice coefficient), or by using the model's uncertainty estimation module (e.g., Monte Carlo Dropout) to quantify the uncertainty of the segmentation results. These statistics are encoded into a vector to reflect the intrinsic quality of the CT modality data. The WSI quality vector is used to evaluate the overall quality and information validity of the WSI in the whole-slice image. Image quality control statistics can include quantitative evaluations of WSI sharpness, contrast, staining uniformity, and the presence of artifacts (e.g., bubbles, folds, stains). For example, this can be measured by calculating metrics such as local variance, edge intensity, and color histogram distribution of the image. Attention distribution concentration reflects whether the model's attention mechanism effectively focuses on pathologically important areas when processing WSI.

[0052] Based on this, the CT quality vector, the WSI quality vector, the mismatch score, and the difference vector are concatenated to provide comprehensive input for the subsequent routing network. The CT quality vector and WSI quality vector provide intrinsic reliability information for each modality; the mismatch score quantifies the semantic difference between the global CT representation and the global WSI representation, reflecting the consistency of cross-modal information; and the difference vector provides finer-grained cross-modal difference information. Through the concatenation operation, these heterogeneous information vectors are combined into a unified feature vector, enabling the routing network to comprehensively consider modal quality, inter-modal consistency, and differences, thereby making more informed weight allocation decisions. Subsequently, the concatenated vector is input into the routing network, which is constructed using a softmax function. The routing network is a specially designed neural network module whose function is to dynamically calculate the contribution weights of each "expert" based on the input comprehensive feature vector. This network typically consists of one or more fully connected layers used to learn the complex mapping relationship between input features and output weights. Finally, the network's output layer applies a softmax function to transform the network's raw output (logit value) into a probability distribution, ensuring that the sum of all output weights is 1, and that each weight value is between 0 and 1. This design allows the routing network to adaptively adjust the influence of different experts (fusion expert, CT expert, and WSI expert) in the final prediction based on the characteristics of the current sample. The final output of the routing network corresponds to the three weight values ​​for the fusion expert, the CT expert, and the WSI expert, respectively. These weight values ​​are dynamically generated and vary based on the specific quality of each subject's CT and WSI data, as well as the degree of mismatch between modalities.

[0053] This application further proposes to optimize the method during the training phase using a combined loss function, which includes: weighted binary cross-entropy loss, partial AUC pAUC loss for regions with low false positive rates, Align-Guard regularization loss for aligning cross-modal representations, and co-teaching loss for resisting label noise; and, during the inference phase, the Align-Guard regularization term, co-teaching mechanism, and pAUC loss are removed.

[0054] Specifically, the combined loss function is a function that sums multiple independent loss terms with weights. Its purpose is to optimize multiple objectives simultaneously during model training, thereby enabling the model to learn more comprehensive and robust feature representations and predictive capabilities. In this way, the model can not only focus on overall prediction accuracy, but also take into account specific clinical needs (such as low false positive rate) and data quality issues (such as label noise and modality alignment).

[0055] The weighted binary cross-entropy loss is an improvement on the class imbalance problem in binary classification tasks. In medical imaging diagnosis, for example, the number of samples in the tumor regression grade TRG0 may be far fewer than the number of non-TRG0 samples. If the standard cross-entropy loss is used directly, the model may be biased in predicting the majority class. By assigning higher weights to samples in the minority class (e.g., TRG0), this loss function can encourage the model to pay more attention to the prediction accuracy of the minority class, thereby effectively mitigating the negative impact of class imbalance and improving the model's ability to identify key pathological states.

[0056] The partial AUC (pAUC) loss, designed for regions with low false positive rates, aims to optimize model performance in specific clinical scenarios. In many medical diagnostic applications, false positives (misdiagnosing healthy individuals as having the disease) can lead to unnecessary follow-up examinations, patient anxiety, and even overtreatment. Therefore, strictly controlling the false positive rate while maintaining a high true positive rate is crucial. The pAUC loss, optimized by focusing on regions with low false positive rates in the receiver operating characteristic (ROC) curve, directly guides the model to learn decision boundaries with stronger discriminative power at low false positive rates, thereby effectively reducing the risk of misdiagnosis and improving the model's clinical applicability.

[0057] The Align-Guard regularization loss term used for aligning cross-modal representations addresses potential semantic inconsistencies or information redundancy between CT and WSI representations during fusion. Although CT and WSI provide complementary information, they differ significantly in spatial resolution, information type, and noise patterns. The Align-Guard regularization term introduces additional constraints, encouraging the model to learn feature representations with higher consistency and comparability across different modalities. For example, it can penalize unreasonable distances or differences in the cross-modal feature space, ensuring that the fused representation can more effectively integrate complementary information from both modalities and avoid predictive performance degradation due to inter-modal mismatch.

[0058] The co-teaching loss described above, designed to combat label noise, aims to improve the model's robustness to inaccurate training data labels. In real-world medical data acquisition, labeling may be subjective or erroneous. Co-teaching mechanisms typically involve two independent models that learn from and correct each other during training. Each model selects samples it deems "clean" (i.e., with minimal loss) based on its own predictions and uses these samples to guide the learning of the other model. This mutual filtering and learning mechanism enables the model to effectively identify and mitigate the negative impact of noisy labels, thereby learning more accurate and generalized feature representations.

[0059] This application further proposes a difference vector. The specific calculation method for the mismatch score m. Difference vector. Calculated using the following formula: ;in, For differential embedding networks, For global CT characterization, For WSI global representation; mismatch score Calculated using the following formula ;in, For output The scoring head, The sigmoid function has the following pre-gated interaction coefficients: .

[0060] Difference vector The calculation aims to comprehensively capture the global characterization of CT. and WSI global representation The relationship between them. Specifically, it is through... , The difference between the two and their element-wise product The data is then concatenated and input into a differential embedding network. This approach not only incorporates the raw information of both modalities but also explicitly encodes their differences and similarities, providing a rich and comprehensive feature base for subsequent mismatch assessment. (Differential Embedding Network) The purpose is to map these concatenated original features into a more compact and discriminative feature space to extract the latent information most relevant to the differences between modalities. This network can be implemented using neural network structures such as multilayer perceptrons (MLPs), which learn to transform high-dimensional inputs into low- or medium-dimensional difference vectors.

[0061] Mismatched scores The calculation is to use the above difference vector This is transformed into a scalar value to quantify the degree of mismatch between modes. This is done by converting the difference vector... Score head from input to output m Then it passes through the sigmoid function. This is achieved through mapping. (Scoring head) Typically, it is a simple linear layer or a small neural network whose function is to convert the difference vectors. The complex information is aggregated into a single numerical value that reflects the degree of mismatch. This is the sigmoid function. Then the scoring head The output value is compressed to the range of 0 to 1, so that the mismatch score m has the interpretation of a probability or normalized score, where 0 represents a perfect match and 1 represents a perfect mismatch.

[0062] Pre-gated interaction coefficient The calculation method is as follows This ensures a clear negative correlation between the coefficient and the mismatch score m. In other words, when the CT global characterization... and WSI global representation The higher the degree of mismatch m between them, the higher the pre-gated interaction coefficient. The higher the value, the lower the interaction coefficient, and vice versa. This design makes the pre-gated interaction coefficient lower. It can directly reflect the reliability of intermodal alignment, providing a dynamic and adaptive gating mechanism for subsequent cross-modal interaction and fusion.

[0063] Through the above technical solution, this application provides a precise and robust mechanism to quantify the degree of mismatch between CT and WSI global representations and generate corresponding pre-gated interaction coefficients. Specifically, by concatenating the CT global representation, WSI global representation, their difference, and their element-wise product, and inputting them into a difference embedding network, it is possible to comprehensively capture the differences and commonalities between modalities, thereby generating a discriminative difference vector. Based on this, a scoring head and a sigmoid function are used to transform the difference vector into a mismatch score between 0 and 1, achieving standardized quantification of the alignment quality between modalities. This explicit mathematical formula and network structure make the calculation of the degree of mismatch no longer ambiguous, but interpretable and operable. Finally, by defining the pre-gated interaction coefficient as 1 minus the mismatch score, it is ensured that when the degree of mismatch between modalities is high, the interaction coefficient automatically decreases, thereby effectively suppressing noise or misleading information that may be introduced between mismatched modalities; conversely, when the degree of matching between modalities is high, the interaction coefficient increases, promoting more sufficient cross-modal information interaction and fusion. This adaptive gating mechanism significantly improves the accuracy and robustness of fusion prediction, especially when faced with CT and WSI data of different quality or inherent heterogeneity. It can more intelligently integrate information and avoid prediction bias caused by inconsistencies between modalities.

[0064] This application further proposes that the adaptive fusion weights of the fusion expert, CT expert, and WSI expert are calculated through the following routing network: ; in, For CT quality vector, For WSI quality vector, For mismatched scores, For the difference vector, For routing networks.

[0065] Through the above technical solution, the CT mass vector WSI quality vector Mismatched scores and difference vector This multi-dimensional, multi-granular information is processed through a learnable routing network. A nonlinear mapping is performed, and finally normalization is achieved through the softmax function, thereby accurately calculating the adaptive fusion weights of the fusion expert, CT expert, and WSI expert. , , This explicit mathematical formula and network structure resolves the ambiguity in weight allocation that can arise when relying solely on input features and routing networks. Specifically, the use of the softmax function ensures the accuracy of the generated weights. , , The values ​​are always positive and sum to 1, providing a clear interpretation of probability or contribution, thus avoiding unreasonable or unstable weight allocation. Simultaneously, the routing network... It can learn complex nonlinear relationships, enabling the model to intelligently adjust the contribution ratio of each expert based on the actual quality of CT and WSI data, as well as the degree of matching and differences between them.

[0066] This application further proposes the calculation method for the final prediction result and the specific composition of the combined loss function. Specifically, To integrate the logit formula The calculation yielded, where To integrate the prediction results, For CT-only prediction results, For WSI-only prediction results, , , These are adaptive fusion weights for the fusion expert, CT expert, and WSI expert, respectively. The logit value after fusion is obtained. Then, through the sigmoid function Map it to a probability value between 0 and 1 , To ultimately predict the probability of TRG0, i.e. The probability value It directly indicates the likelihood that the subject is classified as having a tumor regression grade of TRG0. In fact, it's a fused logit; the final prediction is the probability obtained through a sigmoid mapping. This residual logit fusion method allows the model to dynamically adjust the contributions of different experts based on their reliability, thereby achieving information complementarity and advantage integration while maintaining the independent judgment capabilities of each expert. For example, when CT data is of high quality but WSI data contains artifacts, the weight of CT experts may be increased accordingly to reduce the potential negative impact of WSI experts.

[0067] Meanwhile, the training phase of this application employs a combined loss function `L` for optimization, which is specifically expressed as: .in, The weighted binary cross-entropy loss is used as a basic loss term to measure the difference between the model's predicted probability and the true label. By introducing weights, the common class imbalance problem in medical data can be effectively solved. For example, TRG0 cases may be far fewer than non-TRG0 cases, ensuring that the model can effectively learn the minority class as well. This is a partial AUC surrogate loss for the false positive rate ≤ β interval, designed to address the specific performance requirements in medical diagnostics for regions with low false positive rates. In some clinical scenarios, the cost of false positives can be very high, therefore the model needs to strictly control the false positive rate while maintaining high recall. This loss term optimizes part of the AUC, enabling the model to maintain excellent performance even when the false positive rate is below a specific threshold β. The Align-Guard alignment regularization loss is used to force CT and WSI representations to maintain consistency in the latent space, reducing cross-modal differences. This helps ensure that information from different modalities can be effectively fused, avoiding poor fusion results due to excessive differences in distribution between modalities. For example, it can promote alignment by minimizing the distance between features from different modalities. Co-teaching loss is used to improve the robustness of models to label noise. In medical data annotation, label noise may exist due to differences in expert judgment or data quality issues. The co-teaching mechanism reduces the negative impact of noise on model training and improves the model's generalization ability by allowing two networks to learn from each other and filter out samples that the other considers to be noise. , , These are balancing hyperparameters used to adjust the contribution ratio of each loss term to the total loss. By setting these hyperparameters appropriately, a balance can be achieved between different optimization objectives, thereby achieving better overall model performance.

[0068] The following example will provide a more detailed explanation of the above technical solution: A medical institution is evaluating esophageal cancer patient A, who has completed neoadjuvant therapy, and needs to determine whether their tumor regression grade has reached TRG0 (i.e., no residual tumor). Accurate TRG0 determination is crucial for subsequent treatment decisions (such as whether to perform surgery or only follow-up), especially under the strict constraint of a low false positive rate (low-FPR), to avoid misclassifying non-TRG0 patients as TRG0, leading to undertreatment. However, traditional prediction methods often face challenges such as mismatch between preoperative CT and whole-slice biopsy (WSI) evidence and postoperative TRG determination criteria, susceptibility of multimodal information fusion to irrelevant interference, and insufficient predictive robustness due to multicenter data distribution bias.

[0069] To address these issues, the medical institution employed an alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method. The specific process is as follows: First, we acquired the pre-treatment 3D CT volume data of subject A and its corresponding preoperative whole-slice biopsy images (WSI). These are the raw input data for TRG0 prediction.

[0070] Next, CT representations are extracted from the CT volume data. Specifically, tumor regions are segmented from the CT volume data to obtain the region of interest (ROI). Based on this ROI, multiple habitat masks representing different spatial extents are constructed, such as a core mask representing the tumor interior, a rim mask representing the tumor boundary, and a ring mask representing the region surrounding the tumor. Subsequently, a pre-trained 3D visual encoder is used to extract voxel-level feature maps from the CT volume data. Pooling is performed on the voxel-level feature maps within each habitat mask to obtain the corresponding CT tokens. Simultaneously, the global CT representation is obtained by aggregating these CT tokens or extracting them from the encoder output. This multi-habitat mask strategy can more comprehensively and stably capture the imaging features of the tumor and its microenvironment, reducing the impact of tumor segmentation uncertainty and helping to solve the problem of preoperative evidence localization.

[0071] Simultaneously, WSI representations are extracted from WSI. Specifically, the WSI is segmented and partitioned into tissue regions, resulting in multiple image patches. Each patch is encoded as a semantic token using a pre-trained pathological baseline model. To focus on the most discriminative regions, each semantic token is scored for importance, and the top K tokens with the highest scores are retained. Subsequently, the state-space sequence model MambaMIL is used to perform contextual modeling on the retained K tokens, and MIL pooling operations are learned through multi-instance learning to obtain global WSI representations and WSI-only prediction results. Using a pathological baseline model and MambaMIL for token extraction and contextual modeling enables more effective extraction of transferable histological features from local and fragmented evidence of biopsy WSI, improving the quality and reliability of WSI representations.

[0072] Subsequently, based on CT global characterization and WSI global representation The cross-modal mismatch degree is calculated, and pre-gated interaction coefficients are generated. Specifically, the operation involves... and The process involves splicing, difference calculation, and element-wise product calculation, and the results are then input into the difference embedding network. To obtain the difference vector Next, the difference vector Input mismatch scoring network The scalar mismatch fraction *m* is obtained by mapping using the sigmoid function σ. The pre-gated interaction coefficient α is set to 1 minus the mismatch fraction. ,Right now This means that the higher the degree of mismatch between the CT and WSI modes ( (The larger the value), the pre-gated interaction coefficient The lower the value, the better. This dynamic calculation of mismatch and generation of pre-gated interaction coefficients can intelligently identify and quantify potential conflicts or differences between CT and WSI evidence, providing guidance for subsequent fusion steps and effectively suppressing spurious correlations amplified by inconsistencies caused by sampling differences and center differences.

[0073] Based on pre-gated interaction coefficients Cross-modal interaction and fusion of CT tokens and WSI tokens are performed to generate a fused representation. Specifically, using... The results of cross-modal projection are weighted to achieve pre-gating. Then, the weighted CT token and WSI token are input into the Cross-Mamba module to update their respective context information. Finally, the updated CT token and WSI token are concatenated, layer normalized, and aggregated to obtain the fused representation. This pre-gated cross-modal interaction and fusion mechanism avoids the problem of amplifying spurious correlations in traditional fusion methods when modal inconsistencies exist, ensuring that only reliable information is fully fused, thereby enhancing the discriminative ability of low FPR regions.

[0074] Next, based on CT representation, WSI representation, and the degree of mismatch, adaptive fusion weights for CT experts, WSI experts, and fusion experts are generated. Specifically, a CT quality vector is constructed. and WSI mass vector . The confidence score is obtained based on tumor segmentation and reflects the reliability of CT data. Based on image quality control statistics and attention distribution concentration, it reflects the reliability of WSI data. , Mismatched scores and difference vector The network is concatenated, and the input is a routing network constructed using the softmax function. The outputs correspond to three weight values ​​for fusion expert, CT expert, and WSI expert, respectively. These weights are adaptively generated and can dynamically adjust the contributions of different expert predictions based on the quality of the current subject A's CT and WSI data and the degree of intermodal mismatch, thereby effectively addressing the problems of multicenter data distribution bias and modal information inconsistency.

[0075] Finally, based on the adaptive fusion weights, the CT-only prediction results are analyzed. WSI-only prediction results And fusion prediction results based on fusion representation Residual fusion is performed to obtain the final prediction result. The fusion formula is: Final predicted probability via the sigmoid function Calculations show that This represents the probability that subject A is predicted to be TRG0. This residual logit fusion method, combined with adaptive weights, allows the model to fully utilize the independent predictive capabilities of each modality and the complementary information after fusion when making the final prediction. At the same time, it weights the predictions according to the reliability of each modality, further enhancing the robustness and accuracy of the predictions. Especially under the constraint of a low false positive rate, it can more reliably indicate whether subject A is TRG0, reducing the clinical risk of misjudging TRG0.

[0076] During the model training phase, a combined loss function is used for optimization. This function includes a weighted binary cross-entropy loss, a partial AUC (pAUC) loss for low false positive rate regions, an Align-Guard regularization term for aligning cross-modal representations, and a co-teaching loss for resisting label noise. These loss functions work together to ensure that the model not only focuses on overall prediction accuracy during training but also specifically optimizes performance in low FPR regions, enhancing the consistency of cross-modal representations and resistance to label noise. When making inference predictions for subject A, the Align-Guard regularization term, co-teaching mechanism, and pAUC loss are removed, and the model uses only the trained parameters for forward propagation to obtain the final TRG0 prediction result.

[0077] Using the above method, the medical institution can provide subject A with a robust prediction of TRG0 status after neoadjuvant therapy for esophageal cancer under the constraint of a low false positive rate, thereby providing auxiliary decision-making basis for the selection of surgical timing and the formulation of postoperative follow-up monitoring strategies.

[0078] Example: Implementation steps during use Step 1: Obtain pre-treatment 3D CT volume data and pre-treatment endoscopic biopsy WSI (H&E), and complete CT and WSI pairing based on patient identification. Obtain TRG 0–3 labels based on the pathological evaluation results of the en bloc resection specimen after surgery, and construct binary classification tasks TRG0 and TRG1–3, where TRG0 is the positive class.

[0079] Step 2: In the CT branch, the tumor region is segmented using nnU-Net to obtain the tumor ROI and softmask; based on the tumor region, three types of habit masks, core, rim and ring, are constructed to characterize the internal, boundary, and surrounding context regions of the tumor.

[0080] Step 3: Extract CT feature maps using the frozen 3D visual encoder and perform masked average pooling within each habitat mask to obtain CT tokens; then aggregate all CT tokens to obtain the global CT representation and CT-only logit, and construct the CT quality vector.

[0081] Step 4: In the WSI branch, first generate a tissue mask to remove the background, then split the patch according to the target magnification, filter low-quality patches through no-reference quality control, and finally encode the patch into a semantic token using the frozen UNI encoder.

[0082] Step 5: Assess the importance of WSI tokens and retain the Top-K tokens; use MambaMIL to model long-sequence tokens, and then use MIL pooling to obtain the global representation of WSI and WSI-only logit to construct the WSI quality vector.

[0083] Step 6: In DRGFuse Core, calculate the cross-modal difference vector and mismatch score based on the CT global representation and WSI global representation, and generate pre-gated interaction coefficients so that the higher the degree of cross-modal mismatch, the lower the intermodal interaction strength.

[0084] Step 7: Based on the pre-gated interaction coefficients, update the context information of the CT token and WSI token through cross-modal projection and the Cross-Mamba module, and then normalize and aggregate the fusion token to obtain the fusion representation and fusion logit.

[0085] Step 8: Fuse the CT quality vector, WSI quality vector, mismatch score, and difference vector to generate adaptive weights for the fusion expert, CT expert, and WSI expert. Then, use residual logit fusion to obtain the final predicted logit and TRG0 probabilities.

[0086] Step 9: During the training phase, weighted BCE loss, low-FPR pAUC loss, Align-Guard alignment constraints, and mismatch-aware co-teaching strategies were adopted to reduce the impact of label noise, cross-center distribution shift, and cross-modal semantic shift; during the inference phase, only the forward inference path between the encoder and DRGFuseCore was retained.

[0087] Please refer to Figures 1-6 As shown, the model structure of this application is as follows: (See model structure diagram). Figure 1(A) The inference phase of DRGFuse retains only the CT / WSI encoder and the DRG-Fuse core; (B) Fusion follows a single reliability-driven chain: mismatch estimation → pre-gated controlled interaction → post-gated routing and residual logit fusion; (C) Training-phase components (Align-Guard, Co-Teach, and low-FPR objectives) are disabled during inference. (D1 / D2) CT and WSI encoders.

[0088] CT and WSI Token Encoder: Frozen Backbone Network. This application freezes the nnU-Net, CT visual encoder (CT-FM), and UNI pathology encoder (no backpropagation); only a lightweight token scoring / aggregation module, reliability gating, fusion core, and prediction head are trained.

[0089] The CT Habitat Token Encoder. The CT branch first uses nnU-Net to obtain the tumor ROI and soft mask. Then, this application defines three habitat masks on the tumor and its surrounding context: core represents the eroded tumor interior, rim represents a thin boundary band, and ring represents the expanded tumor periphery band to characterize the surrounding tissue context. After extracting voxel-level feature maps F using CT-FM, for each habitat mask Mk, masked average pooling is performed on F within the mask to obtain the CT token tkct; all habitat tokens constitute... Global characterization is then obtained through lightweight polymerization. With CT-only logit This application also constructs an ROI quality vector. (such as ROI coverage, mask confidence statistics, etc.) are used to evaluate the reliability of CT after gating.

[0090] WSI Basic Token Encoder: WSI Branch Input Pre-treatment Biopsy WSI Bag This application first generates a tissue mask on a low-resolution thumbnail to remove the background, then cuts tiles according to the target magnification, and filters low-quality patches using a no-reference quality control to obtain a valid patch set. Each patch is embedded into semantics by a frozen UNI encoder. To obtain stable fixed-length representations, the model uses a lightweight network to score patch tokens and retains the top-Kp tokens. Subsequently, MambaMIL was used to model long sequences, the token context was updated through linear-time Mamba modules, and then slice-level representations were obtained through MIL pooling. With WSI-only logit This application also constructs a WSI quality vector. Comprehensive quality control statistics and reliability clues derived from the concentration of attention distribution.

[0091] Figure 2 The table below shows the experimental results in the comparison of the single-modal baseline with the complete method of this application across all metrics (bold indicates better). The lack of demographic and clinical information, as well as the inconsistency of cross-modal evidence between preoperative CT and biopsy WSI, may affect the accuracy of preoperative TRG0 prediction for esophageal cancer. To address this issue, this paper proposes a CT–WSI cross-modal fusion network, DRGFuse, based on alignment guidance and two-layer reliability gating, for preoperative TRG0 vs TRG1–3 classification of esophageal cancer.

[0092] Experimental results show that DRGFuse can effectively fuse tumor region and surrounding contextual features from CT images with histological semantic features from biopsy WSI, suppress harmful cross-modal interactions through a mismatch perception mechanism, and adaptively back off when there is modal conflict or low quality. Multicenter experiments demonstrate that this method has better generalization ability and lower false positive rate prediction performance compared to single-modal models and representative multimodal fusion models.

[0093] Experimental setup and dataset: A retrospective study was conducted on a multicenter, private esophageal cancer cohort at a collaborating hospital. Inputs included pre-treatment 3D CT scans and pre-treatment endoscopic biopsy WSIs, labeled as TRG0–3 obtained from post-operative en bloc resection specimen evaluation. A binary endpoint was defined as TRG0 vs TRG1–3, with TRG0 being the positive class. The cohort included an internal center (n=525) and two independent external centers (n=51 / 62). Included cases required traceable TRGs, pre-treatment CT scans, and pre-treatment WSIs; missing or invalid TRG records were excluded. CT scans and WSIs were paired at the patient level using in-hospital identifiers. Due to privacy restrictions, the data is not publicly available.

[0094] Implementation Details: This application reports AUC and deployment-oriented low-FPR metrics: pAUCn and TPR@Sp95. Patient-level stratified five-fold cross-validation is performed internally: each fold is divided into train / val / test; val is used for early termination, model selection, and determining the Sp95 threshold. This threshold is subsequently fixed for evaluation in the corresponding internal test and external A / B tests. External centers are not involved in training or model / threshold selection. Preprocessing is frozen during development and consistently applied in training and inference. Training uses AdamW, up to 50 epochs, with batch sizes of 4–8, optimized with weighted BCE combined with a low-FPR pAUC surrogate term.

[0095] The experimental results are as follows: Comparison Methods: DRGFuse was compared with radiomics baselines, CT-only depth models, WSI-side weakly supervised MIL / basic characterization methods, and strong multimodal baselines. All methods were evaluated using the same five-fold partitioning and fixed Sp95 threshold protocol, with external centers not involved in any selection. Experimental results showed that DRGFuse performed well in all three centers. In the Internal center, AUC, pAUCn, and TPR@Sp95 reached 0.762, 0.602, and 0.541, respectively, and in External A / B, they reached 0.720 / 0.542 / 0.456 and 0.713 / 0.525 / 0.433, respectively. Compared with strong unimodal and representative multimodal baselines, this application showed a more significant advantage in the low-FPR region, indicating that its improvement comes not only from the overall discriminative performance but also primarily from the low false positive rate operating point, which is of greater clinical concern. By combining pre-treatment 3D CT and endoscopic biopsy WSI information, richer imaging, pathological, and cross-modal complementary features can be learned, thereby enhancing the model's ability to distinguish between TRG0 and TRG1–3 and improving the accuracy and generalization of preoperative efficacy prediction after neoadjuvant therapy for esophageal cancer.

[0096] Ablation Analysis: To explain the more significant gain at low-FPR and the outer center, this application conducted ablation analysis on key components. The results show that removing any key module leads to varying degrees of performance degradation, particularly in metrics such as pAUCn and TPR@Sp95. This indicates that the performance improvement is not solely due to the superposition of multimodal data, but rather to explicit modeling and adaptive reliability control of the structural mismatch between CT and WSI.

[0097] The CT Habitat Token encoder is used to extract tumor ROI and surrounding contextual features, while the WSI-based Token encoder is used to extract transferable histological features. Cross-modal mismatch estimation and bilayer reliability gating can suppress harmful migration caused by conflicting modal information. Align-Guard, co-teaching, and low-FPR objectives further guide the model to focus on clinically critical areas with low false positive rates. Ablation results demonstrate that the modules work synergistically, enabling DRGFuse to form a reliable prediction mechanism for conflict perception even when there is a mismatch between preoperative local CT / WSI evidence and postoperative whole-specimen TRG labels, thereby improving the stability and clinical applicability of TRG0 vs TRG1–3 prediction.

[0098] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method, characterized in that: Includes the following steps: Acquire pre-treatment three-dimensional CT volume data of the subjects and their corresponding preoperative biopsy whole slice images (WSI); Extract CT representations from the CT volume data, the CT representations including at least one CT token and a global CT representation; Extract WSI representations from the WSI, the WSI representations including at least one WSI token and a WSI global representation; Based on the CT global representation and the WSI global representation, the degree of cross-modal mismatch is calculated, and pre-gated interaction coefficients are generated, wherein the pre-gated interaction coefficients are negatively correlated with the degree of mismatch. Based on the pre-gated interaction coefficients, cross-modal interaction and fusion are performed on the CT token and the WSI token to generate a fused representation. Based on the CT representation, the WSI representation, and the degree of mismatch, adaptive fusion weights for CT experts, WSI experts, and fusion experts are generated. Based on the adaptive fusion weights, residual fusion is performed on the CT-only prediction results, WSI-only prediction results, and fusion prediction results based on the fusion representation to obtain the final prediction result. The final prediction result is used to output the prediction probability value representing the possibility of TRG0, which is an intermediate result for medical personnel to refer to.

2. The alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method according to claim 1, characterized in that: The extraction of CT characterization from the CT volume data includes: The CT scan data is segmented to obtain the region of interest (ROI). Based on the tumor ROI, multiple habitat masks representing different spatial ranges are constructed. The habitat mask includes a core mask representing the interior of the tumor, a rim mask representing the tumor boundary, and a ring mask representing the area surrounding the tumor. Voxel-level feature maps of the CT volume data are extracted using a pre-trained 3D visual encoder. Pooling is performed on the voxel-level feature map within each habitat mask to obtain the corresponding CT token.

3. The alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method according to claim 1, characterized in that: The extraction of WSI characterization from the WSI includes: The WSI is segmented and divided into regions to obtain multiple image patches; Each patch is encoded as a semantic token using a pre-trained pathology-based model; Each semantic token is scored for importance, and the top K tokens with the highest scores are retained; The state-space sequence model MambaMIL is used to perform context modeling on the K retained tokens, and the global representation of WSI and the WSI-only prediction results are obtained through multi-instance learning MIL pooling.

4. The alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method according to claim 1, characterized in that: The calculation of cross-modal mismatch and the generation of pre-gated interaction coefficients further include: The global representation of CT and the global representation of WSI are concatenated, the difference is calculated, and the element-wise product is calculated. The result is input into the difference embedding network to obtain the difference vector. The difference vector is input into the mismatch scoring network and mapped by the sigmoid function to obtain the scalar mismatch score. Set the pre-gated interaction coefficient to 1 minus the mismatch score, so that the higher the mismatch score, the lower the pre-gated interaction coefficient.

5. The alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method according to claim 1 or 4, characterized in that: The cross-modal interaction and fusion of the CT token and the WSI token to generate a fused representation includes: Pre-gating is achieved by weighting the results of cross-modal projection using the pre-gated interaction coefficients. Input the weighted CT token and WSI token into the Cross-Mamba module to update their respective context information; The updated CT token and WSI token are concatenated, layer normalized, and aggregated to obtain the fused representation.

6. The alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method according to claim 1, characterized in that: The adaptive fusion weights for generating CT experts, WSI experts, and fusion experts include: A CT quality vector and a WSI quality vector are constructed. The CT quality vector is obtained based on the confidence statistics of tumor segmentation, and the WSI quality vector is obtained based on image quality control statistics and attention distribution concentration. The CT quality vector, the WSI quality vector, the mismatch score, and the difference vector are concatenated and input into a routing network composed of a softmax function. The output is three weight values ​​corresponding to the fusion expert, the CT expert, and the WSI expert, respectively.

7. The alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method according to claim 1, characterized in that: The training phase of the method employs a combined loss function for optimization, which includes: weighted binary cross-entropy loss, partial AUC (pAUC) loss for regions with low false positive rates, Align-Guard regularization loss for aligning cross-modal representations, and Co-teaching loss for resisting label noise; and, during the inference phase, the Align-Guard regularization term, the Co-teaching mechanism, and the pAUC loss are removed.

8. The alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method according to claim 4, characterized in that: The difference vector and the mismatch score are calculated as follows: Difference vector Calculated using the following formula: ; in, For differential embedding networks, For global CT characterization, For WSI global representation; Mismatched scores Calculated using the following formula ;in, For output The scoring head, The sigmoid function has the following pre-gated interaction coefficients: .

9. The alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method according to claim 6, characterized in that: The adaptive fusion weights of the fusion expert, CT expert, and WSI expert are calculated through the following routing network: ； in, For CT quality vector, For WSI quality vector, For mismatched scores, For the difference vector, For routing networks.

10. The alignment-guided, two-layer reliability-gated CT-WSI fusion prediction method according to claim 6 or 7, characterized in that: Calculated using the following residual logit fusion formula: ； ； in, To integrate the prediction results, For CT-only prediction results, For WSI-only prediction results, , , These are adaptive fusion weights for fusion experts, CT experts, and WSI experts, respectively. For the sigmoid function, To integrate with Logit, To predict the probability of TRG0 in the final calculation; The combined loss function is specifically expressed as follows: ; in, For weighted binary cross-entropy loss, False positive rate Partial AUC proxy loss over the interval For Align-Guard alignment regularization loss, To compensate for the losses in co-teaching, , , These are the corresponding equilibrium hyperparameters.

Images