A method for accurately predicting the clinical prognosis of colorectal cancer patients

Abstract

The present application relates to pathological section image recognition technical field, specifically to a kind of accurate prediction colorectal cancer patient clinical prognosis method, including full field digital slide acquisition, full field digital slide preprocessing, Transformer-dynamic proxy attention-Mamba network and explainability analysis.The method first integrates tissue section segmentation unit and preliminary feature extraction unit, realize full field digital slide feature extraction, then adopt Transformer-dynamic proxy attention coupling architecture, and realize time series dependence modeling by multilayer Mamba module, finally use feature aggregation unit to output patient risk score, and use heat map to realize the visualization explanation of model decision.The present application solves the technical bottlenecks such as insufficient feature representation ability and long-range dependence modeling difficulty in traditional pathological image analysis by fusing long sequence data processing, dynamic proxy attention mechanism and state space modeling technology.

Description

Technical Field

[0001] This invention relates to the field of pathological slide image recognition technology, specifically to a method for accurately predicting the clinical prognosis of colorectal cancer patients. Background Technology

[0002] Accurately predicting the prognosis of colorectal cancer patients is not only of great significance for developing personalized treatment plans, but also has a profound impact on evaluating treatment effects, guiding clinical decisions, and improving the overall survival rate of patients.

[0003] Traditionally, the assessment of prognosis for colorectal cancer has relied primarily on physicians' clinical experience, pathological staging, and the detection of certain single biomarkers. First, while clinical experience has accumulated a wealth of diagnostic and treatment knowledge over long-term practice, its judgment is highly subjective. Differences in experience, perspective, and observational details among different physicians often lead to inconsistent prognostic assessments. Second, while pathological staging, as an important indicator of tumor progression, reflects the overall trend of tumor development, its ability to differentiate subtle differences in prognosis among patients at the same stage is limited. Furthermore, although the detection of single biomarkers provides some basis for understanding the biological characteristics of tumors, their sensitivity and specificity are often insufficient, making it difficult to comprehensively characterize the complex biological behavior and internal heterogeneity of tumors. These traditional methods often fall short of the requirements of precision medicine for refined data and personalized predictions, necessitating the introduction of more advanced and intelligent technological means.

[0004] In recent years, with the development of whole-slide image (WSI) and artificial intelligence (AI) technologies, mining prognostic information from pathological images using deep learning models has become a research hotspot. However, existing technical solutions still have many key shortcomings, severely restricting their application value and reliability in clinical practice. First, WSI images are usually segmented into tens of thousands of tiny image patches, forming an extremely long sequence. Traditional convolutional neural networks (CNNs) mainly focus on local receptive fields, making it difficult to effectively capture long-distance spatial dependencies between image patches, such as the overall structure and heterogeneity of the tumor microenvironment. While the popular Transformer architecture can model global dependencies, the computational complexity of its self-attention mechanism is proportional to the square of the sequence length. When processing extremely long sequences like WSI, this results in huge computational overhead and memory consumption, leading to low model training and inference efficiency and hindering practical application. Therefore, there is an irreconcilable contradiction between the ability to capture global information and computational efficiency in existing methods. Second, one of the core challenges of medical AI is the "black box" problem. While many existing deep learning models may achieve high prediction accuracy, their internal decision-making logic remains opaque. Doctors cannot understand which specific areas in pathological sections (such as the tumor infiltration front, lymphocyte aggregation areas, etc.) the model bases its judgments on. This lack of interpretability makes it difficult to gain the trust of clinicians, and its predictions are hard to integrate into actual clinical decision-making processes, thus limiting its true clinical application value. Summary of the Invention

[0005] 1. Redefining the technical problem

[0006] The core technical problem to be solved by this invention is how to simultaneously resolve the following two mutually restrictive technical contradictions in the specific scenario of colorectal cancer WSI analysis: (1) The contradiction between computational efficiency and feature integrity: After WSI is segmented into ultra-long sequences (usually >10,000 image blocks), although the Transformer model can capture global dependencies, its computational complexity is high (O(N²)), and although CNN is computationally efficient, it is difficult to capture long-distance dependencies, resulting in the inability of existing methods to simultaneously guarantee computational efficiency and feature integrity. (2) The contradiction between model interpretability and prediction accuracy: Existing deep learning models have a "black box" problem, and doctors cannot understand the basis of the model's decision-making, which makes it difficult to gain clinical trust even if the prediction is accurate, thus limiting the practical application value.

[0007] 2. Technical Solution

[0008] To address the aforementioned technical problems, this invention provides a method for accurately predicting the clinical prognosis of colorectal cancer patients. Based on full-view digital slicing, it constructs an end-to-end deep learning framework using an innovatively designed Transformer-dynamic agent attention-Mamba network structure. This framework effectively extracts key region information affecting patient survival from high-dimensional image features, enabling accurate assessment and interpretable display of patient risk. The core innovation of this invention lies in using a dynamic agent attention mechanism as a bridge to achieve the synergistic operation of Transformer's global modeling capabilities and Mamba's efficient sequence processing capabilities. Specifically, it includes the following steps:

[0009] S1. Full-view digital slice acquisition: acquire full-view digital slices of colorectal cancer pathological sections, wherein the full-view digital slices contain microstructural information of the tumor region and surrounding stromal tissue, and wherein the full-view digital slices are divided into tens of thousands of image blocks to form an ultra-long sequence.

[0010] S2. Full-view digital slice preprocessing: The full-view digital slice is subjected to tissue region segmentation, tumor heterogeneity score calculation, image block segmentation and feature extraction to obtain a feature vector sequence representing the local features of the pathological image;

[0011] S3. Construct a Transformer-Dynamic Agent Attention-Mamba network, which achieves effective fusion of global and local information by capturing short-term and long-term dependencies in sequence data. Specifically, it includes three parts: Transformer-Dynamic Agent Attention Coupling Architecture, multi-layer Mamba module, and feature aggregation unit.

[0012] S4. Interpretability Analysis: By mapping the attention weights calculated internally by the model back to the spatial location of the original full-view digital slices, a heatmap is generated and overlaid with the original image to intuitively demonstrate the model's attention to different regions in the pathological image, especially highlighting key prognostic-related regions such as the tumor infiltration front and lymphocyte aggregation areas.

[0013] The full-view digital slide acquisition involves using a high-precision digital scanner to scan the slide at 40x magnification, ensuring that the generated full-view digital slide file contains complete microstructural information of the tumor region and its surrounding stroma.

[0014] The full-view digital slice preprocessing specifically includes the following parts: a tissue slice segmentation unit segments the full-view digital slice to remove useless background; multi-scale feature extraction is performed on the segmented tumor region; and a heterogeneity score including gray-level co-occurrence matrix texture features, cell morphology diversity, and tissue structure complexity is calculated; then, the segmented image is divided into non-overlapping segments of size [missing information]. The image patches are processed; the preliminary feature extraction unit extracts the tissue structure, cell morphology, and texture distribution of the image patches based on the pre-trained pathological feature extraction networks ResNet50, UNI, and CONCH, and performs feature encoding to obtain the feature vectors corresponding to the full-view digital slices. These feature vectors can comprehensively reflect key pathological information such as tissue structure, cell morphology, and texture distribution, providing rich input data for the subsequent Transformer-Dynamic Agent Attention-Mamba network.

[0015] The Transformer-Dynamic Agent Attention-Mamba network comprises three parts: a Transformer-Dynamic Agent Attention Coupling Architecture, a multi-layer Mamba module, and a feature aggregation unit. The Transformer-Dynamic Agent Attention Coupling Architecture captures the short-term and long-term dependencies of sequence information in full-view digital slice data; the multi-layer Mamba module further refines and enhances the input features through multi-level feature extraction; and the feature aggregation unit integrates global information from the local features processed by the first two parts, achieving effective fusion of global and local information, and ultimately outputting the patient's risk score.

[0016] The interpretability analysis involves spatially mapping the attention weights calculated within the model to generate a corresponding heatmap, which is then overlaid on the original full-view digital slices for display. This intuitive visualization clearly shows the model's attention levels in different image regions, thereby validating the model's predictions and providing strong support for subsequent model optimization and clinical decision-making.

[0017] 3. Correspondence between technical solutions and technical problems (Table 1)

[0018] Table 1. Correspondence between existing technical problems and the proposed technical solution

[0019] Technical issues Solution of the present invention The extremely long WSI sequence results in high computational complexity (N²) for Transformer. Dynamic agent attention mechanism significantly reduces computational complexity CNNs struggle to capture long-range spatial dependencies. Effective fusion of global and local information is achieved through bidirectional interaction in the dynamic agent attention mechanism. In proxy attention mechanisms, a fixed proxy token leads to inaccurate attention regions. The dynamic reconstruction mechanism enables the proxy token to be dynamically adjusted based on the input pathological features. The Mamba model has insufficient ability to perceive local features in pathological image analysis. The Mamba module processes feature sequences that have been streamlined by a dynamic proxy attention mechanism, rather than the original, extremely long sequences. The "black box" problem of the model leads to low clinical trust. The interpretability module pays particular attention to key prognostic regions such as the tumor invasion front and lymphocyte aggregation areas.

[0020] 4. Detailed Explanation of Core Innovations

[0021] 4.1 The innovativeness of dynamic agent attention mechanism

[0022] The core innovation of this invention lies in the dynamic agent attention mechanism, which is fundamentally different from existing technologies:

[0023] (1) Dynamic Reconstruction Mechanism: Unlike the fixed proxy tokens in existing proxy attention mechanisms, the proxy tokens in this invention can be dynamically adjusted according to the input pathological features. When processing colorectal cancer WSI, this mechanism enables the proxy tokens to automatically focus on key pathological areas (such as the tumor invasion front and lymphocyte aggregation areas), rather than being uniformly distributed. Experiments have shown that this dynamic mechanism is crucial for identifying prognostic heterogeneous areas in colorectal cancer, and the C-index is significantly improved compared to fixed proxy mechanisms.

[0024] (2) Bidirectional attention interaction: This invention designs bidirectional attention computation from agent to image block and from image block to agent, so that the agent token first absorbs the key information of the input features and then feeds this information back to the image block features, forming a closed-loop information flow. This design effectively alleviates the information silo problem in traditional attention mechanisms and realizes the deep integration of global information and local features.

[0025] 4.2 Innovative Integration of Mamba Networks

[0026] This invention does not simply combine the Mamba module with the Transformer, but rather achieves their collaborative operation through specific technical design:

[0027] (1) Simplified sequence processing: The Mamba network does not process the original ultra-long sequence, but the feature sequence after simplification by the dynamic proxy attention mechanism. The dynamic proxy attention mechanism simplifies the sequence of more than 10,000 image blocks into 32 to 64 proxy features containing only key pathological information, so that the Mamba module can effectively capture the long-range dependencies of pathological features while maintaining computational efficiency.

[0028] (2) Targeted optimization: The present invention optimizes the key parameter (rate) in the Mamba network for pathological image characteristics, enabling it to better capture the hierarchical representation in the pathological features of colorectal cancer, especially in handling tumor heterogeneity and immune cell distribution patterns.

[0029] (3) SandwichBlock innovative structure: Mamba and NystromAttention are combined in a sandwich form and normalized using layer normalization. Layer normalization parameters are introduced to scale the residual connections. This design can capture sequence dependencies while maintaining the advantages of the global attention mechanism.

[0030] (4) Bi-directional Mamba processing (BiMambaBlock): It implements Mamba processing in both forward and backward directions. The forward Mamba process processes the original sequence, and the reverse sequence is processed through the flip operation. The bi-directional features are fused using a linear layer. This design can better capture contextual information.

[0031] 4.3 Layered Agent Attention

[0032] This invention designs proxy attention at two scales: fine-grained and coarse-grained. Fine-grained attention maintains the original resolution, while coarse-grained attention reduces computational complexity through downsampling. Then, gated MLP is used to adaptively fuse features from both scales. This design balances computational efficiency and feature representation capability.

[0033] 4.4 Heterogeneity Assessment Module

[0034] Feature heterogeneity is evaluated from three dimensions: texture, morphology, and structure. A fusion network is used to calculate a heterogeneity score, which is used to dynamically adjust the number of proxy tokens to achieve adaptive calculation, making it more suitable for the analysis scenario of colorectal cancer pathological images.

[0035] The beneficial effects of the invention are as follows:

[0036] (1) By using a dynamic agent attention mechanism, the computational complexity of attention is reduced from N² to N×agent_num, resulting in a reduction of at least 70% in memory usage when processing full-view digital slices containing more than 10,000 image patches, thus resolving the fundamental contradiction between computational efficiency and feature integrity in WSI analysis. This effect is mainly achieved through a bidirectional attention interaction mechanism, making the number of agent tokens (agent_num) much smaller than the original sequence length (N).

[0037] (2) Through a bidirectional attention interaction mechanism, bidirectional information flow between the proxy token and image patch features is realized, effectively alleviating the information silo problem in traditional attention mechanisms and enabling the model to simultaneously capture the local microstructure and overall spatial distribution features of the tumor. This effect is mainly achieved by calculating the attention from the image patch to the proxy, enabling the image patch features to be integrated with the global information in the proxy token.

[0038] (3) By using multi-scale dynamic deep convolutional positional encoding, the model's ability to perceive multi-scale structural features (such as tumor cell clusters, glandular structures, and immune cell infiltration patterns) in colorectal cancer pathological images is enhanced. The weights of the multi-scale convolutional kernels are automatically adjusted according to the pathological characteristics of local regions. The weights of large-size convolutional kernels are automatically increased in tumor regions, while the weights of small-size convolutional kernels are increased in stroma regions. The multi-scale features are fused with the original features through a gating mechanism to achieve more accurate positional encoding.

[0039] (4) Through a dynamic reconstruction mechanism, the proxy token can be dynamically adjusted according to the input pathological features, focusing on key pathological areas of colorectal cancer, which significantly improves the ability to capture tumor heterogeneity and immune microenvironment features. Dynamic reconstruction of the proxy token is achieved through learnable proxy tokens and bidirectional attention interaction.

[0040] (5) By processing the feature sequence after being simplified by the dynamic agent attention mechanism through the Mamba network, the efficient sequence processing capability of the Mamba network is maintained, while its weak local perception is compensated for. This achieves efficient fusion of global and local features and improves the prediction robustness of the model.

[0041] (6) The heatmap generated by the interpretability module accurately corresponds to the pathologically significant prognostic regions (such as the tumor infiltration front and lymphocyte aggregation areas), enabling clinicians to intuitively verify the correlation between model decisions and pathological features, thus solving the 'black box' problem of medical AI models. Attached Figure Description

[0042] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly described below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without any creative effort:

[0043] Figure 1 This is a flowchart of a method for accurately predicting the clinical prognosis of colorectal cancer patients.

[0044] Figure 2 This is a diagram of the algorithm structure of the Transformer-Dynamic Agent Attention-Mamba network.

[0045] Figure 3 The results are from the ablation experiment. (A) The receptive field of the Mamba network proposed in this invention (Transformer-Dynamic Agent Attention); (B) The receptive field of the model after removing the Transformer component; (C) The receptive field of the model after removing the dynamic agent attention mechanism; (D) The receptive field of the model after removing the Mamba component. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The technical solutions provided by various embodiments of this invention will be described in detail below with reference to the accompanying drawings.

[0047] 1. Technical Principles

[0048] The technical principle of this invention is based on a key understanding: the core technical challenge facing colorectal cancer WSI analysis is not simply a matter of computational efficiency or feature capture, but rather the fundamental contradiction between the two. Traditional methods attempt to find a balance between global modeling capabilities and computational efficiency, but this approach itself has limitations. The innovation of this invention lies in using a dynamic agent attention mechanism as a bridge to organically integrate the global modeling capabilities of Transformer with the efficient sequence processing capabilities of Mamba, rather than simply combining them.

[0049] Specifically, the dynamic agent attention mechanism first simplifies ultra-long sequences (N>10000) into short sequences (agent_num=32 to 64) containing only key pathological information, solving the computational efficiency problem. Then, the Mamba module efficiently processes this simplified sequence, capturing long-range dependencies and solving the feature integrity problem. This specific collaborative working mechanism is not easily obtained by those skilled in the art through conventional technical means.

[0050] 2. Experimental Verification

[0051] To verify the technical effectiveness of this invention, we conducted systematic experiments on the TCGA-COAD and ZN-COAD colorectal cancer datasets. These datasets contain WSI images and clinical follow-up data from 581 and 1031 colorectal cancer patients, respectively.

[0052] 3. Experimental Setup

[0053] (1) Datasets: Exploratory set TCGA-COAD, 581 colorectal cancer patients, divided into training set and test set in a 7:3 ratio; Independent validation set ZN-COAD.

[0054] (2) Image preprocessing: 40x magnification scan, 256×256 pixel image block, ResNet50, UNI, CONCH feature extraction.

[0055] (3) Model Parameters: The model was trained on the TCGA-COAD queue, and the hyperparameters were optimized using a 5-fold cross-validation strategy. The Adam optimizer was used during training, with an initial learning rate of 2e-4 and a Dropout rate of 0.25 to prevent overfitting. To handle survival data, the continuous survival time was discretized into four intervals: low-risk group, low-to-medium risk group, medium-to-high risk group, and high-risk group, and the cross-entropy loss function designed for survival analysis was used. The maximum training epochs of the model were set to 100, and an early stopping strategy based on the validation set C-index was implemented. Specifically, after 5 epochs of warm-up, if the validation set C-index did not improve within 30 consecutive epochs (and the total number of epochs exceeded 50), the early stopping mechanism was triggered, and the best-performing model checkpoint was saved. We rigorously tested the performance of the trained optimal model on an independent ZN-CRC queue and systematically compared it with a series of benchmark models, including classic pooling methods (Max-Pooling, Mean-Pooling), attention-based multiple instance learning (MIL) models (ABMIL, CLAM-MB, CLAM-SB), Transformer-based models (TransMIL), and state-space model-based methods (MambaMIL).

[0056] (4) Evaluation index: C-index (consistency index).

[0057] (5) Ablation Experiments: To verify the necessity of each core component in the model proposed in this invention, we designed and conducted a series of ablation experiments. By systematically removing specific modules, we constructed three model variants: (1) removing the Transformer layer; (2) removing the dynamic agent attention module; and (3) removing the multi-layer Mamba module. We trained these three variants using the same strategy as the main model and compared their prediction performance with the complete model proposed in this invention. In addition, to explore the contribution of each module from the perspective of information aggregation, we performed Effective Receptive Field (ERF) analysis on the complete model and its three ablation variants. ERF analysis visualizes the dependence of the model output on the input features by calculating the gradient of the output layer center point relative to the input feature map.

[0058] Experimental results show that this invention achieves optimal prediction performance while maintaining the lowest memory usage and shortest inference time, verifying the effectiveness of the dynamic agent attention mechanism working in conjunction with the Mamba network. Experimental results also show that, regardless of the feature extractor used, the proposed method exhibits superior prediction performance in 5-fold cross-validation of the TCGA-COAD discovery queue (n=581), with an average consistency index (C-index) superior to seven benchmark models, including CLAM, TransMIL, and MambaMIL. In particular, when CONCH is used as the feature extractor, the proposed method demonstrates the best prediction performance, with an average C-index of 0.752 ± 0.022 (Table 2).

[0059] Table 2. Performance comparison of 5-fold cross-validation between TDAM-CRC and the benchmark MIL model in the TCGA-COADREAD queue based on different feature extractors.

[0060] ResNet-50 UNI CONCH Max-Pooling 0.609±0.090 0.651±0.048 0.596±0.087 Mean-Pooling 0.598±0.051 0.682±0.053 0.669±0.063 ABMIL 0.606±0.071 0.696±0.064 0.700±0.074 CLAM-MB 0.611±0.059 0.709±0.067 0.694±0.070 CLAM-SB 0.628±0.064 0.701±0.070 0.716±0.060 TransMIL 0.608±0.031 0.637±0.078 0.677±0.039 MambaMIL 0.680±0.025 0.699±0.053 0.729±0.066 This invention 0.701±0.012 0.714±0.033 0.752±0.022

[0061] We verified the necessity of each core component through a series of ablation experiments. The results show that removing any one of the Transformer (C-index 0.683±0.041), dynamic proxy attention (C-index 0.722±0.042), or Mamba (C-index 0.737±0.034) modules significantly reduces the model's predictive performance, thus confirming that the integrity of the model design and the synergistic effect of each component are the foundation of its good performance. Furthermore, to explore the contribution of each module from the perspective of information aggregation, we conducted ERF analysis. The results show that the complete model involved in this invention exhibits a broad and heterogeneous ERF, indicating that the model can effectively capture and integrate key regions from local fine features to global contextual information, achieving a balanced aggregation of the two. In contrast, removing the Transformer module leads to a significant decrease in ERF; while the model after removing the dynamic proxy attention mechanism exhibits a high overall ERF, its pattern tends to be homogeneous, lacking the central focusing structure unique to the model involved in this invention. Figure 3 ).

[0062] The full-field digital slide acquisition involved in this invention is mainly used to obtain high-quality full-field digital slides, providing a reliable data foundation for subsequent image analysis and feature extraction. In specific implementation, a high-precision digital scanner is first selected to scan colorectal cancer pathological slides after routine pathological processing. The selected scanner is required to have high resolution, high color fidelity, and good contrast to ensure that at 40x magnification, it can completely capture information about the tumor area, surrounding stromal tissue, and fine structures.

[0063] The full-view digital slice preprocessing involved in this invention includes two steps: tissue slice segmentation and preliminary feature extraction. It plays a crucial role in the entire system, primarily aiming to convert the original full-view digital slice data into high-dimensional feature vectors suitable for deep learning network processing. Tissue slice segmentation first involves preliminary segmentation of the image using preset color ranges and brightness thresholds to remove obvious background areas. Multi-scale feature extraction is then performed on the segmented tumor region, calculating a heterogeneity score (heterogeneity_score) that includes gray-level co-occurrence matrix texture features, cell morphology diversity, and tissue structure complexity. The segmented images are then processed at a fixed size (…). The image is divided into non-overlapping small image patches, each containing local details, laying the foundation for subsequent feature extraction. The preliminary feature extraction step uses a pre-trained pathological feature extraction network (ResNet50, UNI, CONCH) to encode the features of the image patches. Specifically, each image patch is input into the pre-trained network, and after several layers of convolution, activation and pooling operations, the corresponding high-dimensional feature vector is obtained. These feature vectors can capture information such as texture, shape, structure and color distribution in the image, thus providing sufficient information support for the subsequent deep learning module. The specific implementation of the tumor heterogeneity assessment includes the following steps: Step (1), texture feature extraction: calculate the four statistics of contrast, energy, entropy and correlation based on the gray-level co-occurrence matrix; cell morphology feature extraction: obtain the cell outline through the pre-trained cell segmentation model, and calculate the coefficient of variation of cell area, perimeter, roundness and aspect ratio; structural feature extraction: calculate the gland arrangement regularity, size consistency and spacing uniformity index based on the gland segmentation results; Step (2), feature fusion network: The features of the above three dimensions are encoded by independent multilayer perceptrons. Each multilayer perceptron includes: the first layer: the input dimension is the corresponding feature dimension, the output dimension is half of the input dimension, and the activation function is ReLU; the second layer: the input dimension is the output dimension of the previous layer, the output dimension is 64, and the activation function is ReLU; the outputs of the three branches are concatenated to obtain a 192-dimensional feature vector; step (3), heterogeneity score generation: the concatenated feature vector is used to generate the final heterogeneity score through a fusion network: the fusion network includes a three-layer structure: the first layer maps the 192-dimensional features to 128-dimensional features, the second layer maps the 128-dimensional features to 1-dimensional features, and the third layer compresses the output to the [0,1] interval through the Sigmoid function; the heterogeneity score is used to dynamically adjust the number of agent tokens and the positional encoding weights in the Transformer-dynamic agent attention coupling architecture.

[0064] The Transformer-Dynamic Agent Attention-Mamba network processing involved in this invention is the core component. This component consists of three parts: a Transformer-Dynamic Agent Attention Coupling Architecture, a multi-layer Mamba module, and a feature aggregation unit. The Transformer uses a self-attention mechanism to effectively capture long-distance dependencies; it introduces a dynamic agent token to reconstruct the attention query mechanism, effectively overcoming the computational redundancy problem of traditional attention mechanisms in pathological image analysis; the multi-layer Mamba module is a sequence modeling unit built based on a state-space model, which efficiently captures long-range spatiotemporal dependencies of pathological features; the feature aggregation unit is used to output the patient risk probability. These components work together in the overall model to form a system from local features to global decision-making.

[0065] The Transformer-Dynamic Agent Attention Coupling Architecture leverages the self-attention mechanism in Transformer networks to effectively capture long-range dependencies in full-view digital slice data. Simultaneously, it optimizes computational redundancy in traditional attention mechanisms by introducing dynamic agent tokens. Specifically, this includes input preprocessing and linear mapping, token padding and adding a category token, first Transformer layer processing, position encoding, second Transformer layer processing, introduction of a learnable agent token, and linear transformation generation. , , The process involves several steps: calculating the attention of the agent to the image patch, calculating the attention of the image patch agent, local enhancement and branch fusion to output a linear mapping, and token pruning.

[0066] (1) Input preprocessing and linear mapping, assuming input features ,in For batch size, The number of tokens in the input sequence. The feature dimension is determined by first performing a linear transformation on the input through a fully connected layer, mapping all inputs to a unified feature dimension. Then, the modified linear unit (ReLU) activation function is applied, thereby introducing nonlinear characteristics:

[0067] (1)

[0068] in, Indicates a fully connected layer. To modify the activation function of the linear unit, Let be the feature tensor after linear transformation, with dimension . This step ensures that subsequent calculations are performed on a uniform scale, facilitating the model's learning of the relationship between global semantics and local features;

[0069] (2) Fill in the tokens and add a category token. To facilitate subsequent two-dimensional position encoding, fill the number of input tokens into a square rootable form and add a learnable category token at the beginning of the sequence. :

[0070] (2)

[0071] in, The aggregated feature after adding category tokens has a dimension of Category tokens play a role in aggregating global information throughout the network. Their introduction can not only help the model capture global statistical features, but also further integrate local information through subsequent layers of propagation.

[0072] (3) The first Transformer layer processing uses a Nystrom attention-based Transformer layer to process the input data:

[0073] (3)

[0074] in, This represents the features after the first Transformer layer processing. LayerNorm is the layer normalization operation, and the Attention part uses the Nystrom method to approximate the global self-attention. The LayerNorm operation helps stabilize the network training process, and the Nystrom method can approximate the global self-attention matrix while reducing computational complexity, thus effectively solving the computational bottleneck problem that may occur when processing large-scale full-view digital slice data.

[0075] (4) Multi-scale dynamic location encoding: In order to better capture local spatial information, depthwise convolution is used for location enhancement. The weights of the multi-scale convolution kernels are automatically adjusted according to the pathological characteristics of the local region. The weights of large-size convolution kernels are automatically enhanced in the tumor region, and the weights of small-size convolution kernels are increased in the stroma region. The multi-scale features are fused with the original features through a gating mechanism to achieve more accurate location encoding.

[0076] (4)

[0077] in, , , For weights dynamically generated through the attention mechanism, satisfying + + =1, Indicates step size is The kernel size is The deep convolution operation is applied to the input data. By performing deep convolution on the input data, not only can detailed information within the local neighborhood be obtained, but the model's sensitivity to the spatial structure of the image can also be enhanced. To match the two-dimensional position enhancement and window layout, the token number is filled into a square rootable form, and two-dimensional Fourier coordinate features are superimposed to supplement the global position prior.

[0078] (5) Second Transformer layer processing:

[0079] (5)

[0080] After positional encoding, a Transformer layer is used again to further extract and fuse global and local information. Through two stacked Transformer layers, the model's ability to abstract and express input features is significantly improved, laying the foundation for subsequent proxy token interaction.

[0081] (6) Introducing an adaptive variable number of learnable agent tokens: Introducing an adaptive variable number of learnable agent tokens: The heterogeneity score (heterogeneity_Score) is calculated through the aforementioned tumor heterogeneity assessment unit, according to the formula agent_num = base_num + The heterogeneity_Score dynamically determines the number of proxy tokens, where base_num is the minimum number of proxy tokens. The scaling factor is used; then the proxy token is initialized with trainable parameters. Shape transformation To match multi-head attention, where For dynamically calculated values, Furthermore, a hierarchical proxy branch is constructed, consisting of fine-grained and coarse-grained sub-branches, with the fine-grained branch having a downsampling ratio... coarse-grained branch downsampling ratio And by sample-level gating coefficients The two branch outputs are weighted and fused, and the gating coefficients are obtained by combining the sample-level gating coefficients. The result is that the number of coarse-grained proxies is squared (rounded to the square root of the number of fine-grained proxies) to match the coverage area; fusion output ,in , The output is divided into two branches, and the dimensions are consistent with the main branch to facilitate subsequent query key-value generation.

[0082] (7) Generation by linear transformation , , Adjust the feature dimensions output from the upstream Transformer aggregation module to To be square-openable, queries are generated using learnable parameters. ) matrix, key ( ) matrix and value ( )matrix:

[0083] , (6)

[0084] in, and The information is then split into multiple heads and fed into subsequent attention calculations, allowing the model to capture information from different aspects on different heads.

[0085] (8) Calculate the attention from the agent to the image patch, treating the agent token as the query and the input sequence as the key-value pair. , ), calculate attention score:

[0086] , (7)

[0087] This is the attention weight matrix for the two-branch agent to the image patch. For two-branch proxy tokens, This is the transpose of the key matrix. Representing the feature dimension of a single head , Next, calculate the value matrix of the proxy aggregation:

[0088] , (8)

[0089] This allows the proxy token to fully absorb key information from the input features, and the value branch can be adjusted according to the sampling ratio. Downsampling and interpolation are performed to restore the original scale to reduce computational load; the window size is arranged approximately as a square based on the square root of the sequence length to match spatial downsampling, providing effective support for subsequent information interaction;

[0090] (9) Calculate image patch to agent attention, and transfer input sequence features Perform attention interaction again with the previously updated proxy token (which can be viewed as a key and value), and calculate the attention score:

[0091] , (9)

[0092] in, This is the attention weight matrix from the two-branch image patch to the agent. For querying the matrix, , For the bias of image patch to agent attention; then update the image patch representation. and :

[0093] , (10)

[0094] This results in image patch features that incorporate agent information, enabling the model to establish a closer connection between global information and local features, effectively alleviating the information silo problem in traditional attention mechanisms.

[0095] (10) Local enhancement and branch fusion and output linear mapping: Perform depthwise separable convolution on the features updated by the two branches respectively, and pass the sample-level gating coefficient. The outputs of the two branches are weighted and fused, and then the fused features are linearly projected to obtain the architecture output:

[0096] (11)

[0097] in, , This is a depthwise separable convolution operation; after spatial rearranging the value vectors, kernels of size are applied. The depthwise convolutions are weighted and fused by the weights output by the gated network, and the fusion result is added to the image patch update features in the same dimension.

[0098] (11) Token pruning: The token importance metric imp is the mean of the image patch attention to the agent in the head dimension and the agent dimension. Each token participates in subsequent modeling, among which For pruning ratio, The sequence length is given, and at least one token is reserved for subsequent modeling; when Pruning is not performed at times. The number of tokens to be retained. For pruning ratio, , For sequence length, This represents the floor function. The token importance is measured by the mean of the image patch's attention to the proxy in the head dimension and the proxy dimension. This can be achieved through the following steps:

[0099] Step (1), Importance Assessment: Calculate the average attention weight across the head and agent dimensions as the importance score for each token: ,in Attention weights from image patches to the agent; Let be the token importance score vector, with dimension . ; The attention weights from the image patch to the agent are represented by the dimension . , This represents the operation of averaging across dimension 1 (head dimension) and dimension 3 (agent dimension). That is, batch size. That is, the number of heads to focus on. For sequence length, Indicates the number of proxy tokens;

[0100] Step (2), Token Selection: Based on the pruning ratio Before choosing The most important tokens, where N is the sequence length; where... Represents the pruning ratio, with a range of values. , Indicates the number of tokens reserved. This is the floor function. This indicates that at least one token must be retained;

[0101] Step (3), Feature Preservation: The features corresponding to the selected important tokens are preserved through indexing operations, and the features of the remaining tokens are discarded to reduce the complexity of subsequent calculations.

[0102] The multi-layered Mamba module is primarily based on a state-space model and is used to further model the features output by the Transformer module, thereby capturing potential long-range spatiotemporal dependencies in the image. Specifically, it works by... The sequence modeling layers are connected in series, and each layer has a pruned token sequence. On the platform, each layer follows the following process:

[0103] (1) Normalization: for the first Layer input features are subjected to layer normalization get ;

[0104] (2) Context modeling: Select one of the following structures at each layer to generate incremental features. (A) State-space-attention dual-branch: State-space recursion and Nystrom approximate global attention are performed on the transposed sequence respectively; the two outputs are formed by channel-by-channel residual scaling and back-addition. (B) Dual-channel Mamba: The sequence is recursively processed in both forward and reverse states, and the two outputs are scaled channel by channel. The scaled features are then concatenated along the channel dimension and finally fused using a linear mapping to form the final output. ;

[0105] (3) Residual addition: Let ,in, For the first Layer input features;

[0106] Among these, the number of heads and landmarks in Nystrom attention, the state dimension of state space modeling, the kernel size and expansion ratio, and the number of layers are considered. All are configurable hyperparameters; the above structures have linear or near-linear time complexity, used to balance context modeling capabilities and latency budget; only one structure is selected for each layer, and the network can adopt a unified structure or a layer-wise hybrid structure.

[0107] The feature aggregation unit is mainly responsible for fusing the local features extracted by the aforementioned modules into global information and finally outputting the patient's prognostic risk score. This includes several steps: normalization, attention weight calculation, classification prediction, and risk score calculation.

[0108] (1) Normalization processing: normalization of the feature matrix output from the multi-layer Mamba module. Normalize:

[0109] (12);

[0110] (2) Attention weight calculation for normalized features (in First, a nonlinear transformation is performed, and then the attention score is obtained through a linear mapping:

[0111] (13);

[0112] Subsequently, the softmax function is applied to the scores of all tokens to obtain normalized weights, and then the weighted sums are used to obtain the global feature vector:

[0113] (14);

[0114] in, : The aggregated global feature vector For the first Normalized attention weights for each token;

[0115] (3) Classification prediction: The aggregated global feature vector is input into the classifier, and by mapping it to a preset class space, the classification prediction of the patient's prognostic risk is realized.

[0116] (15);

[0117] in, The unnormalized classification probability has the following dimensions: ,, This represents the number of patient categories based on follow-up data; the classification probabilities are scaled using temperature. ,in Temperature parameter. Number of patient categories based on follow-up data. The classifier uses a fully connected layer and is trained using loss functions such as cross-entropy, enabling the model to accurately distinguish between high-risk and low-risk patients under weak supervision.

[0118] (4) Risk score calculation: To further quantify prognostic risk, the logits output by the classifier are converted into survival risk probabilities using the sigmoid function. Then, the survival probability is calculated using a cumulative multiplication operation, and the final risk score is obtained by a negative cumulative summation operation of the survival probability. This score can reflect the patient's prognostic risk and provide a quantitative basis for clinical decision-making.

[0119] The interpretability module involved in this invention achieves intuitive heatmap generation by mapping the attention weights calculated internally by the model back to the spatial location of the original full-view digital slices. Specifically, the attention weights generated by the aforementioned feature aggregation unit are first... The dimensions were adjusted to This process maps the attention weights to the actual positions of image patches within the original full-view digital slice, ensuring that each weight value accurately corresponds to the relevant region in the original image. Then, the mapped attention weights are used in conjunction with a predefined kernel function. (used to describe the first) Image patch pairs location (Impact), generating a heatmap of the same size as the original full-view digital slice, the heatmap at spatial points The value at this location can be described as:

[0120] (16);

[0121] The value of each spatial point in the heatmap reflects the model's level of attention to that region; a higher value indicates a greater influence of that region on the model's decisions. Finally, the generated heatmap is overlaid with the original full-view digital slice to visually demonstrate the model's attention distribution across different regions. Clinicians can observe the overlay effect of the heatmap and the original image to verify the correlation between the model's decisions and pathological features, thereby increasing their confidence in the model's output and providing auxiliary decision support for subsequent treatment plans.

[0122] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0123] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations. The above-described embodiments of this invention do not constitute a limitation on the scope of protection of this invention.

Claims

1. A method for accurately predicting the clinical prognosis of colorectal cancer patients, characterized in that, The method includes the following steps: S1. Collect postoperative routine pathological diagnostic slides from colorectal cancer patients and scan them into full-field digital slides using a high-precision digital scanner. S2. Preprocess the full-view digital slice to obtain the heterogeneity score and feature vector corresponding to the full-view digital slice; S3. Construct a Transformer-Dynamic Agent Attention-Mamba network, which achieves effective fusion of global and local information by capturing short-term and long-term dependencies in sequence data. Specifically, it includes three parts: Transformer-Dynamic Agent Attention Coupling Architecture, multi-layer Mamba module, and feature aggregation unit. The implementation of the Transformer-dynamic agent attention coupling architecture includes the following steps: (1) Feature embedding and dynamic position encoding: Linear mapping is performed on the input features and a category token is added. Multi-scale dynamic position encoding is introduced. Local features are extracted using deep convolution kernels of different sizes, and multi-scale position features are fused with input features through a gating mechanism. (2) Dynamic dual-granularity proxy token generation: The number of proxy tokens is dynamically determined based on the heterogeneity score obtained in step S2 preprocessing. The higher the score, the more proxy tokens there are. Fine-grained and coarse-grained dual-branch proxy tokens are constructed, each corresponding to a different spatial downsampling scale, in order to capture global context information at different levels. (3) Bidirectional proxy attention interaction: In each branch, firstly, the attention of the proxy token to the image patch features is calculated, and the local information of the image patch is aggregated to the proxy token; then, the attention of the image patch features to the updated proxy token is calculated, and the global information contained in the proxy token is fed back to the image patch features to realize feature update; (4) Branch fusion and token pruning: The output features of fine-grained and coarse-grained branches are weighted and fused through a gating mechanism; the importance of the token is calculated according to the attention weight from the image patch to the agent, and the image patch tokens with importance lower than the preset ratio are removed to obtain the simplified sequence features for subsequent Mamba module modeling. S4. Based on the attention weights generated by the Transformer-Dynamic Agent Attention-Mamba network, generate an interpretability analysis heatmap.

2. The method for accurately predicting the clinical prognosis of colorectal cancer patients according to claim 1, characterized in that, In step S1, a high-precision digital scanner is used to scan the tumor into a full-view digital slice at 40x magnification. This ensures that a single full-view digital slice file contains complete microstructural information of the tumor region and surrounding stroma, with a resolution that allows for the identification of individual cell morphology. This provides a foundation for the subsequent identification of key pathological features in the tumor microenvironment.

3. The method for accurately predicting the clinical prognosis of colorectal cancer patients according to claim 1, characterized in that, Step S2, the preprocessing of the full-view digital slice specifically includes the following parts: segmenting the full-view digital slice to remove useless background, extracting multi-scale features from the segmented tumor region, calculating a heterogeneity score including gray-level co-occurrence matrix texture features, cell morphology diversity, and tissue structure complexity, and then dividing the segmented image into non-overlapping segments of size [missing information]. The image patches are used to extract the tissue structure, cell morphology and texture distribution of the image patches based on a pre-trained pathological feature extraction network, and feature encoding is performed to obtain the feature vector corresponding to the full-view digital slice.

4. The method for accurately predicting the clinical prognosis of colorectal cancer patients according to claim 1, characterized in that, Step S3, namely the Transformer-Dynamic Proxy Attention-Mamba Network, involves a multi-layered Mamba module composed of... The sequence modeling layers are connected in series, and each layer has a pruned token sequence. On the platform, each layer follows the following process: (1) Normalization: for the first Layer input features are applied to layer normalization ; (2) Context modeling: Select one of the following structures at each layer to generate incremental features. (A) State-space-attention dual-branch: State-space recursion and Nystrom approximate global attention are performed on the transposed sequence respectively; the features of the two outputs are fused to generate incremental features containing multiple contextual information. (B) Dual-channel Mamba: The sequence is recursively processed in both forward and reverse states, and the two outputs are scaled channel by channel. The scaled features are then concatenated along the channel dimension and finally fused using a linear mapping to form the final output. ; (3) Residual addition: Let ,in, For the first Layer input features.

5. The method for accurately predicting the clinical prognosis of colorectal cancer patients according to claim 1, characterized in that, Step S3, the feature aggregation unit of this step specifically includes the following 4 steps: Step (1), Normalization: Normalization of the feature matrix output by the multi-layer Mamba module Normalize: （1）； Step (2), Attention weight calculation: For normalized features First, a nonlinear transformation is performed, and then the attention score is obtained through a linear mapping. : （2） Apply softmax to the scores of all tokens to obtain normalized weights, then sum them using weighted averages to obtain the global feature vector: （3） in, This is the aggregated global feature vector. For the first Normalized attention weights for each token; Step (3), Classification Prediction: Aggregate the features Mapping to the category space via a classifier: （4） in, The unnormalized classification probability has the following dimensions: , This represents the number of patient categories based on follow-up data; the classification probabilities are scaled using temperature. ,in For temperature parameters; Step (4), Risk Score Calculation: Obtain the survival risk by transforming logits using a sigmoid transformation, obtain the survival probability using a cumulative multiplication operation, and finally obtain the risk score by calculating the negative cumulative sum of the survival probabilities. ， （5） in, To mitigate survival risks, the sigmoid function is used. The result is obtained by transforming logits. For the probability of survival, Indicates time The risk value, This represents the cumulative multiplication operator.

6. The method for accurately predicting the clinical prognosis of colorectal cancer patients according to claim 1, characterized in that, Step S4 specifically involves using the attention weights generated by the aforementioned feature aggregation unit. The dimensions were adjusted to Aligning with the spatial location information corresponding to each image patch, the heatmap is located at spatial points. The value at this location can be described as: （6） in, It is a kernel function used to describe the first... Image patch pairs location The influence of spatial mapping is used to generate heatmaps, which are then overlaid on the original images to visually demonstrate the model's attention distribution in different regions.

7. The method according to claim 3, characterized in that, The tumor heterogeneity assessment is specifically implemented by extracting multi-scale features from the preprocessed tumor region, including texture features based on the gray-level co-occurrence matrix, cell morphological diversity features obtained through a pre-trained cell segmentation model, and glandular structure regularity features. The above features are integrated into a heterogeneity score ∈ [0,1] through a learnable fully connected network; the heterogeneity score is used to dynamically adjust the number of agent tokens and the positional encoding weights in the Transformer-dynamic agent attention coupling architecture.

Images