A Colorectal Cancer Pathological Image Analysis Platform Based on CTE-MIL Deep Learning Architecture

Abstract

This invention discloses a colorectal cancer pathological image analysis platform based on the CTE-MIL deep learning architecture, which relates to the field of pathological image recognition technology. The platform includes the following steps: A foreground training image processing module acquires and preprocesses whole-slice images of known pathological regions to generate a foreground extraction training dataset. A Ti ssue-Seg segmentation network module is constructed, and the model is trained based on the training dataset to output a foreground mask of the pathological tissue. A feature unit selection module meshes the foreground mask, calculates the feature performance score of the patch images, and selects high-response patch images. A CFE feature extraction network module is constructed, and the model parameters are specifically adjusted based on the high-response patches for weakly supervised training. An attention mechanism optimization module introduces the TFAM attention mechanism, performs feature fusion based on the attention score, and improves the accuracy of pathological tissue type identification.

Description

Technical Field

[0001] This invention relates to the field of pathological image recognition technology, specifically to a colorectal cancer pathological image analysis platform based on the CTE-MIL deep learning architecture. Background Technology

[0002] Colorectal cancer (CRC) is the third most common cancer worldwide and a leading cause of cancer-related deaths. Statistics show that in 2024, there were approximately 1.9 million new cases of colorectal cancer globally, with 930,000 deaths, posing a serious threat to human health. In recent years, the incidence and mortality rates of colorectal cancer have continued to rise, showing a trend towards younger onset and increased malignancy. This situation highlights the urgent need for early screening, accurate classification, and precise diagnosis of colorectal cancer. Therefore, promoting the transformation of pathological diagnosis from "empirical pathology" to "intelligent pathology" has become an important direction for current tumor diagnosis, treatment, and research.

[0003] With the continuous maturation of whole-slide imaging technology, traditional glass slide sections can now be digitized at high resolution, greatly improving the storability, shareability, and automated processing capabilities of pathological images. Simultaneously, the rapid development of artificial intelligence, especially deep learning, has provided effective tools for the intelligent analysis of large-scale medical images, facilitating the automation and standardization of tumor-assisted diagnosis. Machine learning models, in particular, have demonstrated excellent performance in tasks such as whole-slide image classification, tumor subtyping, and prognostic prediction, becoming an important research direction in the field of digital pathology.

[0004] Currently, most related research remains at the algorithm validation stage, with low application rates of models in real clinical settings. On one hand, model design often prioritizes accuracy optimization, neglecting sample diversity and cross-center generalization adaptability, limiting itself to fixed-scale feature extraction, and failing to effectively capture multi-scale information. On the other hand, many models lack sufficient interpretability and cannot determine the main analysis region, resulting in significant image redundancy, wasted computational resources, and increased error rates, making them difficult to integrate into doctors' actual workflows. Furthermore, analysis of whole-slice images still suffers from poor accuracy and long processing times.

[0005] Therefore, an accurate and rapid colorectal cancer pathological image analysis platform is of great practical value for improving the early detection capability of colorectal cancer, alleviating the shortage of pathology department resources, and promoting the clinical application of AI algorithms.

[0006] In the prior art, CN114266794A discloses a cancer region segmentation system for pathological slide images based on a fully convolutional neural network. The system includes: extracting masks of tissue regions from the pathological slide image and removing blank backgrounds; combining the tissue regions with the labeled cancer regions in the pathological slide image; segmenting the entire field of view of the pathological slide image to obtain training sample data; augmenting the sample data; constructing a Unet segmentation network with ResNet50 as the encoder, replacing the convolutional units of the first-level encoder with combined convolutional units to extract information at different scales from the input image; introducing a feature fusion module in the decoder to fully utilize the information output by each level of the decoder; training and optimizing the segmentation network on the augmented dataset; and using a grid processing algorithm to predict and identify cancer regions within the entire pathological slide image. While this invention introduces multi-scale information in the feature extraction stage, improving the segmentation accuracy of cancer regions, fully convolutional networks are often limited to fixed-scale feature extraction and may not effectively capture multi-scale information. During segmentation, the lack of effective fusion of features at different levels can lead to blurred boundaries and loss of detail. Therefore, the real-time performance and effectiveness of the identification and analysis system are reduced.

[0007] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0008] The purpose of this invention is to provide a colorectal cancer pathological image analysis platform based on the CTE-MIL deep learning architecture to solve the problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A colorectal cancer pathology image analysis platform based on the CTE-MIL deep learning architecture specifically includes:

[0011] The foreground training image processing module is used to acquire full-slice images of colorectal pathology in several known colorectal cancer pathological regions and corresponding pathological tissue types. The acquired full-slice images of colorectal pathology are preprocessed, and the preprocessed full-slice images are recorded as sample training images. The sample training images are mapped one-to-one with the full-slice images of colorectal pathology to generate a foreground extraction training dataset.

[0012] The segmentation network training module is used to construct the Tissue-Seg segmentation network module based on the CLAM algorithm model. The Tissue-Seg segmentation network module is trained based on the data in the foreground extraction training dataset. At the same time, the spatial pyramid pooling fast algorithm is introduced into the Tissue-Seg segmentation network module to perceive the tissue structure and boundary contour of the pathological region of colorectal cancer. The output of the trained Tissue-Seg segmentation network model is to obtain the foreground mask of the corresponding pathological tissue and map it onto the original colorectal pathological whole slice image to achieve foreground segmentation at the whole slice image level of colorectal pathology.

[0013] The feature unit filtering module is used to divide the acquired foreground mask image into grids to obtain several patch unit images; the backbone network of the CFE feature extraction network module is pre-trained based on the patch unit images; a semantic response score is calculated for each patch image using a single hidden layer attention scoring network; the patch images are sorted and filtered according to the semantic response scores, and those that meet the requirements are recorded as high-response patch unit images; the pre-training refers to weakly supervised training of the backbone network of the CFE feature extraction network module with the patch unit images as input and the image-level pathological tissue type corresponding to the source full image as the label;

[0014] The feature extraction network training module is used to construct the CFE feature extraction network module. Based on the high-response patch unit images, the backbone network parameters of the pre-trained CFE feature extraction network module are specifically adjusted, and the CFE feature extraction network module with determined backbone network parameters is trained in a fully supervised manner. Specifically, the high-response patch unit images are uniformly sized and used as the input of the CFE feature extraction network model, and the corresponding feature vector type is used as the label to train the CFE feature extraction network model.

[0015] The attention mechanism optimization module is used to construct the TFAM attention mechanism module after the CFE feature extractor is trained, using the extracted feature vector as input. This module calculates the attention score for each patch, performs weighted feature fusion based on the attention score, generates a fused feature vector, and uses the fused feature vector as input to the MIL classifier. The MIL classifier is trained with the corresponding pathological tissue type as the label, and the trained MIL classifier can distinguish the pathological tissue type.

[0016] Furthermore, several known colorectal cancer pathological areas and corresponding pathological tissue types were acquired, including whole-section images of colorectal cancer tissues. The corresponding pathological tissue types covered two types of colorectal cancer tissues, specifically including colonic adenoma tissue and rectal adenoma tissue.

[0017] The acquired colorectal pathology whole-slice images were preprocessed. The specific preprocessing steps included: downsampling the uniformly sized colorectal pathology whole-slice images, reducing the original image size to 1 / 40th of its original size. The specific formula used for downsampling was:

[0018]

[0019] In the formula, output(i,j) represents the output pixel of the i-th row and j-th column of the colorectal pathology whole slice image after downsampling, k is the downsampling factor, which is 40, m represents the current row offset, n represents the current column offset, Input(i*k+m,j*k+n) represents the original pixel value at position (i*k+m,j*k+n), and i and j represent the row index and column index of the downsampled image, respectively.

[0020] Based on the downsampled whole-slice images of colorectal pathology, the Labelme tool was used to annotate the areas where colorectal cancer pathology was located in the downsampled images. Specifically, the annotation method was to annotate the colorectal cancer pathological tissue using the largest bounding polygon. The annotated whole-slice images of colorectal pathology were then processed by a script to generate a corresponding foreground mask. The foreground mask was specifically a binary mask that distinguished the annotated colorectal cancer pathological tissue from the background of the whole-slice images of colorectal pathology. This completed the preprocessing of the whole-slice images of colorectal pathology.

[0021] The method for generating the foreground extraction training dataset is as follows: the downsampled colorectal pathology whole slice image is mapped one-to-one with the corresponding foreground mask to form a corresponding grid, and the formed grid is recorded as the foreground extraction training dataset.

[0022] Furthermore, based on the data in the foreground extraction training dataset, the Tissue-Seg segmentation network module is trained. The Tissue-Seg segmentation network module uses Convolutional Conv + Batch Normalized BN + SiLU activation as the basic unit. At the same time, a fast spatial pyramid pooling algorithm is introduced into the Tissue-Seg segmentation network module to perceive the tissue structure and boundary contour of the pathological region of colorectal cancer. The fast spatial pyramid pooling algorithm specifically includes multi-scale pooling and feature concatenation.

[0023] The specific training logic for the Tissue-Seg segmentation network module is as follows: The Tissue-Seg segmentation network module is trained using downsampled whole-slice images of colorectal pathology from the foreground extraction training dataset as input, and the corresponding foreground mask as the label. During training, the BCELoss function is used as the loss function, the Adam optimizer is selected, and the initial learning rate is set to 0.001. The specific expression of the BCELoss function is as follows:

[0024]

[0025] In the formula, y represents the loss between the predicted value and the true value of the p-th pixel in the training sample. p The actual label representing the pixel, Let p be the predicted probability value of the pixel by the model, where p is the index of the pixel in the training sample image, and N is the total number of pixels in the image.

[0026] Furthermore, the acquired foreground mask image is divided into grids to obtain several patch unit images. The specific steps for obtaining several patch unit images include: setting the grid size, obtaining the size of the mask image, calculating the number of rows and columns of the grid based on the grid size and the size of the mask image, dividing the image into multiple patch unit images by looping through the image, and using slicing operations to extract each patch unit image, wherein there are no fewer than 1000 patch unit images.

[0027] The semantic response score is calculated using a single hidden layer attention scoring network, and the specific formula used to calculate the semantic response score is as follows:

[0028] π q =sigmoid[W2*ReLU(W1h)] q +b1)+b2]

[0029] In the formula, π q h represents the semantic response score of the q-th patch unit image, specifically ranging from [0,1]. q Let W1 and W2 be the feature vectors of the q-th patch unit image, where W1 and W2 are weight matrices, b1 and b2 represent bias vectors, ReLU is the ReLU activation function, sigmoid is the sigmoid activation function, and q is the patch unit image index.

[0030] Furthermore, the feature vector h of the q-th patch unit image qSpecifically, feature extraction is performed using a pre-trained ResNet50 model. The ResNet50 model is the backbone network of the CFE feature extraction network module. The ResNet50 model has linear fully connected layers removed, and its feature dimension is 1024. The feature vector h... q This includes information on the boundary structure, morphological distribution, and texture semantic features of pathological tissues within the patch unit image region;

[0031] Patch unit images are filtered based on semantic response scores, and those that meet the requirements are designated as high-response patch unit images. The specific logic for filtering is as follows: based on the semantic response scores of the patch unit images, the patch unit images are sorted in descending order according to their semantic response scores to form an adjustment unit image sequence. The top 1000 patch unit images from the adjustment unit image sequence are selected and designated as high-response patch unit images.

[0032] Furthermore, the ResNet50 model is pre-trained, and the standard cross-entropy loss function is used for optimization during the pre-training process. The backbone network of the CFE feature extraction network module is specifically the ResNet50 model. The ResNet50 model is pre-trained, and the standard cross-entropy loss function is used for optimization during the pre-training process. The input is all patch unit images, and the image-level pathological tissue type corresponding to the whole image is used as the label. The label provides supervision signals to guide the network to learn intermediate representations with discriminative ability, rather than directly outputting classification. The finally trained model can generate feature vectors that express the semantic information of each patch. The convolutional layers of the pre-trained ResNet50 model are frozen and unfrozen during the specific adjustment stage.

[0033] The backbone network parameters of the CFE feature extraction network module are specifically adjusted based on high-response patch unit images. The specific logic for this specific adjustment is as follows: a sparse training subset is constructed using high-response patch unit images as input, and the image-level pathological tissue type corresponding to the high-response patch unit images is used as the label. Through end-to-end adjustment training, the ResNet50 model parameters are updated. The ResNet50 model parameters specifically include convolution kernel weights and convolutional layer bias parameters.

[0034] Furthermore, the CFE feature extraction network module with determined backbone network parameters is subjected to fully supervised training. During the fully supervised training process, the cross-entropy loss function is used as the optimization objective, the Adam optimizer is used for gradient update, and image enhancement operations are introduced, specifically including random rotation, horizontal flipping, and scale transformation.

[0035] Furthermore, a specific attention score is assigned to the input feature vector, and the formula for calculating the attention score is as follows:

[0036]

[0037] In the formula, Let a be the attention score for the l-th feature vector. l and b l These are the attention features and fused gated attention features of the l-th high-response patch unit image, processed by the tanh activation function, respectively. C For the already combined feature a l and b l The weight matrix for linear transformation, ⊙ denotes element-wise multiplication, l is the index of the input feature vector, l∈[1,L], where a l and b l The specific formula used for the calculation is as follows:

[0038]

[0039] In the formula, tanh is the tanh activation function. For the image of the l-th high-response patch unit, the feature vector is scaled down, where σ represents the sigmoid function, and W... a and W b are the weight matrices for the attention features and the gated output features, respectively. The formula used for the calculation is:

[0040]

[0041] In the formula, ω c h is the channel attention weight vector. l This is the feature vector of the l-th high-response patch unit image;

[0042] Feature fusion is performed based on the attention scores of each feature vector to obtain a fused feature vector. The formula used to calculate the fused feature vector is as follows:

[0043]

[0044] In the formula, z is the fused feature vector, and softmax is the softmax activation function.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] Firstly, a self-designed Tissue-Seg foreground segmentation module was introduced. Utilizing a lightweight convolutional structure and spatial pyramid pooling mechanism, it efficiently extracts tissue regions and removes backgrounds at low resolution, ensuring the model focuses on areas of true pathological value. This module significantly reduces preprocessing time and improves overall system efficiency in clinical applications, making it a key component of the intelligent diagnostic workflow.

[0047] After tissue region extraction, a colorectal cancer feature extractor module was designed to accurately extract deep semantic features related to tumors. This module first introduces a variational information bottleneck mechanism to evaluate response intensity, automatically selecting high-response regions, effectively compressing redundant information, and highlighting key lesion areas with diagnostic significance. Using ResNet-50 as the backbone network, deep feature extraction is performed on selected patches, modeling image semantics from multiple dimensions such as morphology and texture, thereby improving the model's ability to perceive morphological differences in tumors.

[0048] To further enhance the discriminative power and task adaptability of feature representation, CFE underwent end-to-end fully supervised fine-tuning training based on feature selection. This enabled the model to correct the feature extraction direction from a global perspective, enhancing its ability to distinguish between different tissue categories. This training strategy not only strengthened the model's focus on key regions but also improved its generalization performance under different data distributions. A dual-focus attention mechanism, TFAM, was introduced. Attention distribution was modeled in parallel through two paths: channel-level and instance-level. The channel-level path adjusted the intensity of feature channel expression, strengthening dimensions related to tumor characterization and improving the model's focus on and discriminative ability for key regions. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the overall structure of the present invention;

[0050] Figure 2 This is a schematic diagram of the detection results from the Tissue-Seg foreground segmentation module.

[0051] Figure 3 A schematic diagram illustrating the visualization results of the CFE feature extractor and the pre-trained ResNet-50;

[0052] Figure 4 This is a schematic diagram of the heatmap visualization results of the TFAM attention mechanism. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0054] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0055] Example:

[0056] Please see Figures 1-4 The present invention provides a technical solution:

[0057] A colorectal cancer pathology image analysis platform based on the CTE-MIL deep learning architecture specifically includes:

[0058] The foreground training image processing module is used to acquire full-slice images of colorectal cancer in several known colorectal cancer pathological regions and corresponding pathological tissue types. The acquired full-slice images of colorectal cancer are preprocessed, and the preprocessed full-slice images are recorded as sample training images. The sample training images are mapped one-to-one with the full-slice images of colorectal cancer to generate a foreground extraction training dataset.

[0059] CTE-MIL specifically refers to the (CFE and TFAM Enhanced Multiple Instance Learning) deep learning architecture.

[0060] Collect whole-section images of colorectal cancer pathology in several known colorectal cancer pathological areas and corresponding pathological tissue types. The corresponding pathological tissue types cover two colorectal cancer tissue types, specifically including colonic adenoma tissue and rectal adenoma tissue.

[0061] The acquired colorectal pathology whole-slice images were preprocessed. The specific preprocessing steps included: downsampling the uniformly sized colorectal pathology whole-slice images, reducing the original image size to 1 / 40th of its original size. The specific formula used for downsampling was:

[0062]

[0063] In the formula, output(i,j) represents the output pixel of the i-th row and j-th column of the colorectal pathology whole slice image after downsampling, k is the downsampling factor, which is 40, m represents the current row offset, n represents the current column offset, Input(i*k+m,j*k+n) represents the original pixel value at position (i*k+m,j*k+n), and i and j represent the row index and column index of the downsampled image, respectively.

[0064] The original image is reduced to 1 / 40 of its original size, and a low-magnification thumbnail is extracted for model input. This operation significantly reduces computational resource consumption while preserving the structural integrity of tissue regions, ensuring the segmentability of the downsampled image.

[0065] Based on the downsampled whole-slice images of colorectal pathology, the Labelme tool was used to annotate the areas where colorectal cancer pathology was located in the downsampled images. Specifically, the annotation method was to annotate the colorectal cancer pathological tissue using the largest bounding polygon. The annotated whole-slice images of colorectal pathology were then processed by a script to generate a corresponding foreground mask. The foreground mask was specifically a binary mask that distinguished the annotated colorectal cancer pathological tissue from the background of the whole-slice images of colorectal pathology. This completed the preprocessing of the whole-slice images of colorectal pathology.

[0066] The method for generating the foreground extraction training dataset is as follows: the downsampled colorectal pathology whole slice image is mapped one-to-one with the corresponding foreground mask to form a corresponding grid, and the formed grid is recorded as the foreground extraction training dataset.

[0067] At the data level, the project integrates and uses multiple publicly available pathological image datasets such as TCGA-CRC, NCT-CRC-HE-100K, CRS, and CAMELYON16, covering various tissue types and data sources, and constructs a complete training and validation set configuration.

[0068] The image data collected specifically came from the TCGA-CRC dataset, which includes 660 WSI images of colorectal adenocarcinoma and 120 normal tissue slices, used for learning and optimizing the overall classification performance of the model. Simultaneously, the NCT-CRC-HE-100K dataset was introduced, providing approximately 100,000 patch-level slice images to provide refined supervision signals for the CFE feature extractor, enhancing its ability to learn local semantic features. In the model validation phase, two public datasets, CRS and CAMELYON16, were selected, representing different acquisition centers and different tissue types, respectively. Cross-center, cross-task evaluation scenarios were constructed to test the model's robustness and transfer generalization performance. This validation system ensured that the model not only performed excellently on the original training data but also possessed the ability to handle complex clinical data distributions.

[0069] The segmentation network training module is used to construct a Tissue-Seg segmentation network module based on the CLAM algorithm model. The Tissue-Seg segmentation network module is trained based on the data in the foreground extraction training dataset. At the same time, a fast spatial pyramid pooling algorithm is introduced into the Tissue-Seg segmentation network module to perceive the tissue structure and boundary contour of the pathological region of colorectal cancer. The output of the trained Tissue-Seg segmentation network model is to obtain the foreground mask of the corresponding pathological tissue and map it onto the original whole slice image of colorectal pathology, so as to achieve foreground segmentation at the whole slice image level of colorectal pathology.

[0070] Based on the data in the foreground extraction training dataset, the Tissue-Seg segmentation network module is trained. The Tissue-Seg segmentation network module uses Convolutional Conv + Batch Normalized BN + SiLU activation as the basic unit. At the same time, a fast spatial pyramid pooling algorithm is introduced into the Tissue-Seg segmentation network module to perceive the tissue structure and boundary contour of the pathological region of colorectal cancer. The fast spatial pyramid pooling algorithm specifically includes multi-scale pooling and feature concatenation.

[0071] The Tissue-Seg module employs a compact foreground segmentation network, with its backbone based on a CBL structure (Convolutional Pooling + Batch Normalized Networking (BN) + SiLU activation), providing strong feature extraction capabilities. It also introduces a Spatial Pyramid Pooling-Fast (SPPF) module to enhance the model's perception of multi-scale organizational structures and boundary details. By considering features at multiple scales simultaneously, SPPF effectively improves the model's ability to perceive organizational structures and details. Different organizational structures may exhibit different features at different scales, and SPPF can capture these features more comprehensively. Through spatial pyramid pooling, SPPF preserves spatial structural information during the pooling process. This means that even after downsampling, the model can still have a good understanding of location and structure. Traditional pooling operations (such as max pooling or average pooling) may lead to information loss, while SPPF, through its multi-level pooling strategy, can reduce this loss, making the model more comprehensive and accurate in understanding images.

[0072] In the feature fusion stage, a feature concatenation (Concat) mechanism is used to integrate image features of different depths, enhancing the model's segmentation capabilities. Finally, a foreground mask is generated through the prediction layer to achieve the identification and extraction of tissue regions. Feature concatenation is a commonly used feature fusion strategy, especially in image segmentation tasks, where it can significantly enhance the model's expressive power. The following are the specific steps for integrating image features of different depths using the feature concatenation mechanism: Extracting features from the input image. In this process, the network passes through multiple convolutional layers, activation layers, and pooling layers to generate feature maps of different depths. The selected feature maps are then concatenated. The concatenation operation is typically performed along the channel dimension of the feature maps.

[0073] Finally, the feature map, after feature concatenation and subsequent convolution, is input into the prediction layer to generate the foreground mask.

[0074] The specific training logic for the Tissue-Seg segmentation network module is as follows: Downsampled whole-slice images of colorectal pathology from the foreground extraction training dataset are used as input, with the corresponding foreground mask as the label. The Tissue-Seg segmentation network module is trained using the BCELoss function as the loss function, Adam as the optimizer, and an initial learning rate of 0.001. After each training round, the model performance is evaluated on the validation set, and the model weights with the lowest validation loss are saved as the final model. To facilitate process monitoring, the TensorBoard logging system is integrated during training to record and visualize the loss change curve in real time.

[0075] The specific expression for the BCELoss function is as follows:

[0076]

[0077] In the formula, y represents the loss between the predicted value and the true value of the p-th pixel in the training sample. p The actual label representing the pixel, Let p be the predicted probability value of the pixel by the model, where p is the index of the pixel in the training sample image, and N is the total number of pixels in the image.

[0078] Training results show that the model achieves stable convergence on both the training and validation sets, with the loss function decreasing synchronously, demonstrating good learning ability and generalization performance. The final mask generated by the model on the validation set images accurately covers the tissue region, with clear boundaries and coherent structure, highly consistent with human annotations.

[0079] Real-world testing shows that the tissue mask generated by this module has an average overlap of over 95% with the real tissue region, and the preprocessing time for a single full-slice image is only about 200ms, which greatly improves processing efficiency and provides an accurate and real-time input foundation for subsequent feature extraction and classification reasoning.

[0080] The feature unit filtering module is used to divide the acquired foreground mask image into grids to obtain several patch unit images; pre-train the backbone network of the CFE feature extraction network module based on the patch unit images; calculate the semantic response score of each patch image using a single hidden layer attention scoring network; sort and filter the patch images according to the semantic response scores, and record those that meet the requirements as high-response patch unit images; the pre-training refers to weakly supervised training of the backbone network of the CFE feature extraction network module with the patch unit images as input and the image-level pathological tissue type corresponding to the source full image as the label.

[0081] The acquired foreground mask image is divided into several patch unit images by meshing. The specific steps for obtaining these patch unit images include: setting the mesh size (first, determining the size of each patch unit image, which can be selected based on application requirements and the resolution of the mask image; the patch unit image is a cube-shaped region image); obtaining the dimensions of the mask image (obtaining the height and width of the foreground mask image); calculating the number of rows and columns of the mesh based on the mesh size and the mask image dimensions, specifically by dividing the height and width of the mask image by the side length of the patch unit image; iterating through the image to divide it into multiple patch unit images; and using a slicing operation to extract each patch unit image, with at least 1000 patch unit images.

[0082] The semantic response score is calculated using a single hidden layer attention scoring network, and the specific formula used to calculate the semantic response score is as follows:

[0083] π q =sigmoid[W2*ReLU(W1h)] q +b1)+b2]

[0084] In the formula, π q h represents the semantic response score of the q-th patch unit image, specifically ranging from [0,1]. q Let W1 and W2 be the feature vectors of the q-th patch unit image, where W1 and W2 are weight matrices, b1 and b2 represent bias vectors, ReLU is the ReLU activation function, sigmoid is the sigmoid activation function, and q is the patch unit image index.

[0085] The weight matrices W1 and W2, and the bias vectors b1 and b2, are all learnable parameters, automatically optimized through backpropagation during neural network training. Assume that the input feature vector h of each patch unit image... q If the dimension of the hidden layer is d, then the dimension of the intermediate hidden layer can be set to d′, thus determining the dimensions of each parameter as follows:

[0086] Weight matrix W1∈R d′*d , used to convert the input feature vector h q Mapped to the attention latent space;

[0087] Bias vector b1∈R d′ , which is the corresponding bias term;

[0088] Weight matrix W2∈R d′*1 This is used to map the activated intermediate vector to a semantic response score;

[0089] The bias vector b2∈R is the bias term for the final linear output.

[0090] Specifically, when the input feature dimension is d = 1024, the intermediate hidden layer dimension d′ = 256 can be selected to obtain a more expressive intermediate attention representation space. These parameters are randomly initialized during model initialization using the Xavier (Glorot) method to maintain consistency in the variance of activation values ​​between layers and avoid gradient vanishing or exploding. During training, the parameters are iteratively updated using the Adam optimizer by minimizing the loss function of downstream tasks, such as cross-entropy loss.

[0091] Ultimately, the attention scoring network outputs a response score π. q The ∈[0,1] can effectively characterize the relative importance of each patch feature in the entire image semantic recognition task, thereby achieving the fusion of focusing on key regions and feature weighting.

[0092] Initializing the weight matrix is ​​crucial, impacting both the model's convergence speed and final performance. Common initialization methods include random initialization: generating random values ​​using a normal or uniform distribution. For example, initializing using a normal distribution with a mean of 0 and a standard deviation of 0.01. In PyTorch, random weights can be generated using `torch.randn()` or `torch.rand()`.

[0093] Bias vectors b1 and b2 are typically initialized to zero. The weights and biases are updated using the backpropagation algorithm to minimize the loss function. The specific steps include: Forward propagation: Calculate the output using the current weights and biases. Loss calculation: Calculate the loss based on the model's output and the true labels. Backpropagation: Calculate the gradient of the loss with respect to the weights and biases. Parameter update: Update the weights and biases using optimization algorithms such as SGD, Adam, etc.

[0094] The feature extraction network training module is used to construct the CFE feature extraction network module. Based on the high-response patch unit images, the backbone network parameters of the pre-trained CFE feature extraction network module are specifically adjusted, and the CFE feature extraction network module with determined backbone network parameters is trained under full supervision. Specifically, the high-response patch unit images are uniformly sized and used as input to the CFE feature extraction network model, and the corresponding feature vector type is used as the label to train the CFE feature extraction network model.

[0095] Where the feature vector h of the q-th patch unit image q Specifically, feature extraction is performed using a pre-trained ResNet50 model. The ResNet50 model is the backbone network of the CFE feature extraction network module. The ResNet50 model has linear fully connected layers removed, and its feature dimension is 1024. The feature vector h... q This includes information on the boundary structure, morphological distribution, and texture semantic features of pathological tissues within the patch unit image region;

[0096] Patch unit images are filtered based on semantic response scores, and those that meet the requirements are designated as high-response patch unit images. The specific logic for filtering is as follows: based on the semantic response scores of the patch unit images, the patch unit images are sorted in descending order according to their semantic response scores to form an adjustment unit image sequence. The top 1000 patch unit images from the adjustment unit image sequence are selected and designated as high-response patch unit images.

[0097] The backbone network of the CFE (CRC Feature Extractor) feature extraction network module is a ResNet50 model. This ResNet50 model undergoes pre-training, optimized using the standard cross-entropy loss function. The input consists of all patch unit images, labeled with the corresponding image-level pathological tissue type. These labels provide supervision, guiding the network to learn discriminative intermediate representations rather than directly outputting a classification. The final trained model generates feature vectors expressing the semantic information of each patch. The convolutional layers of the pre-trained ResNet50 model are frozen and then unfrozen during the specificity adjustment phase. This allows the model to learn deeper and more discriminative semantic features in task-relevant regions while reducing reliance on background or irrelevant areas, further improving the model's expressive efficiency and discriminative ability. This guides the model to focus on diagnostically valuable regions, enhancing its feature representation ability in key areas.

[0098] The backbone network parameters of the CFE feature extraction network module are specifically adjusted based on high-response patch unit images. The specific logic for this specific adjustment is as follows: a sparse training subset is constructed using high-response patch unit images as input, and the image-level pathological tissue type corresponding to the high-response patch unit images is used as the label. Through end-to-end adjustment training, the ResNet50 model parameters are updated. The ResNet50 model parameters specifically include convolution kernel weights and convolutional layer bias parameters.

[0099] The objective of making specific adjustments is:

[0100]

[0101] In the formula, δ fine-tune This represents the objective function value adjusted specifically. This represents the conditional probability model, indicating the probability distribution of label Y given input O. Specifically, it represents the model's ability to generate output. It may be the result of a certain network structure or probabilistic model. Y represents the feature vector of a high-response patch unit image, and O represents the set of feature vectors for all high-response patch unit images. The specific formula for the set of feature vectors for all high-response patch unit images is as follows:

[0102]

[0103] In the formula, Agg represents the aggregation function, which is used to integrate multiple high-response patch unit image representations into a full-slice image representation.

[0104] The CFE feature extraction network module, with its backbone network parameters determined, underwent fully supervised training, where high-response patch unit images were standardized to 224*224 pixels. During fully supervised training, the cross-entropy loss function was used as the optimization objective, and the Adam optimizer was employed for gradient updates. Appropriate learning rate scheduling and regularization strategies were combined to prevent overfitting. Image enhancement operations, such as random rotation and horizontal flipping, were introduced to improve the model's generalization ability under complex data distributions. This training strategy, while retaining the learned structural sensitivity, further enhanced the model's stable representation of local organizational structures and its class discrimination performance.

[0105] Through fully supervised training in this stage, the CFE feature extractor evolves from a weakly supervised structure-focused model into a general image encoder with task adaptability and multi-class recognition capabilities, providing a stable and transferable feature foundation for the entire CTE-MIL model.

[0106] The attention mechanism optimization module is used to construct the TFAM attention mechanism module after the CFE feature extractor is trained, using the extracted feature vector as input. This module calculates the attention score for each patch, performs weighted feature fusion based on the attention score, generates a fused feature vector, and uses the fused feature vector as input to the MIL classifier. The MIL classifier is trained with the corresponding pathological tissue type as the label, and the trained MIL classifier can distinguish the pathological tissue type.

[0107] The TFAM (Two-Focused Attention Mechanism) module is used to set specific attention scores for the input feature vectors.

[0108] The formula used to calculate the attention score is as follows:

[0109]

[0110] In the formula, Let a be the attention score for the l-th feature vector. l and b l These are the attention features and fused gated attention features of the l-th high-response patch unit image, processed by the tanh activation function, respectively. C For the already combined feature a l and b l The weight matrix for linear transformation, ⊙ denotes element-wise multiplication, l is the index of the input feature vector, l∈[1,L], where a l and b lThe specific formula used for the calculation is as follows:

[0111]

[0112] In the formula, tanh is the tanh activation function. For the image of the l-th high-response patch unit, the feature vector is scaled down, where σ represents the sigmoid function, and W... a and W b are the weight matrices for the attention features and the gated output features, respectively. The formula used for the calculation is:

[0113]

[0114] In the formula, ω c h is the channel attention weight vector. l This is the feature vector of the l-th high-response patch unit image;

[0115] Where the channel attention weight vector ω c The specific formula used for the calculation is as follows:

[0116]

[0117] In the formula, W4 and W5 are both weight matrices of the channel attention weight vector. This represents the feature vector after global average pooling, calculated using the following formula:

[0118]

[0119] Feature fusion is performed based on the attention scores of each feature vector to obtain a fused feature vector. The formula used to calculate the fused feature vector is as follows:

[0120]

[0121] In the formula, z is the fused feature vector, and softmax is the softmax activation function.

[0122] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0123] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0124] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0125] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A colorectal cancer pathological image analysis platform based on CTE-MIL deep learning architecture, characterized in that, The specific steps include: The foreground training image processing module is used for collecting a plurality of known colorectal cancer pathological regions and corresponding pathological tissue types of colorectal pathological whole section images, pre-processing each collected colorectal pathological whole section image, recording the pre-processed whole section image as a sample training image, mapping the sample training image and the colorectal pathological whole section image one by one, and generating a foreground extraction training data set; The segmentation network training module is used for taking the CLAM algorithm model as a theoretical model, constructing a Tissue-Seg segmentation network module in the theoretical model, training the Tissue-Seg segmentation network module based on the data in the foreground extraction training data set, introducing a spatial pyramid pooling fast algorithm into the Tissue-Seg segmentation network module, perceiving the organization structure and boundary profile of the colorectal cancer pathological region, outputting the corresponding pathological tissue foreground mask from the trained Tissue-Seg segmentation network model, and mapping the foreground mask to the original colorectal pathological whole section image to realize foreground segmentation at the colorectal pathological whole section image level; The feature unit screening module is used for obtaining a plurality of patch unit images by grid division on the obtained foreground mask image; the backbone network of the CFE feature extraction network module is pre-trained based on the patch unit image, and the semantic response score of each patch image is calculated by using a single hidden layer attention scoring network; the patch images are sorted and screened according to the semantic response score, and the patch unit images meeting the requirements are recorded as high-response patch unit images; the pre-training refers to weakly supervised training of the backbone network of the CFE feature extraction network module by taking the patch unit image as input and taking the image-level pathological tissue type of the whole image corresponding to the patch unit image as label; The feature extraction network training module is used for constructing the CFE feature extraction network module, specifically adjusting the backbone network parameters of the pre-trained CFE feature extraction network module based on the high-response patch unit image, and fully supervising the training of the CFE feature extraction network module with the determined backbone network parameters, specifically: taking the high-response patch unit image as the input of the CFE feature extraction network model after uniform size, and taking the corresponding feature vector type as the label, training the CFE feature extraction network model; The attention mechanism optimization module is used for constructing a TFAM attention mechanism module by taking the feature vector extracted by the CFE feature extractor as input after the training of the CFE feature extractor is completed, calculating the attention score of each patch through the module, performing feature weighted fusion based on the attention score, generating a fusion feature vector, and taking the fusion feature vector as the input of the MIL classifier, taking the corresponding pathological tissue type as the label to train the MIL classifier, and realizing the discrimination of the pathological tissue type by the trained MIL classifier.

2. The CTE-MIL deep learning architecture based platform for colorectal cancer pathological image analysis according to claim 1, wherein: A plurality of known colorectal cancer pathological regions and corresponding pathological tissue types of colorectal pathological whole section images are collected, wherein the corresponding pathological tissue types cover 2 types of colorectal cancer tissues, specifically including: colon adenoma tissue and rectal adenoma tissue; The collected colorectal pathological whole section images are preprocessed, and the specific preprocessing steps include: performing downsampling processing on the uniform size colorectal pathological whole section images, reducing the original image to 1 / 40 of the original size, and the specific formula for performing downsampling processing is: In the formula, output(i,j) represents the output pixel of the i-th row and j-th column of the colorectal pathological whole section image after downsampling processing, k is the downsampling factor, which is 40, m represents the current row offset, n represents the current column offset, Input(i*k+m,j*k+n) represents the original pixel value at position (i*k+m,j*k+n), i and j represent the row index and column index of the image after downsampling, respectively; Based on the downsampling processed colorectal pathological whole section image, the Labelme tool is used to label the colorectal cancer pathological region in the downsampling processed image, and the specific labeling method is: the colorectal cancer pathological tissue is labeled by the maximum circumscribed polygon, and the labeled colorectal pathological whole section image is processed by a script to generate a corresponding foreground mask, which is a binary mask that distinguishes between labeled colorectal cancer pathological tissue and colorectal pathological whole section image background, thus completing the preprocessing of the colorectal pathological whole section image. The generation method of the foreground extraction training data set is: one-to-one mapping of the downsampling processed colorectal pathological whole section image and the corresponding foreground mask to form a corresponding grid, and the formed grid is denoted as the foreground extraction training data set.

3. The CTE-MIL deep learning architecture based colorectal cancer pathological image analysis platform according to claim 2, wherein: Based on the data in the foreground extraction training data set, the Tissue-Seg segmentation network module is trained, wherein the Tissue-Seg segmentation network module uses convolution Conv+batch normalization BN+SiLU activation as the basic unit, and introduces a spatial pyramid pooling fast algorithm into the Tissue-Seg segmentation network module for perceiving the tissue structure and boundary contour of the colorectal cancer pathological region, wherein the spatial pyramid pooling fast algorithm specifically includes multi-scale pooling and feature splicing. The specific logic of training the Tissue-Seg segmentation network module is: taking the downsampling processed colorectal pathological whole section image in the foreground extraction training data set as input, and taking the corresponding foreground mask as label, training the Tissue-Seg segmentation network module, using the BCELoss function as the loss function during training, and using the Adam optimizer with an initial learning rate of 0.001, wherein the specific expression of the BCELoss function is: In the formula, represents the loss of the pth pixel prediction value of the training sample and the true value, y p represents the true label of the pixel, is the predicted probability value of the model for the pixel, where p is the index of the pixel point of the training sample image, and N is the total number of pixels in the image.

4. The CTE-MIL deep learning architecture based colorectal cancer pathological image analysis platform according to claim 1, wherein: The obtained foreground mask image is meshed and divided to obtain a plurality of patch unit images, and the specific steps of obtaining the plurality of patch unit images include: setting a mesh size, obtaining the size of the mask image, calculating the number of rows and columns of the mesh according to the mesh size and the size of the mask image, dividing the image into a plurality of patch unit images by traversing the image in a loop, and extracting each patch unit image using a slicing operation, wherein the number of patch unit images is not less than 1000; the foreground mask image specifically refers to an image formed by mapping the foreground mask corresponding to the pathological tissue onto the original colorectal pathological whole section image; The semantic response score is calculated by a single hidden layer attention scoring network, and the formula for calculating the semantic response score is: π q = sigmoid [W2 * ReLU (W1h q + b1) + b2] In the formula, π q The semantic response score of the qth patch unit image is represented, and the specific range is between [0, 1], h q The feature vector of the qth patch unit image is h W1 and W2 are weight matrices, b1 and b2 are bias vectors, ReLU is a ReLU activation function, sigmoid represents a sigmoid activation function, and q is a patch unit image index.

5. The CTE-MIL deep learning architecture based platform for colorectal cancer pathological image analysis of claim 4, wherein: a feature vector h of the qth patch unit image q The feature vector h is extracted by completing a pre-trained ResNet50 model, which is a backbone network of a CFE feature extraction network module, and a linear fully connected layer in the ResNet50 model is deleted, and the feature dimension is 1024. q The feature vector h includes boundary structure, morphological distribution and texture semantic feature information of pathological tissues in a patch unit image region. The patch unit images are screened according to the semantic response score, and the patch unit images meeting the requirements are recorded as high-response patch unit images, and the specific screening logic is: based on the semantic response score of the patch unit image, the patch unit images are arranged in descending order according to the semantic response score to form an adjusted unit image sequence, and the first 1000 patch unit images are selected from the adjusted unit image sequence and recorded as high-response patch unit images.

6. The CTE-MIL deep learning architecture based platform for colorectal cancer pathological image analysis of claim 5, wherein: The backbone network of the CFE feature extraction network module is specifically a ResNet50 model, wherein the ResNet50 model is pre-trained, and a standard cross-entropy loss function is used for optimization in the pre-training process, the input is all patch unit images, the image-level pathological tissue type corresponding to the whole image is used as a label, a supervision signal is provided through the label, the network is guided to learn an intermediate representation with discriminative ability, and a classification is not directly output. The finally trained model can generate a feature vector expressing the semantic information of each patch, and the convolutional layers of the pre-trained ResNet50 model are frozen and unfrozen in the specificity adjustment stage; The specificity of the backbone network parameters of the CFE feature extraction network module is adjusted based on the high-response patch unit image, and the specific adjustment logic is: a sparse training subset is constructed with the high-response patch unit image as input, and the image-level pathological tissue type corresponding to the high-response patch unit image is used as a label, the ResNet50 model parameters are updated through end-to-end adjustment training, and the parameters of the ResNet50 model specifically include convolution kernel weights and bias parameters of convolutional layers.

7. The CTE-MIL deep learning architecture based platform for colorectal cancer pathological image analysis according to claim 6, wherein: The CFE feature extraction network module with determined backbone network parameters is subjected to full supervision training, and a cross-entropy loss function is used as an optimization objective in the full supervision training process, an Adam optimizer is used for gradient update, and an image enhancement operation is introduced, and the image enhancement operation specifically includes random rotation, horizontal flip and scale change.

8. The CTE-MIL deep learning architecture based platform for colorectal cancer pathological image analysis according to claim 1, wherein: A specific attention score is set for the input feature vector, and the formula for calculating the attention score is: In the formula, is the attention score of the lth feature vector, a l and b l are the lth high-response patch unit image, the attention feature processed by the tanh activation function, and the fusion gate attention feature, respectively, W C is the weight matrix for linear transformation of the combined features a l and b l , is the element-wise multiplication, l is the index of the input feature vector, l∈[1,L], L is the total number of input feature vectors, wherein a l and b l The formula according to which the calculation is specifically based on is: wherein tanh is a tanh activation function, is the feature vector of the lth high response patch unit image after scaling, and a and W b are the weight matrices of the attention feature and the gating output feature, respectively, wherein The formula for calculation is: In the formula, ω c is a channel attention weight vector, h l is a feature vector of the lth high-response patch unit image; The features are fused based on the attention scores of the feature vectors to obtain a fused feature vector, and the formula for calculating the attention score is: The formula for calculating the fusion feature vector is: In the formula, z is the fusion feature vector, and softmax is a softmax activation function.

Images