Weakly supervised pathological image tissue segmentation method based on text prompt learning

Abstract

The invention discloses a weak supervision pathological image tissue segmentation method based on text prompt learning. The method comprises the steps of feature extraction and initial class activation graph generation; using an MCRM module to optimize the initial class activation graph to obtain a refined class activation graph; and aggregating the plurality of refined class activation graphs to form a fused pseudo mask, taking the fused pseudo mask as a supervision signal, training a segmentation model, and after the training is completed, segmenting the new pathological image tissue by using the segmentation model. According to the method, a text prompt learning mechanism is utilized to focus the model on learning high-discrimination features, so that the influence of tissue co-occurrence is reduced. An initial class activation graph is optimized through a multi-mode class activation graph refining module, and the integrity of boundary segmentation is enhanced. Meanwhile, pseudo masks from different network layers are fused to train a segmentation model, and semantic segmentation of the pathological image is realized. According to the method, high-annotation data dependence is effectively relieved, and the generalization ability of the model is improved, so that application in the field of artificial intelligence-assisted medical treatment is promoted.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically a weakly supervised pathological image tissue segmentation method based on text prompt learning. Background Technology

[0002] Histopathological images contain rich information about the tumor microenvironment, revealing the mechanisms of tumor development and progression, and are the gold standard for cancer diagnosis. However, manual analysis of pathological images is time-consuming, labor-intensive, and highly dependent on the professional clinical knowledge and experience of pathologists.

[0003] In recent years, various supervised learning-based methods have been able to automatically identify and segment tissue structures in pathological images through precise, dense pixel-level annotation, thereby reducing the workload of pathologists. However, these reliable manual annotation processes are labor-intensive and highly dependent on physician expertise, limiting the application of supervised learning methods in actual clinical diagnosis.

[0004] Therefore, it is crucial to accurately analyze tissue structures in pathological images without requiring extensive pixel-level annotation.

[0005] Previous studies have confirmed that weakly supervised semantic segmentation (CAM) can achieve this goal with coarse image-level annotation. Class activation maps generated by classification networks can learn pixel-level pseudomasks from image-level labels. Based on this concept, many CAM-based weakly supervised semantic segmentation methods have been proposed in recent years for pathological image segmentation. Most of these methods use convolutional neural networks as their backbone architecture. These networks typically focus on the most discriminative regions, resulting in incomplete tissue structure localization and poor contour delineation. To address this, researchers have introduced techniques such as SAM, unbiased active learning, and superpixel clustering to optimize CAM, thereby further improving its performance in tissue segmentation tasks.

[0006] The prospects for weakly supervised semantic segmentation methods based on CAM (Cardiography-Assisted Localization) are limited because they primarily borrow from CAM designs used in natural image processing. This is mainly because histopathological images lack the specific semantic knowledge inherent in natural images and possess their own unique characteristics. For example, multiple tissue types often coexist within a single image segment (i.e., tissue co-occurrence), leading to blurred activation in CAM-based methods. Due to feature competition, when coexisting tissues have similar texture patterns, the discriminative region may be suppressed or incorrectly activated. Furthermore, the flexibility and variability of tissue morphology often result in blurred or overlapping boundaries in pathological tissues (i.e., blurred boundaries), which severely hinders the effectiveness of CAM localization methods lacking pixel-level supervision. Therefore, a method that can effectively utilize weakly supervised information and improve segmentation accuracy is needed. Summary of the Invention

[0007] The purpose of this invention is to provide a weakly supervised pathological image tissue segmentation method based on text prompt learning, so as to solve the problem of inaccurate segmentation caused by tissue co-occurrence and blurred boundaries.

[0008] The technical solution to achieve the objective of this invention is:

[0009] A weakly supervised pathological image tissue segmentation method based on text prompting learning includes the following steps:

[0010] Input a pathological tissue image and the corresponding text prompt, use a pre-trained text encoder and image encoder to extract text features and image features respectively, and use a cross-attention mechanism to interact the features to generate an initial class activation map;

[0011] The initial class activation map is optimized using the Multimodal Context Optimization Module (MCRM), and boundary optimization is performed using text features to obtain a refined class activation map.

[0012] Multiple refined class activation maps are aggregated to form a fusion pseudomask, and the fusion pseudomask is used to train a segmentation model. After training, the segmentation model is used to segment tissues in new pathological images.

[0013] Preferably, the text encoder is a pre-trained MLTE encoder, used to convert text tags into dense embeddings and incorporate location information to obtain text features;

[0014] The image encoder is a ResNet encoder, used to extract multi-scale visual features from pathological tissue images.

[0015] Preferably, in the cross-attention mechanism, text features are used as the query matrix, and image features are used as the key matrix and value matrix, and an initial class activation map is generated by calculating attention scores.

[0016] Preferably, the MCRM module uses a reverse attention mechanism, where image features are used as the query matrix and text features are used as the key and value matrices to align local features of the image with text semantics and optimize the boundaries of the class activation graph.

[0017] Preferably, the fused pseudomask is generated by aggregating refined class activation graphs of multiple network layers to capture multi-scale contextual information.

[0018] Preferably, the method uses a hybrid loss function to train the model, including classification loss, alignment loss, and segmentation loss;

[0019] Classification loss is used to indicate whether the target category described in the text prompt exists;

[0020] Alignment loss includes negative sample suppression loss and total variation regularization loss;

[0021] The segmentation loss includes cross-entropy loss and Dice loss.

[0022] The beneficial effects of this invention are as follows: by learning through text prompts, the reliance on pixel-level annotation is reduced, thus lowering annotation costs; by utilizing multimodal interaction and reverse attention mechanisms, the problems of tissue co-occurrence and blurred boundaries are effectively alleviated, thereby improving segmentation accuracy and model generalization ability. Attached Figure Description

[0023] Figure 1 This is a flowchart of the method of the present invention.

[0024] Figure 2 This is a schematic diagram of the network architecture of the method of the present invention. Detailed Implementation

[0025] The present invention will now be described in detail with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0026] Example

[0027] Weakly supervised tissue segmentation methods for pathological images based on text prompts, such as Figure 1 and Figure 2 As shown, it includes the following steps:

[0028] Feature extraction and initial class activation map generation:

[0029] Text features are extracted using a pre-trained MLTE encoder, and image features are extracted using a ResNet encoder. Through a cross-attention mechanism, text features are used as the query matrix, and image features are used as the key and value matrices. Attention scores are calculated to generate an initial class activation map.

[0030] Class activation graph optimization: The initial class activation graph is optimized using the MCRM module. This module adopts a reverse attention mechanism, using image features as the query matrix and text features as the key and value matrices, aligning local features with text semantics, optimizing the boundaries of the class activation graph, and obtaining a refined class activation graph.

[0031] Pseudomask generation and segmentation training: Multiple refined class activation maps are aggregated to form a fused pseudomask. The fused pseudomask is used as a supervision signal to train the DeepLabV3+ segmentation model. During training, a hybrid loss function is used, including classification loss, alignment loss (negative sample suppression loss and total variation regularization loss), and segmentation loss (cross-entropy loss and Dice loss). After training, the segmentation model is used to segment new pathological image tissues.

[0032] The specific loss function is as follows:

[0033] The classification loss function is:

[0034] ; in, This represents a category label, indicating whether a category exists. This represents the average value of the class activation map.

[0035] The negative sample suppression loss function is:

[0036] ; The total variation regularized loss function is:

[0037] ; in, Initial class activation graph Activate the pixel y-coordinate in the initial class graph. The x-coordinate of the pixel in the initial class activation graph.

[0038] In this invention, during the segmentation model training phase, a decoder converts the class activation map into a pseudomask. The decoder consists of a sequence of upsampling layers and convolutional layers. Due to the inherent challenges of weak supervision, relying solely on a single pseudomask may lead to incomplete segmentation. To overcome this, this invention employs a fusion module that combines pseudomasks from multiple network layers, utilizing complementary activations from different resolutions to generate a more robust and complete pseudomask. The final step is to train a robust, fully supervised segmentation network.

[0039] This invention employs the DeepLabV3+ architecture as the segmentation model. This network is trained end-to-end to learn the mapping from an input medical image to its corresponding pixel-level segmentation map. The training objective is to minimize the difference between the segmentation network's predictions and a carefully generated fused pseudomask. This is achieved by incorporating a standard segmentation loss function, which is:

[0040] ; in, For cross-entropy loss, Let G(I) be the Dice loss, G(I) be the prediction result of the segmentation model, and PM be the fusion pseudomask.

[0041] This invention also employs auxiliary loss functions to guide different aspects of the model, utilizing two auxiliary loss terms: one for classification. and for optimization modules .

[0042] The final loss of the weakly supervised semantic segmentation process based on text prompts can be defined as: .

[0043] The overall optimization goal of the model is to obtain the optimal segmentation model after training using hybrid loss.

[0044] The pseudocode for the algorithm structure of the weakly supervised pathological image tissue segmentation method based on text prompt learning in this invention is as follows:

[0045] .

[0046] The network architecture of the method of this invention, such as Figure 2 As shown, the framework of Phase 1 includes a ResNet encoder for learning medical image features, an MLTE encoder for parsing text features, a multi-layer module for generating an initial class activation map, specifically including multimodal interaction, combining cross-attention to fuse image and text features, activation functions, an upsampling module, and an MCRM module for optimizing the initial class activation map; the framework of Phase 2 includes a decoder for converting the class activation map into a pseudomask, a fusion unit for fusing multi-layer pseudomasks, and a segmentation model.

[0047] This invention improves upon text-based prompting learning by utilizing text semantic features. It aims to construct a pseudo-mask using an image encoder and a text encoder, and then train a segmentation model to solve the problem of high manual annotation costs for medical image data and improve segmentation accuracy.

[0048] The pathological image tissue segmentation method based on text prompt learning of the present invention has the following beneficial effects:

[0049] The application of text-based learning significantly improves the accuracy of semantic segmentation models for pathological images; it allows segmentation training in the absence of pixel-level annotations, enhancing the clinical applicability of the model; and it reduces reliance on pixel-level annotated medical images, alleviating the burden on pathologists.

[0050] Comparative experiment:

[0051] 1. Experimental Preparation

[0052] To demonstrate the superiority of this invention for semantic segmentation of tissues in weakly supervised pathological images, experiments were conducted on two publicly available datasets.

[0053] The publicly available datasets for this invention are LUAD-HistoSeg and BCSS-WSSS. LUAD-HistoSeg is a dataset for weakly supervised semantic segmentation of lung adenocarcinoma pathological images. It contains 16,678 training patches, with only patch-level annotation used during training. The validation and test sets each have an additional 300 and 307 pixel-level annotated patches, respectively. Blurry, contaminated, background-excessive, and abnormally stained regions and divergent samples have been removed. It is primarily used for segmenting four types of regions: tumor epithelium, tumor-associated stroma, necrosis, and lymphocytes, and is suitable for training pixel-level segmentation models with limited annotation. BCSS-WSSS, on the other hand, is a dataset for weakly supervised semantic segmentation of breast cancer pathological images, derived from a breast cancer semantic segmentation dataset. The training set contains 23,422 patches, employing patch-level annotation for training and pixel-level ground truth annotation for validation and testing. It provides RGB images and corresponding segmentation masks, supporting the training and evaluation of weakly supervised segmentation models for breast cancer pathological images. Both datasets focus on weakly supervised semantic segmentation tasks; the former focuses on lung adenocarcinoma tissue microenvironment segmentation, while the latter provides large-scale weakly supervised training samples for breast cancer, reducing the cost of manual annotation while facilitating high-precision histological image segmentation research.

[0054] 2. Comparative Experiment

[0055] This method not only compares experimental results with traditional weakly supervised segmentation models, but also with some of the latest methods in the field.

[0056] Grad-CAM++ is an interpretable method based on gradient-weighted class activation mapping that can accurately locate multiple instances of the same class in a single image.

[0057] OEEM is a training strategy for weakly supervised gland segmentation in histological images that can suppress noise interference by dynamically adjusting loss weights.

[0058] HisynSeg is a weakly supervised semantic segmentation framework for pathological images based on image fusion synthesis and consistency regularization. It can combine image filtering and self-supervised consistency constraints to transform weakly supervised tasks into fully supervised training.

[0059] By comparing with methods used by other researchers, it has been demonstrated that the method of this invention can effectively perform semantic segmentation of pathological tissue images, and the segmentation accuracy is superior to other methods.

[0060] Table 1. Comparison and analysis results of this invention with other network models on public datasets (Test IoU) .

[0061] The data analysis results in Table 1 show that GradCAM++, as a traditional weakly supervised segmentation method, achieved good results on both datasets. However, due to the differences in the datasets themselves, GradCAM++'s performance varied slightly between the two datasets. Furthermore, it can be seen that the OEEM method, designed for gland segmentation, performed worse than GradCAM++ on the LUAD-HistoSeg dataset, but outperformed GradCAM++ on the BCSS-WSSS dataset. This phenomenon indicates that in practical applications involving complex and varied diseases, the generalization ability of weakly supervised models faces performance bottlenecks.

[0062] Of particular note is that the recent HisynSeg method in the field of weakly supervised segmentation achieved good results on both datasets, and its performance difference between the two datasets was smaller than that of GradCAM++ and OEEM, demonstrating a certain level of model generalization ability and segmentation performance. Compared to the HisynSeg method, our proposed method exhibits superior segmentation performance on both datasets. Similarly, the difference between our proposed method and the two datasets is also smaller. These results demonstrate that the method proposed in this invention performs excellently on semantic segmentation tasks and possesses stable generalization ability.

[0063] This invention presents a text-based prompt learning-based method for pathological image tissue segmentation, aiming to improve the segmentation performance of weakly supervised pathological images. By incorporating text-based prompt learning, the dependence of semantic segmentation models on pixel-level annotation is reduced, alleviating the human burden and significantly improving data annotation efficiency. Simultaneously, the introduction of text-based prompt learning enhances segmentation performance; through cross-attention constraints, the identification and localization of regions such as tumors are more accurate, and the model's generalization ability is also improved. Combining the advantages of text-based prompt learning and weakly supervised learning is particularly important in today's complex medical environment. This method provides a new approach to advancing medical image segmentation technology and contributes to more efficient decision support in clinical diagnosis.

Claims

1. A weakly supervised pathological image tissue segmentation method based on text prompting learning, characterized in that, Includes the following steps: Input a pathological tissue image and the corresponding text prompt, use a pre-trained text encoder and image encoder to extract text features and image features respectively, and use a cross-attention mechanism to interact the features to generate an initial class activation map; The initial class activation map is optimized using the Multimodal Context Optimization Module (MCRM), and boundary optimization is performed using text features to obtain a refined class activation map. Multiple refined class activation maps are aggregated to form a fusion pseudomask, and the fusion pseudomask is used to train a segmentation model. After training, the segmentation model is used to segment tissues in new pathological images.

2. The weakly supervised pathological image tissue segmentation method according to claim 1, characterized in that: The text encoder is a pre-trained MLTE encoder, used to convert text tags into dense embeddings and incorporate location information to obtain text features; The image encoder is a ResNet encoder, used to extract multi-scale visual features from pathological tissue images.

3. The weakly supervised pathological image tissue segmentation method according to claim 1, characterized in that: In the cross-attention mechanism, text features are used as the query matrix, and image features are used as the key matrix and value matrix. An initial class activation graph is generated by calculating attention scores.

4. The weakly supervised pathological image tissue segmentation method according to claim 1, characterized in that: The MCRM module uses a reverse attention mechanism, where image features are used as the query matrix and text features are used as the key and value matrices to align local features of the image with text semantics and optimize the boundaries of the class activation graph.

5. The weakly supervised pathological image tissue segmentation method according to claim 1, characterized in that: The fusion pseudomask is generated by aggregating refined class activation graphs from multiple network layers to capture multi-scale contextual information.

6. The weakly supervised pathological image tissue segmentation method according to claim 1, characterized in that: The method uses a hybrid loss function to train the model, which includes classification loss, alignment loss, and segmentation loss. The classification loss is used to indicate whether the target category described in the text prompt exists or not; The alignment loss includes negative sample suppression loss and total variation regularization loss; The segmentation loss includes cross-entropy loss and Dice loss.

7. The weakly supervised pathological image tissue segmentation method according to claim 6, characterized in that: The classification loss function is: ； in, This represents a category label, indicating whether a category exists. This represents the average value of the class activation map.

8. The weakly supervised pathological image tissue segmentation method according to claim 6, characterized in that: The negative sample suppression loss function is: ； The total variation regularized loss function is: ； in, Initial class activation graph Activate the pixel y-coordinate in the initial class graph. The x-coordinate of the pixel in the initial class activation graph.

9. The weakly supervised pathological image tissue segmentation method according to claim 6, characterized in that: The segmentation loss function is: ； in, For cross-entropy loss, Let G(I) be the Dice loss, G(I) be the prediction result of the segmentation model, and PM be the fusion pseudomask.

10. The weakly supervised pathological image tissue segmentation method according to claim 1, characterized in that: The segmentation model is based on the DeepLabV3+ architecture.

Images