Self-adaptive prototype learning and visual double-flow collaborative WSI classification method

Abstract

The invention discloses a self-adaptive prototype learning and visual double-flow collaborative WSI classification method. The method comprises the following steps: firstly, cutting WSI into blocks and extracting image block features; a self-adaptive enhanced prototype mining strategy is adopted, and a static visual prototype obtained by combining a hierarchical clustering strategy and a self-adaptive learning dynamic prototype form visual prompt representation; then, generating instance-level text description and slice-level text description by utilizing LLM, respectively putting the instance-level text description and the slice-level text description into a template for word segmentation, embedding and splicing learnable tokens to obtain corresponding text prompts, and obtaining corresponding text features; effective information is decoupled from the features of the image blocks by using the feature sensors driven by two prompts and utilizing multi-mode prompts respectively; and finally, fusing the decoupled effective information by using a panoramic representation reconstructor to obtain slice-level representation, and performing category prediction. The method can effectively guide the model to accurately focus on the key pathological region, and remarkably improves the classification performance and the thermodynamic diagram positioning capability.

Description

Technical Field

[0001] This invention belongs to the technical field of pathological digital images, specifically relating to a WSI classification method that combines adaptive prototype learning with visual-verbal dual-stream collaboration. Background Technology

[0002] Histopathological examination is internationally recognized as the "gold standard" for cancer diagnosis. Pathologists observe and analyze the morphology and arrangement of cells in tissue sections under a microscope, providing precise evidence for assessing tumor status and patient prognosis. However, traditional pathological diagnosis is highly dependent on manual labor and faces numerous bottlenecks. Due to the severe scarcity and uneven distribution of pathologists, manual slide reading is extremely labor-intensive, cumbersome, and the diagnostic results are somewhat constrained by the doctor's subjective experience and fatigue, resulting in poor reproducibility. In recent years, the maturity and widespread adoption of whole-slide scanning technology have driven the digital and intelligent transformation of pathology. Computational pathology has emerged, utilizing computer vision and machine learning algorithms to automate the analysis of whole-slide images (WSI), aiming to improve the efficiency, accuracy, and objectivity of diagnosis. However, directly applying deep learning technology to WSI analysis still faces challenges. A single WSI image can have a resolution of up to one billion pixels, far exceeding that of conventional images by several orders of magnitude, making it impossible to input standard convolutional neural networks for end-to-end training. Simple downsampling, on the other hand, results in the loss of a significant amount of cellular-level microscopic information. Furthermore, deep learning methods rely on large-scale, high-quality labeled data. However, experienced pathologists are scarce in my country, and pixel-level fine annotation requires a significant investment of time and effort, making it extremely costly. Therefore, most WSIs only have coarse-grained labels at the slice level (such as positive or negative), lacking fine annotations suitable for fully supervised learning. Additionally, due to patient privacy and ethical concerns, as well as various rare diseases, collecting a sufficient number of the required types of WSIs remains challenging.

[0003] Currently, there are three main approaches for WSI classification: The first is patch-based classification. This method relies on doctors manually annotating regions of interest (ROIs), dividing the WSI into smaller patches, and then using deep learning algorithms to extract features and classify each patch. However, this method requires fine-grained annotation of a massive number of patches. Given the scarcity of qualified pathologists, the annotation time and economic costs of this fully supervised learning are extremely high, and it is difficult to quickly scale to new diseases or large-scale datasets. The second approach is based on Multiple Instance Learning (MIL), which has become the mainstream paradigm for WSI classification. In the MIL framework, a WSI is considered a "bag," and the numerous non-overlapping image patches segmented from the WSI are considered "instances" within the bag. The model only needs to be trained using slice-level labels. Its core assumption is that if a bag (WSI) is labeled positive, then it contains at least one positive instance (tumor image patch). This assumption naturally aligns with the logic of pathological diagnosis, making MIL an ideal choice for WSI analysis in the absence of fine annotations. While traditional MIL methods lower the labeling threshold by requiring only WSI bag-level labels, their performance heavily relies on massive training samples. When faced with scarce rare disease data or insufficient samples due to privacy ethical restrictions, traditional MIL methods struggle to learn robust feature representations and exhibit poor performance. Furthermore, because the model learns features implicitly from raw image data without prior knowledge guidance, it is highly sensitive to data distribution shifts, exhibits poor generalization performance, and typically has poor interpretability.

[0004] The third approach involves prompting and fine-tuning the visual language model. This method primarily utilizes efficient prompting and fine-tuning of pre-trained visual language models (VLMs) to provide an effective solution for the task of few-shot weakly supervised whole-slice image classification (FSWC). This strategy fully leverages the rich semantic and visual association knowledge learned by VLMs from massive multimodal data and transfers it to specific pathological scenarios. This avoids the stringent resource and data requirements of pre-training large models from scratch and effectively improves the generalization performance of the model under few-shot conditions. For example, Qu et al. first proposed a two-level prompt learning MIL framework that integrates language prior knowledge by fine-tuning CLIP (Qu, Linhao, et al. "The rise of ai language pathologists: Exploring two-level prompt learning for few-shot weakly-supervised whole slide image classification." Advances in Neural Information Processing Systems 36 (2023): 67551-67564.) to solve the FSWC task; Shi et al. (Shi, Jiangbo, et al. "Vila-mil: Dual-scale vision-language multiple instance learning for whole slide image classification." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024.) used dual-scale visual descriptive text prompts to guide the model to focus on pathological features at different scales, further improving classification performance.Furthermore, Qu et al. argued that textual prompts alone are often insufficient for complex pathological images, and proposed the PEMP method (Qu, Linhao, et al. "Pathology-knowledge enhanced multi-instance prompt learning for few-shot whole slide image classification." European conference on computer vision. Cham: SpringerNature Switzerland, 2024.) to further introduce visual prompts, helping the model better understand visual features. This third type of method effectively addresses the problems of high annotation costs, reliance on large amounts of training data, and poor generalization with small samples found in the first and second types of methods, but it suffers from a "cue information bottleneck." This is because plain text cues suffer from modal gaps and descriptive limitations. Most existing techniques (including some multi-text cues) attempt to guide models by adding textual descriptions of staining, morphology, and other dimensions. However, pathological images contain high-dimensional and complex microscopic visual textures, such as inter-tumor heterogeneity and complex matrix reactions, which are inherent visual features that are difficult to accurately describe with limited discrete text, thus failing to precisely guide the model to focus on important detailed features. Furthermore, some existing methods for introducing visual cues face problems of subjective selection bias and insufficient representativeness. Some studies (such as PEMP) attempt to rely on experts manually selecting a small number of typical patch and slice examples as visual cues, but this approach is subjective, causing model performance to be limited by the quality of the selected examples. Since expert selection often focuses on the most typical morphological features, it is difficult to consider features at the boundary or in a blurred state, resulting in the loss of some discriminative information. On the other hand, manually constructed small sample sets cannot statistically represent the complex visual distribution of the entire dataset, limiting the model's generalization ability in real-world scenarios. Summary of the Invention

[0005] This invention addresses the bottlenecks in VLM (Visual-Language-Based Learning) prompting fine-tuning methods, such as the deficiencies of pure text prompts, human selection bias, and lack of representativeness of prompts. It provides a WSI (Wide-Site Indicator) classification method that combines adaptive prototype learning with visual-linguistic dual-modal collaboration. The method introduces visual prompting strategies to compensate for the inherent defects of pure text descriptions and the limitations of human selection. Furthermore, it constructs a deep collaborative and complementary mechanism for visual-linguistic dual-modal prompts, effectively guiding the model to accurately focus on key pathological areas and significantly improving classification performance and heatmap localization capabilities.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An adaptive prototype learning and vision-speech dual-stream collaborative WSI classification method includes the following steps:

[0008] The WSI tissue is segmented from the background and cut into image patches, and image patch features are extracted using an image encoder;

[0009] An adaptive and enhanced prototype mining strategy is adopted to form visual cue representations. Specifically, a hierarchical clustering strategy is used to perform two-level clustering of image patch features to obtain static visual prototypes; a set of learnable dynamic prototypes is introduced and spliced ​​with the static visual prototypes to form visual cue representations.

[0010] Instance-level and slice-level text descriptions are generated using a large language model. These descriptions are then put into templates for word segmentation and embedding, and then concatenated to learnable tokens to obtain corresponding text prompts. Finally, these tokens are input into a text encoder to obtain the corresponding text features.

[0011] Instance-level text features and image patch features are input together into a cue-driven feature perceptron to obtain a decoupled representation of text cue; visual cue representation and image patch features are input together into a cue-driven feature perceptron to obtain a decoupled representation of visual cue.

[0012] The panoramic representation reconstructor is used to fuse the representations of text cues decoupled from visual cues to obtain slice-level representations.

[0013] Calculate the similarity between slice-level representations and slice-level text features to obtain category prediction results.

[0014] As a preferred technical solution, the method of using a hierarchical clustering strategy to perform two-level clustering of image patch features to obtain a static visual prototype specifically involves:

[0015] Extract all image patch features for each WSI and perform unsupervised clustering to obtain k local cluster centers for each WSI;

[0016] For each WSI, the k local cluster centers are further clustered to extract the m most representative global cluster centers, which serve as the static visual prototype for each WSI.

[0017] As a preferred technical solution, the image encoder uses the Vision Transformer in the CLIP model; the large language model adopts Chat-GPT; and the text encoder adopts the Transformer in the CLIP model.

[0018] As a preferred technical solution, during the embedding process, category information is embedded into the corresponding text description; after embedding, a learnable token is concatenated to obtain the corresponding text prompt.

[0019] As a preferred technical solution, the cue-driven feature perceptron includes an input layer, a self-attention layer, a cross-attention layer, and a fusion output layer connected in sequence; the self-attention layer is also skipped to the fusion output layer.

[0020] As a preferred technical solution, the operation process of the prompt-driven feature sensor is as follows:

[0021] Instance-level text features or visual cue representations are input from the input layer along with image patch features;

[0022] Self-attention layers are used to perform self-attention operations on image patch features to obtain self-attention features that contain contextual information.

[0023] By employing a cross-attention layer, using instance-level text features or visual cue representations as query vectors and self-attention features as key and value vectors, cross-attention operations are performed to obtain the corresponding decoupled cue representations.

[0024] As a preferred technical solution, the softmax operation in the self-attention operation adopts the Nyström method.

[0025] As a preferred technical solution, the panoramic representation reconstructor includes a stitching layer, a cross-attention layer, an averaging layer, and a fusion layer;

[0026] The splicing layer is connected to the cross-attention layer and the averaging layer respectively; the cross-attention layer and the averaging layer are connected in parallel to the fusion layer; and a learnable projection layer is connected after the cross-attention layer, the averaging layer and the fusion layer.

[0027] As a preferred technical solution, the process by which the panoramic representation reconstructor obtains slice-level representations is as follows:

[0028] The decoupled representations of textual and visual cues are input into a concatenation layer and concatenated to obtain the decoupled information features of the cues.

[0029] A learnable token is introduced as a query vector and input together with the decoupled information features of the prompts into the cross-attention layer to calculate cross-attention; at the same time, the decoupled information features of the prompts are input into the averaging layer to calculate the feature mean.

[0030] The query vector, cross-attention output, and feature mean are fed together into the fusion layer to obtain a slice-level representation.

[0031] As a preferred technical solution, after obtaining the slice-level representation, the CLIP similarity calculation paradigm is used for category prediction. Specifically, slice-level text features of each category are obtained, and the cosine similarity between the slice-level representation and the slice-level text features of each category is calculated to obtain the prediction probability of each category.

[0032] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0033] 1. Overcoming the modal gap of existing methods' plain text prompts and eliminating subjective bias from manual selection. This invention employs an adaptive enhanced prototype mining strategy, implementing a dual construction strategy of "static mining and dynamic adjustment" to automatically construct statistically representative and dynamically malleable visual cues. This effectively compensates for the deficiency of plain text in accurately describing the microscopic texture of pathology, while avoiding the lack of representativeness caused by existing technologies relying on experts manually selecting samples.

[0034] 2. Possesses good comprehension and interpretability, and can generate accurately located heatmaps. This invention employs a cue-driven feature perceptron, which differs from the passive feature aggregation of existing technologies. This feature perceptron uses multimodal cues as active anchors to accurately decouple key pathological information from complex instance features. This proactive cross-modal alignment mechanism enables the feature perceptron to deeply understand image content and accurately locate the corresponding pathological tissue region based on the cues.

[0035] 3. The obtained slice-level representations possess strong discriminative power, excellent intra-class compactness, and inter-class separability, achieving better classification performance. This invention employs a panoramic representation reconstructor, realizing the deep integration of microscopic visual texture and macroscopic semantics at the panoramic scale, enabling the generated slice-level representations to possess highly structured characteristics in the feature space. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart of a WSI classification method that combines adaptive prototype learning with visual-verbal dual-stream collaboration, as described in an embodiment of the present invention.

[0038] Figure 2 This is a framework diagram of a WSI classification method based on adaptive prototype learning and visual-verbal dual-stream collaboration in an embodiment of the present invention.

[0039] Figure 3 This is a t-SNE clustering diagram of each model in the embodiments of the present invention on TCGA-RCC.

[0040] Figure 4 This is a heatmap obtained by applying the method to three public datasets in an embodiment of the present invention. Detailed Implementation

[0041] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0042] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0043] The third type of method in existing technologies can effectively address the problems of high annotation costs, reliance on large amounts of training data, and poor generalization with small samples found in the first and second types of methods. This invention primarily addresses the bottlenecks in prompt information in VLM prompt fine-tuning methods, such as the deficiencies of pure text prompts, human selection bias, and lack of representativeness of prompts. It proposes an adaptive prototype learning and visual-linguistic dual-modal WSI classification method (PVL). On one hand, it introduces a more efficient and objective visual prompt strategy; an adaptive enhanced prototype mining strategy is used to obtain visual prompts to compensate for the inherent deficiencies of pure text descriptions and the limitations of human selection. On the other hand, it constructs a deep collaborative and complementary mechanism for visual-linguistic dual-modal prompts. A prompt-driven feature perceptron utilizes multimodal prompts to decouple effective information from complex image patch features, while a panoramic representation reconstructor constructs a highly discriminative slice-level representation through deep synergistic reasoning. This invention can effectively guide the model to accurately focus on key pathological regions, significantly improving classification performance and heatmap localization capabilities.

[0044] Please see Figure 1 , 2 One embodiment of this application provides a WSI classification method that combines adaptive prototype learning with visual-verbal dual-stream collaboration, comprising the following steps:

[0045] S1. Segment and cut the WSI tissue from the background into image patches, using the image encoder E. I (·) Extracting image patch features. For a given WSI slice S, the extracted image patch features are represented as: X={x1,x2,…,x Np}, where Np is the number of image blocks in WSI slice S.

[0046] S2. An adaptive and enhanced prototype mining strategy is adopted to form visual cue representations. Specifically, a hierarchical clustering strategy is used to perform two-level clustering of image patch features to obtain static visual prototypes. A set of learnable dynamic prototypes is introduced and spliced ​​with the static visual prototypes to form visual cue representations.

[0047] To construct visual cues that are both visually representative and adaptive, this invention proposes a dual visual cue construction strategy combining static mining and dynamic adjustment, namely, an adaptive enhanced prototype mining strategy. This strategy first significantly reduces the computational bottleneck caused by massive WSI data through hierarchical clustering, mining objective static visual prototypes; then, it combines these with learnable dynamic prototypes for dynamic calibration, forming the final adaptive visual cue.

[0048] Furthermore, to address the excessive computational burden caused by an excessive number of patches, a hierarchical clustering strategy is first employed to perform two-level clustering of image patch features X to obtain a static visual prototype. Specifically:

[0049] Extract all image patch features X from each WSI and perform unsupervised clustering to obtain k local cluster centers C={c1,c2,…,c k This step aims to remove redundant information within a single slice, achieving local feature compression. Next, the k local cluster centers C of each WSI are further clustered to extract m of the most representative global cluster centers, serving as the static visual prototype V for each WSI. static =[v1,v2,…,v m ].

[0050] While static visual prototypes provide prior knowledge of data distribution, hard clustering can suffer from quantization errors and is relatively rigid. Therefore, this invention introduces a set of learnable dynamic prototypes. The final adaptive visual cue is constructed by splicing together a learnable dynamic prototype and a static visual prototype, and can be expressed as:

[0051] ,

[0052] This design endows the model with powerful adaptive adjustment capabilities. During backpropagation training, the learnable prototype can dynamically calibrate the feature space of the static prototype according to the specific classification task requirements. This approach preserves the objective representativeness of the visual prototype while compensating for clustering errors through adaptive learning, thus providing the model with accurate visual guidance that is both robust and flexible.

[0053] S3. Utilize the Large Language Model (LLM) to generate instance-level and slice-level text descriptions, respectively. Place them into templates for word segmentation and embedding, then concatenate the learnable tokens to obtain the corresponding text prompts. Input them into the text encoder to obtain the corresponding text features.

[0054] For text prompts, this invention follows mainstream practices, employing a large language model to generate instance-level and slice-level descriptive text. For example, for invasive ductal carcinoma of the breast (IDC), the question "In a whole-slice image, what are the visual descriptive features of {IDC} at high resolution?" is asked to obtain the slice-level text description: "IDC: visually descriptive characteristics of irregular nests, cords,...". The instance-level text description mainly references TOP (Qu, Linhao, et al. "The rise of AI language pathologists: Exploring two-level prompt learning for few-shot weakly-supervised whole slide image classification." Advances in Neural Information Processing Systems 36 (2023):67551-67564.) to describe various specific cells or tissues, such as columnar epithelium "Columnarepithelium: elongated cells with a basally located...". The category information and the generated descriptive text are then embedded in the following template: "An H&E stained pathological image of..." <cls>, whichexhibits <description>After word segmentation and word embedding, the text prompts [t1, t2, ..., t] are obtained. n Then concatenate the learnable token. The obtained text features are:

[0055] ,

[0056] Represent instance-level text hints as Slice-level text hints are represented as , where N ins and N c Let N represent the number of instance-level text hints and the number of slice-level text hints, respectively. Let L be the sequence length of the descriptive text, and D be the feature dimension. Since each slice category corresponds to one slice-level text hint, N... c It refers to both the number of slice categories and the number of slice-level text prompts.

[0057] Then through the text encoder E T (·) Extract instance-level text features corresponding to the text prompts respectively. and slice-level text features : , .

[0058] Furthermore, this invention employs a pre-trained CLIP model to extract patch features and text features after segmentation, wherein the image encoder E I (·) Using the Vision Transformer in the CLIP model; Text Encoder E T (·) The Transformer in the CLIP model is used; the Chat-GPT is used for the large language model LLM.

[0059] S4. Input instance-level text features and image patch features together into a cue-driven feature perceptron to obtain a decoupled representation of text cue; input visual cue representation and image patch features together into a cue-driven feature perceptron to obtain a decoupled representation of visual cue.

[0060] Furthermore, in order to extract effective information from complex patch features, this invention designs a cue-driven feature perceptron. For example... Figure 2 As shown in (b), the cue-driven feature perceptron includes an input layer, a self-attention layer, a cross-attention layer, and a fusion output layer connected in sequence; the self-attention layer is also skipped to the fusion output layer.

[0061] Furthermore, the working process of the cue-driven feature perceptron is as follows:

[0062] First, instance-level text features or visual cue representations are input into the input layer along with image patch features. Then, a self-attention layer is used to perform self-attention operations on the image patch features to obtain self-attention features containing contextual information. Finally, a cross-attention layer is used to perform cross-attention operations with instance-level text features or visual cue representations as query vectors (Query, Q) and self-attention features as key vectors (Key, K) and value vectors (Value, V) to obtain the corresponding decoupled cue representations.

[0063] Furthermore, to address the long instance sequence problem in WSI, the softmax operation for attention computation in this invention adopts the Nyström method proposed in the paper "Xiong, Yunyang, et al. "Nyströmformer: A nyström-based algorithm for approximating self-attention." Proceedings of the AAAIconference on artificial intelligence. Vol. 35. No. 16. 2021." to reduce computational complexity.

[0064] To more specifically illustrate the working process of the cue-driven feature perceptron of the present invention, this embodiment takes visual cue guidance feature parsing as an example, and represents the patch feature as follows: Visual cue representation is as follows Let SelfAttention(·) denote the self-attention operation, CrossAttention(·,·) denote the cross-attention operation, and GAP(·) denote global average pooling. (·, ·) represents concatenating the two sequences along the dimension of sequence number. The specific processing can be represented as follows:

[0065] A = SelfAttention(X),

[0066] ,

[0067] Where A is the output of the self-attention layer in the visual cue-driven feature perceptron, and W... o W q W k W v It is a learnable projection layer parameter matrix, initialized to the identity matrix. To prevent patch features and visual cue representations from being corrupted in the initial stage, N p and N v These represent the number of patch features and the number of visual cue representations, respectively. D is the feature dimension, and V is the feature dimension. p For visual cue representation, T represents the matrix transpose operation, and d k Scaling factor This represents a decoupled visual cues. Similarly, for instance-level text features... This mechanism is also used for feature decoupling to extract the representation of text prompt parsing. :

[0068] B = SelfAttention(X),

[0069] ,

[0070] Where B is the output of the self-attention layer in the text-cue-driven feature perceptron.

[0071] S5. Use the panoramic representation reconstructor to fuse the representations of text cues decoupled from visual cues to obtain slice-level representations.

[0072] To obtain highly discriminative slice-level representations for WSI classification, this invention proposes an efficient panoramic representation reconstructor. For example... Figure 2 As shown in (c), the panoramic representation reconstructor includes a stitching layer, a cross-attention layer, a mean layer, and a fusion layer. The stitching layer is connected to the cross-attention layer and the mean layer, respectively. The cross-attention layer and the mean layer are connected in parallel to the fusion layer. Each of the cross-attention layer, the mean layer, and the fusion layer is connected to a learnable projection layer (FC1, FC2, FC3).

[0073] Furthermore, the process by which the panoramic representation reconstructor obtains slice-level representations is as follows:

[0074] First, decouple the representation of the text prompt. Representation decoupled from visual cues The input splicing layer is used to splice the information to obtain the decoupling information feature X. f Then, a learnable token is introduced. X, as an information feature decoupling query vector and prompts f The cross-attention layer is used to compute cross-attention; simultaneously, the decoupled information feature X from the cue is input together. f The input averaging layer calculates the feature mean F; finally, the query vector z, the cross-attention output E, and the feature mean F are fed together into the fusion layer to obtain the slice-level representation. The specific process can be represented as follows:

[0075] ,

[0076] E=W c CrossAttention(z, X) f ),

[0077] F=W d ·Mean(X f ),

[0078] x=W h ·(E+F+z),

[0079] Among them W c W d and W h These are the parameter matrices of the learnable projection layers FC1, FC2, and FC3, respectively. Mean(·) represents the averaging operation, used to calculate the mean of multiple vectors.

[0080] S6. Calculate the similarity between slice-level representation and slice-level text prompt representation to obtain category prediction results.

[0081] Furthermore, after obtaining the slice-level representation x, the CLIP similarity calculation paradigm is used for classification. Specifically, let the slice-level text features of category c be represented as ( Next, the cosine similarity between x and the slice-level text features of each category is calculated to obtain the predicted probability of category c. The predicted probability of category c can be expressed as:

[0082] ,

[0083] in, To predict the category, τ is the temperature coefficient of CLIP learning, cos(·,·) is the cosine similarity function, and K is the total number of categories in this classification task.

[0084] Finally, standard cross-entropy loss is used to train all parameters in this method, including the learnable dynamic prototype, all learnable vectors, cue-driven feature perceptron, panoramic representation reconstructor, etc., which can be defined as L=CE(p,y), where p is the predicted probability and y is the corresponding true label.

[0085] To evaluate this method, this embodiment used three publicly available whole-slice image datasets: TCGA-RCC (renal carcinoma, class 3), TCGA-LUNG (lung cancer, class 2), and TCGA-BRCA (breast cancer, class 2). Following a small-sample experimental setup, 16 samples were randomly selected from each class in the training set for training, and the mean ± standard deviation of AUC, F1 score, and accuracy ACC for five independent trials are reported. This experiment compared several different methods, including a series of multi-instance learning methods such as Max-pooling, Mean-pooling, ABMIL (Ilse, Maximilian, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. International conference on machine learning. PMLR, 2018.), CLAM (Lu, Ming Y., et al. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering 5.6 (2021): 555-570.), TransMIL (Shao, Zhuchen, et al. Transmil: Transformer based correlated multiple instance learning for wholeslide image classification. Advances in neural information processing systems 34 (2021): 2136-2147.), and TOP (Qu, Linhao, et al. The rise of AI language pathologists: Exploring two-level prompt learning for few-shot weakly-supervised whole slide image). classification."Advances in Neural Information Processing Systems 36 (2023): 67551-67564.The proposed method, ViLa-MIL (Shi, Jiangbo, et al. "Vila-mil: Dual-scale vision-language multipleinstance learning for whole slide image classification." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024.), includes patches at 5X and 10X scales to meet the requirements of ViLa-MIL. Furthermore, since this method performs PVL at a single scale, the performance of ViLa-MIL High at a single high scale was also compared separately. Additionally, the performance of PVL without a visual prototype (without a prototype) was compared, and the results are shown in Table 1.

[0086] Table 1 Performance results of each method

[0087]

[0088] Table 1 shows the best results in bold and the second-best results underlined to visually reflect the experimental results. The data in Table 1 shows that the Visual-Language Model (VLM) based method generally outperforms the traditional MIL method. This is mainly because the traditional MIL method often struggles to learn discriminative visual features with limited data in small sample settings, while VLM effectively overcomes this limitation by utilizing semantic prior knowledge provided by text cues. Building on this, the PVL framework of this invention further improves this paradigm. PVL not only utilizes text cues but also mines representative visual prototypes as powerful visual cues. This "visual-language" multimodal cue guidance mechanism endows the model with a more comprehensive and deeper understanding of the data. Furthermore, by comparing the complete PVL model with the variant PVL that removes visual cues… w / o prototype The performance of the complete model is clearly superior, which strongly demonstrates the effectiveness of the visual cueing mechanism introduced in this invention in improving model performance.

[0089] Regarding model visualization, this embodiment of the invention uses t-SNE dimensionality reduction to visualize the slice-level feature clustering results on the TCGA-RCC test set, and compares them with several other methods. For example... Figure 3 As shown, compared with other methods, the feature space of the PVL method of the present invention is more structured, exhibiting better intra-class compactness and inter-class separability, which visually confirms that the method of the present invention can effectively learn and distinguish features.

[0090] Furthermore, embodiments of the present invention also visualize heatmaps for models under different prompts, such as... Figure 4 As shown in the image, samples were selected from three datasets for visualization. Red represents regions that the model pays more attention to, while blue represents regions that receive less attention. Figure 4 In the middle (a), Ground Truth represents the true label; the area marked in green is the tumor area. Figure 4 (b) Tumor represents the model's attention level when the input prompt is a tumor-related description. Figure 4 In the middle (c) Normal, the model focuses on the region when the input prompt is a description of normal tissue. It can be seen that when the prompt is a description of tumor, the model highlights the tumor region in red across the three dataset slices, indicating high attention and demonstrating the model's ability to effectively locate key regions in the image. Conversely, when the prompt is a description of normal tissue, the model highlights non-tumor regions in red and tumor regions in blue, demonstrating that the model, guided by the prompt text, effectively understands the difference between normal and lesion regions in the image, achieving efficient localization.

[0091] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0092] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.< / description> < / cls>

Claims

1. A WSI classification method that combines adaptive prototype learning with visual-verbal dual-stream collaboration, characterized in that, Includes the following steps: The WSI tissue is segmented from the background and cut into image patches, and image patch features are extracted using an image encoder; An adaptive and enhanced prototype mining strategy is adopted to form visual cue representations. Specifically, a hierarchical clustering strategy is used to perform two-level clustering of image patch features to obtain static visual prototypes; a set of learnable dynamic prototypes is introduced and spliced ​​with the static visual prototypes to form visual cue representations. Instance-level and slice-level text descriptions are generated using a large language model. These descriptions are then put into templates for word segmentation and embedding, and then concatenated to learnable tokens to obtain corresponding text prompts. Finally, these tokens are input into a text encoder to obtain the corresponding text features. Instance-level text features and image patch features are input together into a cue-driven feature perceptron to obtain a decoupled representation of text cue; visual cue representation and image patch features are input together into a cue-driven feature perceptron to obtain a decoupled representation of visual cue. The panoramic representation reconstructor is used to fuse the representations of text cues decoupled from visual cues to obtain slice-level representations. Calculate the similarity between slice-level representations and slice-level text features to obtain category prediction results.

2. The WSI classification method based on adaptive prototype learning and visual-verbal dual-stream collaboration according to claim 1, characterized in that, The specific steps for obtaining a static visual prototype by performing two-level clustering of image patch features using a hierarchical clustering strategy are as follows: Extract all image patch features from each WSI and perform unsupervised clustering to obtain the features of each WSI. k Local cluster centers; For each WSI k Further clustering of local cluster centers m The most representative global cluster center serves as the static visual prototype for each WSI.

3. The WSI classification method based on adaptive prototype learning and visual-verbal dual-stream collaboration according to claim 1, characterized in that, The image encoder uses the Vision Transformer in the CLIP model; the large language model uses Chat-GPT; and the text encoder uses the Transformer in the CLIP model.

4. The WSI classification method based on adaptive prototype learning and visual-verbal dual-stream collaboration according to claim 1, characterized in that, During the embedding process, category information is embedded into the corresponding text descriptions; after embedding, a learnable token is concatenated to obtain the corresponding text prompts.

5. The WSI classification method based on adaptive prototype learning and visual-verbal dual-stream collaboration according to claim 1, characterized in that, The cue-driven feature perceptron includes an input layer, a self-attention layer, a cross-attention layer, and a fusion output layer connected in sequence; the self-attention layer is also skipped to the fusion output layer.

6. The WSI classification method based on adaptive prototype learning and visual-verbal dual-stream collaboration according to claim 5, characterized in that, The operation of the cue-driven feature perceptron is as follows: Instance-level text features or visual cue representations are input from the input layer along with image patch features; Self-attention layers are used to perform self-attention operations on image patch features to obtain self-attention features that contain contextual information. By using a cross-attention layer, instance-level text features or visual cue representations are used as query vectors, and self-attention features are used as key and value vectors to perform cross-attention operations to obtain the corresponding decoupled cue representations.

7. The WSI classification method based on adaptive prototype learning and visual-verbal dual-stream collaboration according to claim 5, characterized in that, The softmax operation in the self-attention operation uses the Nyström method.

8. The WSI classification method based on adaptive prototype learning and visual-verbal dual-stream collaboration according to claim 1, characterized in that, The panoramic representation reconstructor includes a stitching layer, a cross-attention layer, an averaging layer, and a fusion layer; The splicing layer is connected to the cross-attention layer and the averaging layer respectively; the cross-attention layer and the averaging layer are connected in parallel to the fusion layer; and a learnable projection layer is connected after the cross-attention layer, the averaging layer and the fusion layer.

9. The WSI classification method based on adaptive prototype learning and visual-verbal dual-stream collaboration according to claim 8, characterized in that, The process by which the panoramic representation reconstructor obtains slice-level representations is as follows: The decoupled representations of textual and visual cues are input into a concatenation layer and concatenated to obtain the decoupled information features of the cues. A learnable token is introduced as a query vector and input together with the decoupled information features of the prompts into the cross-attention layer to calculate cross-attention; at the same time, the decoupled information features of the prompts are input into the averaging layer to calculate the feature mean. The query vector, cross-attention output, and feature mean are fed together into the fusion layer to obtain a slice-level representation.

10. The WSI classification method based on adaptive prototype learning and visual-verbal dual-stream collaboration according to claim 1, characterized in that, After obtaining the slice-level representations, the CLIP similarity calculation paradigm is used for category prediction. Specifically, slice-level text features of each category are obtained, and the cosine similarity between the slice-level representations and the slice-level text features of each category is calculated to obtain the prediction probability of each category.

Images