Small sample full-slice pathological image classification method based on cross-modal double-branch decoding
By constructing a cross-modal dual-branch decoding network and combining local instances and global semantic cues, the limitations of visual-language models in understanding complex pathological features in small-sample pathological classification are addressed, achieving efficient pathological image classification and improving classification accuracy and feature capture capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-10
- Publication Date
- 2026-03-10
AI Technical Summary
Existing small-sample pathological classification methods based on vision-language models have limitations in understanding the complex relationship between the fine-grained features unique to pathology and macroscopic diagnostic criteria. They also lack cross-modal bidirectional interaction, resulting in insufficient ability of the model to capture complex pathological features and insufficient accuracy in discrimination under small-sample conditions.
A cross-modal dual-branch decoding network is constructed, including an image patch decoding branch and a text decoding branch. Hierarchical learning is performed through local instance cues and global semantic cues. Combined with a frozen backbone network and a few-sample parameter efficient adaptation strategy, a two-way interaction between text-guided visual aggregation and visually enhanced text semantics is achieved.
It significantly improves the classification accuracy of pathological images in small sample scenarios, solves the overfitting problem, and achieves effective capture and discrimination of complex pathological features.
Smart Images
Figure CN121640460A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer-aided medical diagnosis and deep learning technology, specifically to a small-sample whole-slice pathological image classification method based on cross-modal bi-branch decoding. Background Technology
[0002] In recent years, with the development of digital pathological slide technology, automated analysis based on whole slide images (WSI) has become an important means of assisting cancer diagnosis. Traditional methods based on multiple instance learning (MIL) usually require a large amount of labeled training data to achieve good performance. However, in the field of pathology, obtaining high-quality pixel-level or slide-level annotations is often costly and time-consuming, especially when dealing with rare pathological types, where the problem of data scarcity is particularly prominent.
[0003] To address the classification challenges in small-sample scenarios, visual-language pre-trained models (such as CLIP) have been introduced into pathological image analysis due to their strong generalization capabilities. These methods attempt to leverage the semantic information of the text modality to compensate for the limitations of visual samples. However, existing small-sample pathological classification methods based on visual-language models still have significant limitations. On the one hand, general visual-language models struggle to directly understand the complex relationship between pathologically specific fine-grained features (such as nuclear atypia) and macroscopic diagnostic criteria, and simple fixed text modalities (such as "a picture of a tumor") cannot provide effective diagnostic guidance. On the other hand, existing fine-tuning methods typically employ unidirectional feature transfer strategies, i.e., using only text to guide visual feature extraction or only using visual features to align with text, lacking true cross-modal bidirectional interaction. This "semi-modal" approach prevents the model from accurately injecting medical prior knowledge into the selection process of visual features, and also prevents text features from perceiving specific image instance evidence, thus limiting the model's ability to capture and accurately distinguish complex pathological features under small-sample conditions. Summary of the Invention
[0004] To address the problems existing in the prior art, this invention discloses a small-sample whole-slice pathological image classification method based on cross-modal dual-branch decoding, specifically including the following steps: A small-sample whole-slice pathological image classification task is constructed. The whole-slice pathological images and their corresponding pathological diagnosis labels are obtained to construct a pathological image dataset. The pathological image dataset is divided into a training set containing a small number of labeled samples and a test set to be predicted. The images are then tiled to generate an image patch sequence. We construct a feature extractor for a visual language pre-trained model based on frozen parameters and design a hierarchical cue learning module, which is used to capture local instance cues of microscopic cellular features and global semantic cues to characterize macroscopic diagnostic criteria. Construct a cross-modal dual-branch decoding network, which includes an image patch decoding branch and a text decoding branch. The image patch decoding branch utilizes local instance cues, filters and aggregates image patch features through a cross-modal attention mechanism, and generates slice-level visual features; the text decoding branch uses image patch features as contextual information, injects visual evidence into global semantic cues through a reverse cross-modal attention mechanism, and generates visually perceptual text features. A cross-modal bi-branch decoding network model is trained based on a small-sample parameter efficient adaptation strategy. During training, the backbone parameters of the pre-trained model are kept frozen, and the cue vectors in the hierarchical cue learning module and the adapter parameters in the cross-modal bi-branch decoding network are iteratively updated using only the training set data. On the test set, the trained cross-modal bi-branch decoding network is used to extract slice-level visual features and visual perception text features, respectively, and the whole-slice pathological images are classified based on the similarity between the two.
[0005] The cross-modal dual-branch decoding network classifies text by calculating the matching degree between slice-level visual features and visually perceived text features in a shared semantic space. The calculation process of the matching degree is expressed as follows:
[0006] in: This represents the input whole-slice pathological image, where C represents the total number of categories. Represents cosine similarity. This represents the slice-level visual features output from the text-guided image patch decoding branch. This represents the visually perceived text feature corresponding to the i-th class, output by the image-enhanced text decoding branch.
[0007] When constructing the image patch decoding branch, a fine-grained semantic association is established between local instance cues and image patch features: Local instance prompts are input into a text encoder to obtain local text features. The image patch sequence is input into a visual encoder to obtain image patch features. ; Calculate image patch features With local text features The dot product similarity between them is used to obtain the aggregation weight through Softmax normalization. The formula is as follows:
[0008] Based on aggregate weight Weighted aggregation of image patch features yields slice-level visual features. The formula is as follows:
[0009] Among them, aggregate weight This indicates the degree of activation of key features by local cues, thereby capturing key features with limited samples.
[0010] When constructing the text decoding branch, visual features are embedded into the text semantic space, specifically in the following manner: The global semantic cues are input into the text encoder to obtain global text features. And map it to a shared semantic space to obtain At the same time, the instance features are mapped to obtain ; Feature fusion is performed through stacked text-guided cross-attention layers, where global text features... as query vector Image patch features as a key vector Sum value vector The calculation is as follows:
[0011]
[0012]
[0013] Finally, the intensity of visual information fusion is controlled by a multilayer perceptron, and visually perceived text features are output. :
[0014] in, To output the projection matrix, As a scaling parameter, the text decoding branch ensures that the generated text features contain concrete visual instance evidence.
[0015] The hierarchical prompting learning module includes local instance prompts. and global semantic hints Local instance hints From learnable context vectors Standardized organizational phenotypic names And microscopic visual descriptions of cell nucleus morphology generated by large language models. Concatenated; Global semantic hints From learnable category context vectors Standardized diagnostic description templates And macroscopic pathological descriptions of tissue structures generated by large language models It was pieced together.
[0016] By adopting the above technical solutions, this invention provides a small-sample whole-slice pathological image classification method based on cross-modal bi-branch decoding. By constructing a cross-modal bi-branch decoding network, it breaks through the limitations of unidirectional feature transfer in existing technologies and realizes bidirectional deep interaction between text-guided visual aggregation and visually enhanced text semantics. At the same time, by combining the strategy of freezing the backbone network and hierarchical prompting learning, it effectively solves the overfitting problem and significantly improves the classification accuracy of pathological images in small-sample scenarios that require only a very small number of labeled samples. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 The flowchart of the method of the present invention Figure 2 This is a diagram showing the overall structure of the cross-modal dual-branch decoding network in this invention. Detailed Implementation To make the technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention: like Figure 1 The method for classifying small-sample whole-slice pathological images based on cross-modal dual-branch decoding, as shown, specifically includes the following steps: S1: Obtain whole-slice pathological images and their corresponding pathological diagnostic labels, construct a pathological image dataset, divide the pathological image dataset into a training set containing a small number of labeled samples and a test set to be predicted, and perform tiling processing on the images to generate image patch sequences. S11: Obtain public histology datasets or clinical private datasets as raw data, including whole-slice pathology images (WSI) and their corresponding diagnostic labels (such as "normal" or "metastasis"); at the same time, obtain medical text descriptions related to the diagnosis (such as pathological diagnostic criteria, cell morphology characteristics, etc.).
[0019] S12: For the few-shot learning task, the dataset is divided into a training set and a test set. In the K-shot setting, K whole-slice pathological images (e.g., 1, 2, 4, 8, 16) are randomly selected from each category as the training set for training, and the remaining samples are used as the test set for evaluation.
[0020] S13: Preprocess the above whole-section pathological images. Due to the extremely high resolution of WSI, the background area needs to be removed at a specific magnification (such as 5x or 20x), and the foreground area containing tissue is tiled and cropped into a non-overlapping image block sequence.
[0021] S14: The size of the cropped image blocks is uniformly set to 224×224 pixels, and each whole-slice pathological image is converted into a sequence containing multiple image blocks.
[0022] like Figure 2 As shown: S2: Use a visual language pre-trained model (such as CLIP) as a feature extractor, and extend its functionality while freezing its backbone parameters to generate multimodal feature representations for subsequent decoding; S21: The extension includes a frozen image encoder. Frozen text encoder Local instance suggestion module and global semantic suggestion module.
[0023] S22: Hint for building a local instance : Utilize large language models (such as GPT-4) to generate visual descriptions of microscopic features such as cell nuclear morphology and chromatin distribution. Combined with learnable context vectors and standardized organizational phenotypic names Hints for creating partial examples by piecing together: Enter it into the text encoder. Obtain local text features .
[0024] S23: Constructing Global Semantic Hints Generate macroscopic pathological descriptions of tissue structure and disease diagnostic criteria using large language models. Combined with learnable category context vectors and standardized diagnostic description templates The concatenation of these elements forms a global semantic prompt: Enter it into the text encoder. Obtain global text features .
[0025] S24: Input the image patch sequence from step S14 into the visual encoder Image patch features were extracted. .
[0026] like Figure 2 As shown: S3: Construct a cross-modal dual-branch decoding network, which includes an image patch decoding branch and a text decoding branch, and achieves feature alignment through bidirectional interaction; S31: Image patch decoding branch: designed to capture key features using prior knowledge of local pathology.
[0027] First, calculate the image patch features. With local text features The dot product similarity between the images measures the degree of matching between each image patch and the microscopic pathological description; the similarity is normalized using the Softmax function to generate the aggregation weight. ; Utilizing weights Weighted aggregation of image patch features yields slice-level visual features. This allows us to focus on key features.
[0028] S32: Text Decoding Branch: Designed to inject specific visual evidence into global text features.
[0029] global text features and image patch features Mapped to a shared semantic space respectively, we obtain and A text-guided cross-attention layer is introduced to utilize global text features. As the query vector (Q), with image patch features As the key vector (K) and value vector (V), the calculation is as follows:
[0030] By controlling the intensity of visual information fusion through a multilayer perceptron, visual perceptual text features are output. .
[0031] S33: Classification Prediction: Calculating Slice-Level Visual Features Visually perceived text features The degree of match between them. Calculated as follows:
[0032] S4: The cross-modal bi-branch decoding network is iteratively trained using whole-slice pathological image data from the training set to update the network parameters. This training process includes forward propagation of network features and backward propagation of errors. S41: During implementation, ResNet50 is used as the visual backbone of the CLIP model, and all pre-trained weights of CLIP are frozen. Training is performed only on a small number of learnable context vectors from local instance cues and global semantic cues, as well as the parameters in the cross-modal dual-branch decoding network.
[0033] S42: Loss Function: In addition to the main classification loss, a minimum correlation constraint loss is introduced during training. This encourages different text prompts to focus on different visual patterns in the image, thus avoiding feature homogenization.
[0034] S43: Parameter settings: On the Camelyon16 dataset, set the learning rate to 0.002, the training period to 500 epochs, and the batch size to 1.
[0035] S5: Update the network parameters in each iteration, perform training and validation in each iteration, save the optimal weights of the cross-modal dual-branch decoding network based on the optimal validation set accuracy, initialize the network with the optimal weights, and evaluate the performance of the network on the test set.
[0036] During implementation, tests were conducted using small sample settings of 1-shot, 2-shot, 4-shot, 8-shot, and 16-shot. AUC (Area Under the ROC Curve) was used as the evaluation metric. Experimental results show that, in the 16-shot setting, our method achieves an AUC of 76.20% on the Camelyon16 dataset, significantly outperforming existing multi-instance learning methods.
[0037] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A small sample whole-slide pathological image classification method based on cross-modal double-branch decoding, characterized by The application relates to a small sample whole slice pathological image classification method based on a cross-modal dual-branch decoding network. The application comprises the following steps: A feature extractor based on a visual language pre-training model with frozen parameters is constructed, and a hierarchical prompt learning module is designed, which is used to capture local instance prompts of microscopic cell features and global semantic prompts for macroscopic diagnostic criteria. A cross-modal dual-branch decoding network is constructed, which comprises an image block decoding branch and a text decoding branch. The image block decoding branch uses local instance prompts to filter and aggregate image block features through a cross-modal attention mechanism to generate slice-level visual features. The text decoding branch uses image block features as context information to inject visual evidence into global semantic prompts through a reverse cross-modal attention mechanism to generate visual perception text features. The cross-modal dual-branch decoding network model is trained based on a small sample parameter efficient adaptation strategy.
2. The small sample whole-slide pathological image classification method based on cross-modal dual-branch decoding according to claim 1, characterized in that: During the training process, the pre-training model backbone parameters are kept frozen, and only the prompt vectors in the hierarchical prompt learning module and the adapter parameters in the cross-modal dual-branch decoding network are iteratively updated using the training set data. wherein: represents an input whole-slide pathology image, C represents a total number of classes, represents a cosine similarity, represents a slice-level visual feature output by a text-guided image block decoding branch, represents a corresponding i-th class of visual perception text feature output by an image-enhanced text decoding branch.
3. The small sample whole-slide pathological image classification method based on cross-modal dual-branch decoding according to claim 1, characterized in that: On the test set, the trained cross-modal dual-branch decoding network is used to extract slice-level visual features and visual perception text features, and the whole slice pathological image is classified based on the similarity between the two features. inputting the local instance prompt into a text encoder to obtain local text features inputting the image block sequence into a visual encoder to obtain image block features ; Computing image patch features Dot product similarity between local text features and normalized by Softmax to get aggregated weights , as follows: Aggregating weights based on Weighted aggregation of image block features to obtain slice-level visual features The formula is as follows: where the aggregated weights represent the degree of activation of the local cues to the key features, thereby capturing the key features under limited samples.
4. The small sample whole-slide pathological image classification method based on cross-modal dual-branch decoding according to claim 1, characterized in that: The cross-modal dual-branch decoding network classifies by calculating the matching degree of slice-level visual features and visual perception text features in a shared semantic space. When constructing the image block decoding branch, a fine-grained semantic association between local instance prompts and image block features is established. When constructing the text decoding branch, visual features are embedded into a text semantic space in the following way. input the global semantic prompt into the text encoder to obtain global text features and map them to a shared semantic space to obtain while mapping the instance features to obtain ; Feature fusion is performed by a cross-attention layer guided by the stacked text, where the global text features are used as query vectors , the image patch features are used as key vectors , and the value vectors are computed as follows: Finally, the visual information fusion strength is controlled by a multi-layer perception layer, and the visual perception text features are output : wherein, is an output projection matrix, is a scaling parameter, the text decoding branch ensures that the generated text features contain concrete visual instance evidence.
5. The small sample whole-slide pathological image classification method based on cross-modal dual-branch decoding according to claim 1, characterized in that: The hierarchical prompt learning module includes local instance prompts and global semantic prompts wherein the local instance prompts are concatenated from learnable context vectors , standardized organizational phenotype names , and microscopic visual descriptions of nuclear morphology generated by a large language model ; and the global semantic prompts are concatenated from learnable class context vectors , standardized diagnostic description templates , and macroscopic pathological descriptions of tissue architecture generated by a large language model .