Training method and system for small-amount multi-modal pathological annotation data mixed prompt learning framework

Abstract

The invention discloses a training method and system for a small amount of multi-modal pathological annotation data mixed prompt learning framework. The training method comprises the steps of obtaining an original CLIP model, an initial text prompt and a training image corresponding to the initial text prompt, wherein the original CLIP model comprises an image encoder and a text encoder; inserting a learnable visual prompt mark in the image encoder, and inserting a learnable text prompt mark in the text encoder; inputting the training image into an image encoder and combining the training image with a learnable visual prompt mark for feature coding to obtain visual prompt embedding; inputting the initial text prompt into a text encoder and combining the initial text prompt with a learnable text prompt mark for feature coding to obtain text prompt embedding; on the basis of the similarity of visual prompt embedding and text prompt embedding, parameters of learnable visual prompt marks, learnable text prompt marks and coupling functions are adjusted, a mixed prompt learning framework is obtained, and the multi-modal combination capacity of the model is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and in particular to a training method and system for a hybrid prompt learning framework for a small amount of multimodal pathology annotation data. Background Art

[0002] CLIP (Contrastive Language-Image Pretraining) is a technique that performs contrastive learning pretraining on large-scale image-text pair data, effectively capturing the semantic associations between images and text. By mapping images and text into the same feature space, CLIP places semantically similar images and texts close together in this space, enabling powerful cross-modal understanding capabilities. Specifically, CLIP consists of a text encoder and an image encoder. The text encoder, typically based on a Transformer architecture such as BERT, converts the input text into a fixed-dimensional vector; the image encoder, typically using a deep convolutional neural network (CNN) such as ResNet or Vision Transformer (ViT), converts the image into a vector. By jointly training these two encoders, CLIP is able to project images and text into a shared vector space and, using a contrastive loss, maximize the similarity between images and corresponding text while minimizing their similarity to unrelated text.

[0003] This pre-training process enables CLIP to effectively capture the semantic similarities and differences between images and text, thereby supporting zero-shot learning tasks. For example, CLIP can classify images based on text descriptions or retrieve corresponding text descriptions from images without specific training. Furthermore, CLIP is capable of handling complex cross-modal tasks such as image and text retrieval, generation, and restoration. In some cases, it can even be combined with generative models such as DALLE to achieve text-driven image generation. CLIP's advantage lies in its ability to efficiently leverage large amounts of unlabeled data available on the internet, demonstrating strong generalization capabilities and maintaining excellent performance across a variety of unseen data and tasks. Therefore, CLIP, as a foundational model, can play an important role in many tasks and, after fine-tuning for specific tasks, demonstrates strong zero-shot transfer capabilities, achieving excellent results in downstream tasks. This makes it widely applicable to tasks such as image classification, image retrieval, and image captioning, without the need for extensive task-specific training.

[0004] Prompt tuning is a model tuning method. Prompts can be manually created in the form of natural language instructions or automatically generated through training as natural language instructions or vector representations. Unlike traditional full fine-tuning methods, prompt tuning efficiently fine-tunes the model by adding a small number of trainable parameters (task-specific prompts), adapting the model to new tasks. This improves the performance of large multimodal models such as CLIP on a variety of downstream tasks. Compared to full fine-tuning, prompt tuning requires less manually annotated data and computing resources, shortens training time, and is more cost-effective while maintaining performance. It also retains the knowledge of the original pre-trained base model, resulting in stronger generalization. This method has achieved remarkable results in fields such as natural language processing (NLP), computer vision (CV), and multimodal model transfer research. For example, by adding learnable prompts to the CLIP cross-modal alignment model and then efficiently fine-tuning it, it can demonstrate strong performance on a variety of downstream tasks or new data distributions, with reasonable generalization and versatility.

[0005] In the field of medical image classification, for disease detection with a small amount of multimodal pathology annotated data, the CLIP model is generally used in combination with prompt word fine-tuning technology, such as CoOp, CoCoOp, VPT, etc. In the process of implementing the existing technology, the inventors found that:

[0006] (1) Limited domain generalization ability: Traditional cue word fine-tuning techniques, such as CoOp and CoCoOp, have limited generalization ability for categories that have not appeared in the training set in data-scarce scenarios, which makes the model prone to overfitting to the category characteristics of the training data. Even VPT, which enhances the generalization ability of the model by adding learnable cues to the visual input, may still find it difficult to fully capture the fine-grained features in pathological images, which is particularly disadvantageous for complex lesion features in medical images. (2) Limitations of few-shot learning: In the medical field, labeled data is often scarce and expensive to obtain. Existing cue word fine-tuning methods may not be able to fully learn effective features when faced with few-shot situations, resulting in a decline in model performance. Although VPT, like other methods, adjusts visual features by adding learnable cues (i.e., visual cues) to the visual input without fine-tuning the parameters of the entire pre-trained model, thereby significantly reducing the number of parameters that need to be optimized, its performance in few-shot learning scenarios is still insufficient. When there are only a small number of labeled samples (e.g., only 1-16 images per category), the performance is significantly reduced. (3) Insufficient multimodal integration capabilities: Most current methods focus on optimization in the visual field and lack the effective use of specific textual knowledge in the medical field. Pathological images are often highly complex and uncertain. Without the integration of domain-specific textual knowledge, the robustness and accuracy of classification tasks are significantly reduced. Summary of the Invention

[0007] The main purpose of the present invention is to provide a training method and system for a hybrid prompt learning framework for a small amount of multimodal pathology annotation data, thereby overcoming the shortcomings of the existing technology.

[0008] To achieve the aforementioned object of the invention, the technical solutions adopted by the present invention include:

[0009] A first aspect of the present invention provides a training method for a hybrid prompt learning framework for a small amount of multimodal pathology annotated data, comprising the following steps:

[0010] Obtaining an original CLIP model, an initial text prompt, and a training image corresponding to the initial text prompt, wherein the original CLIP model includes an image encoder and a text encoder;

[0011] Inserting a learnable visual cue marker into the image encoder and inserting a learnable text cue marker into the text encoder, wherein the learnable visual cue marker is generated by mapping the learnable text cue marker through a coupling function;

[0012] Inputting the training image into an image encoder and combining it with the learnable visual cue marker to perform feature encoding to obtain a visual cue embedding; inputting the initial text cue into a text encoder and combining it with the learnable text cue marker to perform feature encoding to obtain a text cue embedding;

[0013] Based on the similarity between the visual cue embedding and the textual cue embedding, the learnable visual cue marker, the learnable textual cue marker and the parameters of the coupling function are adjusted to obtain a hybrid cue learning framework.

[0014] In some more specific solutions, during the training process, each feature layer of the original CLIP model is frozen.

[0015] Some of the more specific plans include:

[0016] Dividing the training image into blocks and linearly projecting the blocks to obtain multiple image block labels;

[0017] splicing the image category label, the learnable visual cue label, and multiple image block labels to form a label sequence, wherein the image category label is obtained by initializing the original CLIP model;

[0018] Inputting the tag sequence into the image encoder to obtain a visual cue embedding;

[0019] Perform word embedding on the initial text prompt to obtain a text category tag;

[0020] Concatenate the learnable text prompt tag and the text category tag to form a text embedding vector;

[0021] Inputting the text embedding vector into the text encoder to obtain a text prompt embedding;

[0022] Performing similarity calculation on the visual cue embedding and the text cue embedding, and outputting a similarity matrix;

[0023] The similarity in the similarity matrix is ​​maximized, the learnable visual cue marker, the learnable text cue marker, and the parameters of the coupling function are updated to obtain a hybrid cue learning framework.

[0024] In some more specific solutions, any feature layer of the image encoder includes a first normalization layer, an attention layer, a second normalization layer, and an MLP neural network, and the tag sequence is input into the image encoder to obtain a visual cue embedding, specifically including:

[0025] Inputting the labeled sequence into the first normalization layer to perform normalization on the labeled sequence;

[0026] Inputting the normalized tag sequence into the attention layer, using the learnable visual cue tag as a query vector, and outputting fine-grained feature association information;

[0027] Adding the fine-grained feature association information to the coarse-grained spatial information of the label sequence to obtain a fused feature;

[0028] Inputting the fused features into the second normalization layer to perform normalization on the fused features;

[0029] Input the standardized fusion features into the MLP neural network, perform nonlinear mapping and dimension expansion, and output the expanded features;

[0030] Adding the extended feature to the fused feature to form a composite feature;

[0031] Projecting the composite features to obtain a visual cue embedding;

[0032] Preferably, the formula for the learnable visual cue marker is: v ={p1, p2, ..., p m}, m represents the length of the learnable visual cue token,

[0033] Preferably, the formula for embedding the visual cue is: v ′=f v ([P v ;x v ]), x vrepresents the image block label, [P v ;x v ] represents the concatenation of learnable visual cue labels and image patch labels,

[0034] Preferably, the formula for the learnable text prompt mark is: P t ={p1, p2, ..., p n}, n represents the length of the learnable text prompt token,

[0035] Preferably, the formula for embedding the text prompt is: t ′=f t ([P t ;z t ]), z t Indicates the text category label, [P t ;z t ] represents the concatenation of the learnable text prompt tag and the text category tag,

[0036] Preferably, the marker sequence is standardized by calculating the mean and variance;

[0037] Preferably, the fused features are input into the second normalization layer, and the fused features are normalized using the normalization parameters of the first normalization layer.

[0038] Furthermore, the method of using the learnable visual cue marker as a query vector to output fine-grained feature association information specifically includes:

[0039] Selecting the learnable visual cue marker as a query vector;

[0040] Extracting image features of each image block mark;

[0041] Calculating a degree of match between the learnable visual cue marker and the image features of each image block marker;

[0042] According to the matching degree, each image block is marked with a corresponding weight;

[0043] Based on the weight difference, fine-grained feature association information is output.

[0044] In some more specific solutions, the text encoder includes multiple encoding layers, and the text embedding vector is input into the text encoder to obtain the text prompt embedding, including:

[0045] The learnable text prompt marker and text category marker are sequentially input into each encoding layer for training; during the training process, the learnable text prompt marker is in an active state for optimization and adjustment, and the text category marker is in a frozen state; through the multi-layer encoding of the text encoder, the deep features of the text are gradually extracted to determine the text prompt embedding.

[0046] Some of the more specific plans include:

[0047] Through the coupling function F c The learnable text prompt mark is converted into a learnable visual prompt mark, and the coupling function F c It includes a linear transformation network MLP for labeling the learnable text prompt P t Perform projection to generate a label P with the learnable visual cue v The visual cue mark P in the same space v ', establish the learnable visual cue marker P v With learnable text hint mark P t The dynamic connection of , the process is expressed as:

[0048] P v ′=F c (P t )=MLP(P t ).

[0049] In some more specific schemes, the training method includes multiple rounds, which are divided into an initial stage, an intermediate stage and an ending stage in sequence. In the intermediate stage, the combined weight of the learnable visual prompt markers and / or the learnable text prompt markers is greater than the combined weight of the initial stage and the ending stage.

[0050] Furthermore, the adjustment process of the combined weight specifically includes:

[0051] Set optimization tips GPA , the optimization tip P GPA represents the optimized learnable visual cue marker and / or learnable text cue marker,

[0052]

[0053] Among them, E is the total number of training rounds, P i represents the learnable visual cue label and / or learnable textual cue label of the i-th training round, W i is the prompt weight of the i-th stage, W i From Gaussian distribution w i ~N(μ,σ 2 )generate.

[0054] A second aspect of the present invention provides a training system for a hybrid prompt learning framework for a small amount of multimodal pathology annotated data, specifically comprising:

[0055] a data acquisition module, configured to acquire an original CLIP model, an initial text prompt, and a training image corresponding to the initial text prompt, wherein the original CLIP model includes an image encoder and a text encoder;

[0056] a cue marker module, configured to insert a learnable visual cue marker into the image encoder and a learnable text cue marker into the text encoder, wherein the learnable visual cue marker is generated by mapping the learnable text cue marker through a coupling function;

[0057] An image training module, configured to input the training image into an image encoder, perform feature encoding on the training image in combination with the learnable visual cue marker, and obtain a visual cue embedding;

[0058] A text training module, configured to input the initial text prompt into a text encoder, combine it with the learnable text prompt marker, perform feature encoding, and obtain a text prompt embedding;

[0059] A similarity calculation module, configured to calculate the similarity between the visual cue embedding and the text cue embedding;

[0060] A parameter updating module is used to adjust the parameters of the learnable visual cue marker, the learnable textual cue marker and the coupling function based on the similarity between the visual cue embedding and the textual cue embedding to obtain a hybrid cue learning framework.

[0061] Compared with the prior art, the advantages of the present invention include at least:

[0062] First, the present invention provides a training method for a hybrid prompt learning framework for a small amount of multimodal pathology annotated data. On the basis of freezing the original CLIP model parameters, by introducing a small number of additional learnable visual prompt markers and learnable text prompt markers, the model can quickly adapt to new knowledge in the medical field, significantly enhancing the model's pathology image classification performance under conditions of limited annotated samples, especially in the classification tasks of rare or emerging diseases.

[0063] Second, the present invention provides a training method for a hybrid prompt learning framework for a small amount of multimodal pathology annotation data. Through a coupling function, the learnable visual prompt mark is explicitly adjusted dynamically according to the corresponding visual content in the learnable text prompt mark. As a bridge between the visual and text modalities, it supports gradient backpropagation and can effectively optimize the collaborative prompts of the two modalities, thereby improving classification performance.

[0064] Third, the present invention provides a training method for a hybrid prompt learning framework for a small amount of multimodal pathology annotated data. The prompts are learned hierarchically in a multi-layer Transformer, combined with the Gaussian weighted prompt integration (GPA) mechanism, to dynamically adjust the prompt weights of each stage, suppress initial noise and avoid overfitting in the later stage, balance general features and task-specific features, and improve cross-domain generalization capabilities. The prompts in the middle rounds of the training stage are given higher weights, while the prompts in the initial stage and the later stage are given lower weights. Gaussian weighted prompt integration balances the expression of contextual features and task distribution in different training stages. The generated prompts can not only capture the collaborative feature representation of visual and language modalities, but also significantly improve the robustness and adaptability of the model in few-sample and cross-domain tasks. This strategy takes into account the task relevance of prompt optimization and the general representation ability of the model, providing greater flexibility for the dynamic adjustment of visual and language representations. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a flowchart of a training method for a mixed prompt learning framework for a small amount of multimodal pathology annotation data provided by an embodiment of the present invention;

[0066] Figure 2 1 is a training diagram of a hybrid prompt learning framework provided by an embodiment of the present invention;

[0067] Figure 3 is a training diagram of a learnable visual cue marker provided by an embodiment of the present invention;

[0068] Figure 4 1 is a training diagram of a learnable text prompt marker provided by an embodiment of the present invention;

[0069] Figure 5 1 is a schematic diagram of multimodal prompt coupling training provided by an embodiment of the present invention;

[0070] Figure 6 It is a schematic diagram of a Gaussian weighted prompt integration mechanism provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In view of the shortcomings of the prior art, the inventors of this case, after long-term research and extensive practice, have proposed the technical solution of the present invention. The following will further explain this technical solution, its implementation process and principles.

[0072] Please refer to Figure 1-Figure 2 ,A training method for a hybrid cue learning framework for a small amount of multimodal pathology annotated data, comprising the following steps:

[0073] S1: Obtain the original CLIP model, the initial text prompt, and the training image corresponding to the initial text prompt. The original CLIP model contains two main modules: image encoder (f v ) and text encoder (f t ).

[0074] S2: inserting a learnable visual cue marker into the image encoder and inserting a learnable text cue marker into the text encoder, wherein the learnable visual cue marker is generated by mapping the learnable text cue marker through a coupling function.

[0075] Specifically, a learnable visual prompt mark and a learnable text prompt mark are preset. The formula of the learnable visual prompt mark is: v ={p1, p2, ..., p m}, m represents the length of the learnable visual prompt mark. The formula of the learnable text prompt mark is: P t ={p1, p2, ..., p n}, n represents the length of the learnable text prompt token.

[0076] Furthermore, through the coupling function F c The learnable text prompt mark is converted into a learnable visual prompt mark, and the coupling function F c It includes a linear transformation network MLP for labeling the learnable text prompt P t Perform projection to generate a label P with the learnable visual cue v The visual cue mark P in the same space v ', establish the learnable visual cue marker P v With learnable text hint mark P t The dynamic connection of , the process is expressed as:

[0077] P v ′=F c (P t )=MLP(P t ).

[0078] In order to achieve the coordinated optimization of visual and textual modal cues, an explicit cue coupling mechanism is proposed. By introducing a modality coupling function, the learnable text cue markers are mapped to the space of learnable visual cue markers, thereby establishing a dynamic connection between the two modalities and improving the joint representation ability of the two modalities. In this mechanism, the learnable visual cue markers (P v ) and learnable text hint tags (P t ) through the synergistic effect of the coupling function F cExplicitly establishing connections and supporting bidirectional gradient propagation enables the visual modality to fully utilize the rich domain knowledge contained in the text modality.

[0079] This mechanism significantly improves the multimodal optimization of prompts, particularly in the few-shot medical setting. Because medical language prompts contain rich domain knowledge, such as disease names or pathology descriptions, the coupling mechanism effectively transfers this knowledge to visual modality feature learning, significantly improving the accuracy of few-shot classification tasks. Furthermore, while reducing the need for labeled data, the prompt coupling mechanism enhances the model's ability to jointly represent multimodal features, providing strong support for complex scenarios such as medical image analysis.

[0080] S3: Input the training image into an image encoder, combine it with the learnable visual cue marker to perform feature encoding, and obtain a visual cue embedding.

[0081] First, the training image is divided into blocks and linearly projected to obtain multiple image block labels. Then, the image category label, the learnable visual cue label, and the multiple image block labels are spliced ​​according to a preset arrangement order to form a label sequence.

[0082] The image category labels are built into the model and constructed during model initialization, rather than being dynamically generated based on the input data. Their initial values ​​are assigned through a random process and fixed at the beginning of the label sequence. As learnable parameters, they are independent of the specific category and are continuously updated during training to capture global feature representations. In this way, image category labels effectively capture the global feature representations in the data, providing important information support for other parts of the model.

[0083] Please refer to Figure 3 , there are three images with different background colors on the left, namely pink, yellow and green, representing three types of images. There are also three texts of different colors on the right, namely well-differentiated tubular adenocarcinoma, moderately differentiated tubular adenocarcinoma and poorly differentiated adenocarcinoma, and their colors correspond one to one with the images on the left. In this scheme, the pink training image is introduced in detail, and it is first divided into blocks to extract the corresponding category labels and four image block labels. According to the preset arrangement order, the category label, the learnable visual cue label, and the image block label, namely "1 blue + 2 yellow + 4 blue", are arranged to form a label sequence. In general, a number of learnable visual cue labels are inserted into the image block label sequence of the input image. Finally, the label sequence is input into the Transformer module of the image encoder to obtain the visual cue embedding.

[0084] Specifically, several learnable visual cue markers are inserted into the sequence of image patch markers of the input image. These learnable visual cue markers are fed into the Transformer module of the image encoder along with the image patch markers, allowing the model to flexibly adapt to new task requirements while maintaining the original encoder structure.

[0085] Please refer to Figure 3 The image encoder includes a first normalization layer, an attention layer, a second normalization layer, and an MLP neural network (multi-layer perceptron). The labeled sequence is input into the image encoder of the original CLIP model to obtain a visual cue embedding, which specifically includes:

[0086] First, the labeled sequence is fed into the first normalization layer to normalize it. This process aims to eliminate scale differences between features. For example, the texture features of adenocarcinoma cell nuclei and the geometric features of ductal structures can be unified to the same dimension. This dimensionality can be achieved by calculating the mean and variance of the current batch of data, effectively enhancing the model's robustness to varying illumination conditions in pathology images.

[0087] Next, the standardized tag sequence is input into the attention layer, which uses the learnable visual cue tag as a query vector to output fine-grained feature association information. Specifically, the following steps are performed: selecting the learnable visual cue tag as a query vector; extracting the image features of each image block tag; calculating the degree of match between the learnable visual cue tag and the image features of each image block tag; assigning a corresponding weight to each image block tag based on the degree of match, assigning a high weight to image block tags with a high degree of match with the learnable visual cue tag, thereby amplifying the features of these image blocks; and assigning a low weight to image block tags with a low degree of match with the learnable visual cue tag, thereby deemphasizing the features of these image blocks; and finally, outputting fine-grained feature association information based on the weight difference. In general, the standardized spliced ​​sequence is input into the attention layer to highlight the key areas in the image. In the attention layer, the learnable visual cue tag (e.g., "highly differentiated") is used as a query vector to establish cross-modal associations with the image features of the image block tags (e.g., ductal structure, nuclear atypia). By calculating the attention weight matrix, the model is able to focus on key areas related to pathological grading (such as the complete glandular duct morphology of well-differentiated adenocarcinoma), thereby more accurately identifying and analyzing key features in pathological images.

[0088] Next, by combining this fine-grained feature association information with the coarse-grained spatial information in the labeled sequence, a fused feature is generated. This ensures that key details in pathology image analysis that may be inadvertently weakened or ignored by the attention mechanism are preserved. These key details include, but are not limited to, the distribution pattern of isolated cell nuclei in poorly differentiated adenocarcinomas. Through the generation of this fused feature, a skip connection mechanism is introduced to effectively prevent the vanishing gradient problem, thereby maintaining the stability of the deep network during training.

[0089] The fused features are then fed into the second normalization layer, where they are normalized using the normalization parameters of the first normalization layer, helping to maintain cross-modal alignment between features. After normalization, the distribution of feature data becomes more stable, which is particularly important for nonlinear transformations, as they often require a stable data distribution as input. This is particularly true when processing features from adenocarcinomas of varying degrees of differentiation, which can vary significantly, necessitating data stability.

[0090] Afterwards, the standardized fusion features are input into the multi-layer perceptron (MLP) neural network for nonlinear mapping and dimensional expansion, and the expanded features are output. In this process, the MLP neural network contains a GeLU activation function to perform nonlinear mapping and dimensional expansion on the features. For example, the original 512-dimensional adenocarcinoma feature vector is expanded to 2048 dimensions through the MLP neural network, which significantly enhances the model's ability to characterize the mixed features of a specific type of adenocarcinoma, namely "moderately differentiated tubular adenocarcinoma". This mixed feature includes partial glandular duct structure and local cell atypia, which is a very important indicator in pathological diagnosis. Afterwards, the expanded feature is added to the fusion feature to form a composite feature (such as a composite feature that includes both the overall tissue architecture and local cytological features of adenocarcinoma).

[0091] Finally, the composite features are projected to obtain the visual cue embedding. The formula for visual cue embedding is: v ′=f v ([P v ;x v ]), x v Represents the image block label, which is frozen (not updated) during training, [P v ;x v ] represents the concatenation of learnable visual cue labels and image patch labels.

[0092] In general, this double residual structure, by introducing two additions, enables the Transformer layer to not only effectively capture global semantic association information, which is mainly dominated by the attention mechanism, but also retain local discriminative features, which are mainly dominated by the MLP.

[0093] S4: Inputting the initial text prompt into a text encoder and combining it with the learnable text prompt tag to perform feature encoding to obtain a text prompt embedding, specifically including:

[0094] The initial text prompt is word-embedded to obtain a text category tag.

[0095] The learnable text prompt tag and the text category tag are concatenated to form a text embedding vector.

[0096] There are fundamental differences between text category tags and image category tags. Text category tags are dynamically generated based on the actual category and are fixed at the end of the text embedding vector sequence. Text category tags represent the actual category name and are converted into vectors through the token_embedding process.

[0097] For details, please refer to Figure 4 In the text category tag sequence, several learnable text prompt tags are inserted. These learnable text prompt tags are input into the Transformer module of the text encoder together with the text category tags. The learnable text prompt tags are initially set to the prompt template: "a photo of a{category}", where {category} represents the text category tag.

[0098] The text embedding vector is input into the text encoder to obtain a text prompt embedding. The text encoder includes multiple encoding layers, and the text embedding vector is input into the text encoder of the original CLIP model to obtain a text prompt embedding, specifically comprising:

[0099] Please refer to Figure 4 , the learnable text prompt marker and text category marker are sequentially input into each encoding layer for training; during the training process, the learnable text prompt marker is in an active state for optimization and adjustment, and the text category marker is in a frozen state and does not participate in the parameter update during the training process; through the multi-layer encoding of the text encoder, the deep features of the text are gradually extracted to determine the text prompt embedding.

[0100] The formula for text hint embedding is: t ′=f t ([P t ;z t ]),

[0101] Among them, z t Indicates the text category label, [P t ;z t ] represents the concatenation of the learnable text prompt tag and the text category tag. Figure 3, where The text category label corresponding to the first pink image The text category label corresponding to the second yellow image The text category label corresponding to the third green image.

[0102] Regarding the insertion of learnable visual cue tokens in the image encoder, the insertion of learnable text cue tokens in the text encoder, and the embedding process, please refer to Figure 5 .

[0103] Specific process of text processing: Input the text "a photo of a cat", and first convert it into digital features recognizable by the model through "Word Embedding" (marked as W0). Specifically, for Word Embedding, the input text is first tokenized and mapped into word embedding vectors. Assume the text contains N words, and its initial word embedding representation is: where d l is the embedding dimension of the language branch (e.g., in CLIP, d l = 512).

[0104] In the first J layers (J < K, where K is the total number of layers) of the text encoder, introduce b learnable cue vectors These cue vectors are concatenated with the initial word embedding and then input into the first layer of Transformer: [[ID= 512).

[0105] Input: where P0 is the cue vector of the first layer, and the cue vectors of subsequent layers are dynamically updated through the self-attention mechanism.

[0106] For the i-th layer (i ≤ J), the input is the cue vector P i-1 output from the previous layer and the word embedding W i-1 , and processed through the Transformer layer L i : [P i , W i = L i ([P i-1 , W i-1 ). Here, L i includes multi-head self-attention (Multi-head Self-Attention) and feed-forward network (Feed-Forward Network). For i > J, the cue vectors stop updating and only pass the existing representations: [P j , W j = L j ([P j-1 , W j-1 ) (j = J + 1,..., K)

[0107] The final text representation z is projected into the common embedding space through the category tag (such as the "[EOS]" tag) output by the last layer of Transformer: in,

[0108] The specific process of image processing: Input image I∈R H×W×3 It is divided into M fixed-size blocks, and each block is converted into an embedding vector by linear projection: Among them, d v is the embedding dimension of the visual branch (such as d in CLIP v =768).

[0109] Visual cues Mapping from the language prompt P under the coupling function F yields: in, is the learnable projection matrix, To offset the items, ensure that the visual and verbal cues are aligned in a common space.

[0110] At the i-th layer (i≤J), the image block is embedded in E i-1 and visual cues After splicing, it is input to the Transformer layer V i : Among them, c i-1 For i>J, the visual cue is no longer updated and only the existing representation is passed.

[0111] The final image representation x is projected into a common embedding space with category labels:

[0112]

[0113] The language cue P is mapped to a visual cue through the coupling function F. Make the two types of prompts in the common embedding space This process forces the linguistic and visual representations to work together in the optimization, avoiding feature shift between the modalities.

[0114] S5: Based on the similarity between the visual cue embedding and the textual cue embedding, adjusting the parameters of the learnable visual cue marker, the learnable textual cue marker, and the coupling function to obtain a hybrid cue learning framework.

[0115] First, the visual cue embedding and the text cue embedding are similarly calculated to output a similarity matrix. Each element S[i][j] in the similarity matrix represents the matching score between the i-th image and the j-th text.

[0116] The matching degree between image representation x and text representation z is measured by cosine similarity:

[0117]

[0118] where z y The text embedding corresponding to category y.

[0119] The final classification probability is calculated by the softmax function:

[0120]

[0121] T is the temperature parameter, which is used to adjust the distribution smoothness.

[0122] Then, the similarity in the similarity matrix is ​​maximized through a loss function, and the learnable visual cue marker, the learnable text cue marker, and the parameters of the coupling function are updated to obtain a hybrid cue learning framework.

[0123] Specifically, the loss function used in the pre-training CLIP model stage can be used. During the fine-tuning stage, backpropagation and gradient descent algorithms (e.g., using the Adam optimizer) are used to adjust the parameters of the image encoder and text encoder. The goal of such adjustments is to ensure that the similarity between each pair of matching images and texts reaches the maximum value, while the similarity between unmatched image and text pairs is as small as possible. From a mathematical point of view, this is equivalent to maximizing the log-likelihood of matching pairs, thereby pushing the features of matching image-text pairs closer to each other in the common feature space, while pushing the features of unmatched image-text pairs further apart.

[0124] The details are as follows:

[0125]

[0126] Where sim(·,·) represents the cosine similarity, τ is the temperature coefficient, N is the batch size; i and t represent a pair of matching images and texts; Z i Represents the i-th image category label, corresponding to the feature vector of the classtoken of the image; Z t represents the t-th text category tag; Z j is a general representation that refers to the feature representation of the jth image or text. This representation is used to construct negative samples. Negative samples refer to those mismatched image and text pairs, which are used to train the model to distinguish between correct matches and incorrect matches, thereby improving the model's discriminative ability.

[0127] The core goal of contrastive learning is to maximize the similarity between paired images and text, while minimizing the similarity between unpaired samples. This approach effectively trains the model to recognize and understand the correspondence between image content and text descriptions. Within the context of a cued learning framework, this loss function further incorporates visual and linguistic cues to meticulously optimize the model. This optimization process is particularly well-suited for tasks with a small number of samples, enabling the model to learn effective feature representations and association rules even with limited data.

[0128] To gradually model hierarchical representations of multimodal features, we learn cues across multiple Transformer modules in both the vision and language branches. This deep cueing strategy, by introducing learnable contextual cues, captures task-relevant contextual relationships layer by layer while providing greater flexibility for dynamic adaptation of multimodal representations. In medical image classification, this staged cue learning is able to better capture fine-grained features of complex lesions.

[0129] However, since the random initialization of the prompts in the initial stage will lead to noise influence, and the prompts in the later stage of training tend to be biased towards the supervised distribution of specific tasks, the single-stage optimization of the prompts will lead to a decrease in the generalization ability of the model. To this end, in this solution, the training method includes multiple rounds, which are divided into an initial stage, an intermediate stage and an end stage in sequence. In the intermediate stage, the combined weight of the learnable visual prompt markers and / or the learnable text prompt markers is greater than the combined weight of the initial stage and the end stage. The adjustment process of the combined weight specifically includes:

[0130] Set optimization tips GPA , the optimization tip P GPA represents the optimized learnable visual cue marker and / or learnable text cue marker,

[0131]

[0132] Among them, E is the total number of training rounds, P i represents the learnable visual cue label and / or learnable textual cue label of the i-th training round, W i is the prompt weight of the i-th stage, W i From Gaussian distribution w i ~N(μ,σ 2 )generate.

[0133] Please refer to Figure 6, red is a learnable textual cue marker, and green is a learnable visual cue marker. The curve is a Gaussian distribution (giving the cue Gaussian distribution a weight). It should be noted that this application is not limited to using the Gaussian distribution, which is only used as an example.

[0134] The cues are learned hierarchically in a multi-layer Transformer, combined with the Gaussian Weighted Cues Integration (GPA) mechanism to dynamically adjust the weights of cues at each stage, suppressing initial noise and avoiding overfitting in the later stages, balancing general features with task-specific features, and improving cross-domain generalization capabilities. Cues in the middle rounds of training are given higher weights, while cues in the initial and later stages are given lower weights. i Dynamically adjust in each round of optimization to balance the contextual features and task distribution of prompts at different stages. This mechanism suppresses the noise contribution of immature prompts in the early stage, while preventing the prompts from being too biased towards the supervised task distribution in the later stage.

[0135] Through a phased prompt learning strategy, prompt optimization gradually integrates task-independent generalization features and task-specific contextual representations throughout the training process. The generated prompts can not only improve the performance of downstream tasks, but also effectively retain the general representation ability of the pre-trained model, especially showing superior adaptability in few-sample and cross-domain tasks.

[0136] This application includes at least three image recognition solutions for small amounts of multimodal pathology annotated data. However, it should be noted that this solution is based on image processing and information extraction technology for medical images, and is not a direct disease diagnosis and treatment method. Disease diagnosis requires the professional judgment of a doctor. This technical solution provides auxiliary information based on image analysis, not direct diagnostic information. It is not possible to independently diagnose a specific disease based solely on the information provided by the method provided by this invention.

[0137] The first approach involves inputting the medical image to be detected and a predefined set of object names into the hybrid prompt learning framework to obtain the target object name corresponding to the medical image to be detected. In other words, similar to image retrieval technology, its goal is to find the descriptive text that best matches a specific medical image. The specific steps include:

[0138] The medical image to be detected is divided into blocks to obtain a category label and multiple image block labels of the medical image to be detected; according to a preset arrangement order, the updated learnable visual cue label is combined with the category label and multiple image block labels of the medical image to be detected to form a label sequence; the label sequence is input into the image encoder of the updated CLIP model to obtain pathological image features of the medical image to be detected;

[0139] Perform word embedding on a predefined set of object names to obtain text category tags corresponding to the medical image to be detected; combine the updated learnable text prompt tags and the text category tags corresponding to the medical image to be detected to form a text embedding vector; input the text embedding vector into the text encoder of the updated CLIP model to obtain the category text features of the medical image to be detected;

[0140] The similarity between the pathological image features and the text features of each category is calculated, and the object name corresponding to the category text feature with the highest similarity is selected as the target object name.

[0141] When a pathological image of gastric adenocarcinoma to be examined is input, it is first resized to 224×224 and normalized. A predefined set of object names, including "well-differentiated tubular adenocarcinoma," "moderately differentiated tubular adenocarcinoma," and "poorly differentiated adenocarcinoma," is used. The image to be examined is fed into CLIP's image encoder, where updated learnable visual cue tags are added to the feature sequence. The category description text is fed into CLIP's text encoder, where updated learnable text cue tags are added to the text sequence. Using a multimodal cue coupling mechanism, the system uses a trained linear transformation network to project the text cue into a visual cue space. The coupled feature representations are then obtained in nine Transformer modules in the vision and language branches. Finally, by calculating the similarity between the pathological image features and the text features for each category, the category with the highest similarity is selected as the final classification result, thereby diagnosing the gastric adenocarcinoma subtype for the input pathological image. When a pathological image suspected of gastric adenocarcinoma is input, the system calculates the degree of match between the image and the three predefined category descriptions. If the image has the highest similarity to "well-differentiated tubular adenocarcinoma," the image is classified as this subtype. This reasoning approach, based on visual-linguistic multimodal cues, fully leverages the rich feature representations captured by the pre-trained model, enabling accurate pathology image classification even with minimal labeled data.

[0142] The second solution is to input the name of the object to be detected and a predefined set of medical images into the hybrid prompt learning framework to obtain the target medical image corresponding to the name of the object to be detected. In other words, it is equivalent to a text retrieval process that aims to find the medical image that best matches a specific text description. In short, an image candidate library containing multiple images is first constructed. Subsequently, the images in the image candidate library are combined with visual prompts and input into the image encoder to generate a visual feature vector. At the same time, a single text is combined with a text prompt and input into the text encoder to generate a text feature vector. Next, the similarity between the visual feature vector and the text feature vector is calculated to generate a similarity matrix, where the matrix element S[i][j] represents the matching score between the i-th image and the single text. Finally, the images in the image candidate library are sorted according to the similarity score, and the image with the highest similarity score is selected as the final matching result.

[0143] The third solution is to input the medical image to be detected and its corresponding name of the object to be detected into the hybrid prompt learning framework to obtain the matching result of the medical image to be detected and the name of the object to be detected. In other words, it is a binary classification judgment to determine whether the image and text match. In short, first obtain a single image and its corresponding category text, which covers multiple categories. Then, combine the single image with the visual prompt and input it into the image encoder to generate a visual feature vector. At the same time, combine the category text with the text prompt and input it into the text encoder to generate a text feature vector. Then, calculate the similarity of the visual feature vector and the text feature vector to obtain a similarity matrix, in which each element S[i][j] represents the matching score between the image and the jth category. By comparing these scores, the category with the highest matching score is selected as the classification result of the single image.

[0144] Next, we introduce the experimental settings:

[0145] This protocol was experimented on the PatchGastric dataset, which consists of histopathological images extracted from H&E-stained whole-slide images (WSIs) of gastric adenocarcinoma endoscopic biopsy specimens. The dataset contains 262,777 300×300 patches, derived from 991 WSIs at a magnification of x20. The dataset covers nine gastric adenocarcinoma subtypes. In this protocol, three major subtypes, namely "well-differentiated tubular adenocarcinoma," "moderately differentiated tubular adenocarcinoma," and "poorly differentiated adenocarcinoma," were selected to form a three-class classification task, along with 179,285 patches from 693 WSIs. The dataset was randomly split into a training set (20%) and a validation set (80%) to evaluate model performance. To validate the model's performance in data-limited environments, we selected 1, 4, 8, or 16 WSIs as training sets, which contain the largest number of patches in each class. Classification accuracy was used as the primary evaluation metric.

[0146] In the experiment, the data loader used a batch size of 4 for the training subset and 100 for the test subset, with eight data loading worker threads. The input images were resized to 224×224, the interpolation method was bicubic, and the pixel values ​​were normalized to a mean of [0.485, 0.456, 0.406] with a standard deviation of [0.229, 0.224, 0.225]. During training, data augmentation techniques including color jitter, random flips, and normalization were applied to improve the model's generalization capabilities.

[0147] The optimizer selected SGD (stochastic gradient descent), with an initial learning rate of 0.0035, a momentum of 0.9, and a weight decay coefficient of 1×10 -5 The training adopts the cosine annealing learning rate scheduler with a maximum training round of 5 rounds. In the first epoch, a constant learning rate warm-up strategy is used, and the warm-up learning rate is set to 1×10 -3 , the minimum learning rate is 1×10 -5 .

[0148] The backbone network of the model uses ViT-B / 16 (VisionTransformer). Specifically, the context length of the visual cue N ctx Setting this to 2 results in a hint with floating-point precision of fp16. Experiments have shown that setting the hint depth to 9 yields the best results, meaning that hints are inserted into the first nine layers of the Transformer module to gradually learn multimodal feature representations. A phased multimodal hint learning strategy is introduced during model training, significantly improving model performance when using limited annotated data.

[0149] Next, we introduce the data results:

[0150] The proposed multimodal cue learning framework was thoroughly evaluated and analyzed under both few-shot (1-shot, 4-shot, 8-shot, 16-shot) and full-shot (All) conditions. Table 1 shows the comparison of the classification results of our approach (labeled Ours) and other approaches on the PatchGastric dataset.

[0151] Table 1

[0152] Method 1shot 4shots 8shots 16 shots All Finetune 39.2 43.9 51.6 57.1 66.3 LinerProbe 47.6 51.2 60.1 61.3 65.4 VPT 45.8 47.6 50.4 54.3 63.9 CITE 58.2 59.7 63.2 65.9 68.7 Ours 54.3 58.6 65.7 71.2 73.6

[0153] As shown in Table 1, our scheme (labeled "Ours") outperforms most baseline methods in classification accuracy across all scenarios, demonstrating its significant advantages in both few-shot and full-shot learning. In particular, in the low-shot scenario, our approach achieves an accuracy of 54.3% under the extremely low-shot 1-shot condition, surpassing methods such as Finetune (39.2), LinearProbe (47.6), and VPT (45.8). Although slightly lower than CITE (58.2), our performance improves further as the number of samples increases, reaching an accuracy of 71.2% under the 16-shot condition, surpassing not only CITE (65.9) but all other methods, demonstrating our scheme's effectiveness when utilizing more samples. Furthermore, under the full-shot (All) condition, our approach achieves an accuracy of 73.6%, a significant improvement over CITE (68.7), Finetune (66.3), LinearProbe (65.4), and VPT (63.9). Specifically, our approach achieves accuracies of 58.6% and 65.7% with 4 and 8 shots, respectively, significantly outperforming Finetune (43.9% and 51.6%, respectively), LinearProbe (51.2% and 60.1%, respectively), and VPT (47.6% and 50.4%, respectively). Even compared to the current state-of-the-art CITE method, our approach outperforms it with 8 and 16 shots (65.7% vs. 63.2% and 71.2% vs. 65.9%, respectively), demonstrating its superior generalization and robustness.

[0154] In summary, the experimental results show that our proposed multimodal cue learning method not only performs well in few-shot learning tasks, but also demonstrates strong performance under full-shot conditions, surpassing multiple current mainstream baseline methods, further verifying the effectiveness and superiority of this scheme.

[0155] This application also provides a training system for a hybrid prompt learning framework for a small amount of multimodal pathology annotated data, including:

[0156] a data acquisition module, configured to acquire an original CLIP model, an initial text prompt, and a training image corresponding to the initial text prompt, wherein the original CLIP model includes an image encoder and a text encoder;

[0157] a cue marker module, configured to insert a learnable visual cue marker into the image encoder and a learnable text cue marker into the text encoder, wherein the learnable visual cue marker is generated by mapping the learnable text cue marker through a coupling function;

[0158] An image training module, configured to input the training image into an image encoder, perform feature encoding on the training image in combination with the learnable visual cue marker, and obtain a visual cue embedding;

[0159] A text training module, configured to input the initial text prompt into a text encoder, combine it with the learnable text prompt marker, perform feature encoding, and obtain a text prompt embedding;

[0160] A similarity calculation module, configured to calculate the similarity between the visual cue embedding and the text cue embedding;

[0161] A parameter updating module is used to adjust the parameters of the learnable visual cue marker, the learnable textual cue marker and the coupling function based on the similarity between the visual cue embedding and the textual cue embedding to obtain a hybrid cue learning framework.

[0162] It should be understood that the above embodiments are merely illustrative of the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent variations or modifications made in accordance with the spirit and substance of the present invention are intended to be encompassed within the scope of protection of the present invention.

Claims

1. A training method for a hybrid prompt learning framework for a small amount of multimodal pathology annotated data, characterized by: The following steps are involved: Obtaining an original CLIP model, an initial text prompt, and a training image corresponding to the initial text prompt, wherein the original CLIP model includes an image encoder and a text encoder; Inserting a learnable visual cue marker into the image encoder and inserting a learnable text cue marker into the text encoder, wherein the learnable visual cue marker is generated by mapping the learnable text cue marker through a coupling function; Inputting the training image into an image encoder and combining it with the learnable visual cue marker to perform feature encoding to obtain a visual cue embedding; Inputting the initial text prompt into a text encoder and combining it with the learnable text prompt tag to perform feature encoding to obtain a text prompt embedding; Based on the similarity between the visual cue embedding and the textual cue embedding, the learnable visual cue marker, the learnable textual cue marker and the parameters of the coupling function are adjusted to obtain a hybrid cue learning framework.

2. The training method according to claim 1, characterized in that During the training process, each feature layer of the original CLIP model is frozen.

3. The training method according to claim 1, characterized in that Specifically include: Dividing the training image into blocks and linearly projecting the blocks to obtain multiple image block labels; splicing the image category label, the learnable visual cue label, and multiple image block labels to form a label sequence, wherein the image category label is obtained by initializing the original CLIP model; Inputting the tag sequence into the image encoder to obtain a visual cue embedding; Perform word embedding on the initial text prompt to obtain a text category tag; Concatenate the learnable text prompt tag and the text category tag to form a text embedding vector; Inputting the text embedding vector into the text encoder to obtain a text prompt embedding; Performing similarity calculation on the visual cue embedding and the text cue embedding, and outputting a similarity matrix; The similarity in the similarity matrix is ​​maximized, the learnable visual cue marker, the learnable text cue marker, and the parameters of the coupling function are updated to obtain a hybrid cue learning framework.

4. The training method according to claim 3, characterized in that Any feature layer of the image encoder includes a first normalization layer, an attention layer, a second normalization layer, and an MLP neural network, and the tag sequence is input into the image encoder to obtain a visual cue embedding, specifically including: Inputting the labeled sequence into the first normalization layer to perform normalization on the labeled sequence; Inputting the normalized tag sequence into the attention layer, using the learnable visual cue tag as a query vector, and outputting fine-grained feature association information; Adding the fine-grained feature association information to the coarse-grained spatial information of the label sequence to obtain a fused feature; Inputting the fused features into the second normalization layer to perform normalization on the fused features; Input the standardized fusion features into the MLP neural network, perform nonlinear mapping and dimension expansion, and output the expanded features; Adding the extended feature to the fused feature to form a composite feature; Projecting the composite features to obtain a visual cue embedding; And / or, the formula of the learnable visual cue mark is: P v ={p1, p2, ..., p m }, m represents the length of the learnable visual cue token, And / or, the formula for embedding the visual cue is: v =f v ([P v ;x v ]), x v represents the image block label, [P v ;x v ] represents the concatenation of learnable visual cue labels and image patch labels, And / or, the formula of the learnable text prompt mark is: P t ={p1,p2,...,p n }, n represents the length of the learnable text prompt token, And / or, the formula for embedding the text prompt is: t ′=f t ([P t ;z t ]), z t Indicates the text category tag, [P t ;z t ] represents the concatenation of the learnable text prompt tag and the text category tag, and / or, normalizing the marker sequence by calculating the mean and variance; And / or, the fused features are input into the second normalization layer, and the fused features are normalized using the normalization parameters of the first normalization layer.

5. The training method according to claim 4, characterized in that The method of using the learnable visual cue marker as a query vector to output fine-grained feature association information specifically includes: Selecting the learnable visual cue marker as a query vector; Extracting image features of each image block mark; Calculating a degree of match between the learnable visual cue marker and the image features of each image block marker; According to the matching degree, each image block is marked with a corresponding weight; Based on the weight difference, fine-grained feature association information is output.

6. The training method according to claim 3, characterized in that: The text encoder includes a plurality of encoding layers, and the text embedding vector is input into the text encoder to obtain a text prompt embedding, including: The learnable text prompt marker and text category marker are sequentially input into each encoding layer for training; during the training process, the learnable text prompt marker is in an active state for optimization and adjustment, and the text category marker is in a frozen state; through the multi-layer encoding of the text encoder, the deep features of the text are gradually extracted to determine the text prompt embedding.

7. The training method according to claim 1, characterized in that Specifically include: Through the coupling function F c The learnable text prompt mark is converted into a learnable visual prompt mark, and the coupling function F c It includes a linear transformation network MLP for labeling the learnable text prompt P t Perform projection to generate a label P with the learnable visual cue v The visual cue mark P in the same space v ', establish the learnable visual cue marker P v With learnable text hint mark P t The dynamic connection of , the process is expressed as: P v ′=F c (P t )=MLP(P t )。 8. The training method according to claim 1, characterized in that The training method includes multiple rounds, which are divided into an initial stage, an intermediate stage and an ending stage in sequence. In the intermediate stage, the combined weight of the learnable visual prompt mark and / or the learnable text prompt mark is greater than the combined weight of the initial stage and the ending stage.

9. The training method according to claim 8, characterized in that The adjustment process of the combined weight specifically includes: Set optimization tips GPA , the optimization tip P GPA represents the optimized learnable visual cue marker and / or learnable text cue marker, Among them, E is the total number of training rounds, P i represents the learnable visual cue label and / or learnable textual cue label of the i-th training round, W i is the prompt weight of the i-th stage, W i From Gaussian distribution w i ~N(μ,σ 2 )generate.

10. A training system for a hybrid prompt learning framework for a small amount of multimodal pathology annotated data, characterized by: include: a data acquisition module, configured to acquire an original CLIP model, an initial text prompt, and a training image corresponding to the initial text prompt, wherein the original CLIP model includes an image encoder and a text encoder; a cue marker module, configured to insert a learnable visual cue marker into the image encoder and a learnable text cue marker into the text encoder, wherein the learnable visual cue marker is generated by mapping the learnable text cue marker through a coupling function; An image training module, configured to input the training image into an image encoder, perform feature encoding on the training image in combination with the learnable visual cue marker, and obtain a visual cue embedding; A text training module, configured to input the initial text prompt into a text encoder, combine it with the learnable text prompt marker, perform feature encoding, and obtain a text prompt embedding; A similarity calculation module, configured to calculate the similarity between the visual cue embedding and the text cue embedding; A parameter updating module is used to adjust the parameters of the learnable visual cue marker, the learnable textual cue marker and the coupling function based on the similarity between the visual cue embedding and the textual cue embedding to obtain a hybrid cue learning framework.

Images