Second-order few-sample medical image classification method and system based on multi-mode prompt learning, storage medium and equipment

Abstract

The invention belongs to the technical field of computer vision image processing, and provides a second-order few-sample medical image classification method and system based on multi-modal prompt learning, a storage medium and equipment, and the method comprises the steps: obtaining few-sample medical image data, introducing auxiliary text input to further describe an image or concept category, promoting cross-modal learning, and carrying out multi-modal prompt learning. Two classification heads are combined: one is a shared classification head for processing image category marks in a visual encoder and prompt representations encoded by a text encoder; another is to classify a visual marker feature distribution from a visual encoder by aggregating visual markers using global covariance pooling with efficient matrix power normalization. According to the method, the classification precision of the few-sample medical images is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a second-order few-sample medical image classification method, system, storage medium and device for multimodal prompt learning. Background Art

[0002] Multimodal recognition technology integrates data from multiple modalities such as text, images, and sound, enhancing the performance of tasks such as image recognition. CLIP, a visual language pre-training model, builds a multimodal framework to learn a transferable visual model from natural language supervision, expanding the scope of the model. By introducing learnable text prompt vectors, lossless adjustments to the original model are achieved to adapt to the challenges of diverse data sets.

[0003] Medical image classification methods rely on the latest progress in deep learning. With the development of deep learning technology, especially the introduction of convolutional neural networks (CNN) and Transformer, the accuracy of medical image classification has been significantly improved. However, due to the scarcity and difficulty of annotation of medical image data, traditional supervised learning methods face challenges. Few-shot learning trains models with good performance through limited sample data, providing a solution for medical image classification in the case of data scarcity.

[0004] Vision-language pre-trained models provide excellent zero-shot and few-shot performance by performing multimodal learning through correspondence between images and text. The CLIP model achieves alignment between images and text through large-scale pre-training. However, these models have limited application in medical image analysis. The development of vision-language models for medical images is of great significance. This patent combines vision-language models with medical image classification, and improves the accuracy and generalization ability of medical image classification by introducing multimodal learning and prompt learning techniques.

[0005] Hint learning guides the model to adapt to new tasks by designing or learning hints. In medical image classification, hint learning helps the model better understand and identify objects in medical images. This patent explores a variety of hint methods, including manually designed class names, common methods, and manually crafted methods, using GPT to generate descriptions related to data categories as hints, providing additional information, and using learnable hints to improve classification performance.

[0006] In addition, CLIP only uses first-order information as a representation of the whole, which ignores potential high-order correlation information between features. Second-order pooling captures global information by calculating the covariance matrix between feature maps. In medical image classification, second-order pooling helps the model utilize high-order statistical information and improve classification accuracy. This patent applies second-order pooling to medical image classification, combines multimodal learning and prompt learning techniques to further improve model performance, explores the characteristics and applicability of different second-order pooling methods, and provides new ideas for large-scale visual-language pre-training models in medical image analysis tasks. Summary of the invention

[0007] In view of the defects in the prior art, the present invention proposes a second-order few-sample medical image classification method with multimodal prompt learning, which further describes the image or concept category by introducing text input, promotes the few-sample learning of medical images, and improves the accuracy and generalization ability of medical image classification.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A second-order few-shot medical image classification method with multimodal cue learning, the steps are as follows:

[0010] S1, obtain a few-sample medical image data for model training. Introduce text input for each image to further describe the image or concept category.

[0011] Furthermore, in step S1, before introducing text input into each image, data enhancement processing is performed on the acquired few-sample medical image data.

[0012] S2, build the model: The model uses CLIP as the backbone network, including an image encoder and a text encoder. The medical image and category description are used as visual input and text input respectively. After the image encoder and text encoder extract features, the image encoder generates a category label X c and visual marker X v , the text encoder generates a global sentence representation;

[0013] Furthermore, before model training, a prompt method needs to be selected from the prompt pool to generate prompts for each category. Then, the respective encoders are used to extract visual and text features;

[0014] Furthermore, the prompt pool includes five different prompt methods, including class name, general prompt, manual prompt, prompt generated by GPT, and CoOp learnable prompt method.

[0015] The GPT generates prompts as follows: GPT uses the Transformer architecture and is extensively pre-trained on large text datasets to achieve coherent text generation and understanding. In addition, an interactive tool ChatGPT based on GPT 3.5 is used to create prompts of different categories. Specifically, two methods are used to generate these prompts: Method 1 is: before requesting prompts for categories in a dataset, first confirm whether ChatGPT understands a dataset, and then ask GPT to give a detailed description of the dataset category as a prompt. Method 2 is: use a template customized for the dataset to formulate prompts, that is, use the template to combine specific categories to ask GPT questions and let it give category-related content.

[0016] The CoOp learning tip: Using the CoOp framework, the input formula of the text encoder is designed as:

[0017] T=[V]1[V]2...[V] M [cls]

[0018] Among them, each [V] m They all match the dimension of the word embedding and are shared among all categories [cls], m∈{1,2,...,M}, where M is a hyperparameter that determines the number of context tokens.

[0019] Furthermore, in step S2, the text encoder is used to generate a feature representation of the text description, and the specific process is as follows: each token in the text description is encoded into a unique digital ID through a byte pair encoding algorithm, and these digital IDs are mapped to 512-dimensional word embedding vectors and input into a text encoder stacked with a Transformer layer. Each text description is surrounded by [SOS] and [EOS] tags, [SOS] means "start sentence", and [EOS] means "end sentence", and the feature at the [EOS] tag position is finally linearly projected as a global sentence representation;

[0020] Furthermore, in step S2, the image encoder of CLIP can adopt ResNet or ViT architecture. When ViT is used as the image encoder, the image is divided into N non-overlapping image blocks and projected into the patch embedding P v These embeddings P v and a learnable class label P c It is input into K Transformer blocks for sequential processing. When ResNet is used as the image encoder, the image passes through convolutional layers and multi-head attention pooling in sequence, combining the learnable category labels with the image block visual features.

[0021] Furthermore, for efficiency, the text encoder and image encoder are frozen during training; only the learnable cues and parameters of the classifiers involved in the CoOp method are updated.

[0022] S3, constructing a visual classifier: the classifier includes a shared classifier and a global covariance pooling classifier. The shared classifier performs category prediction based on text features and visual class tokens, and the global covariance pooling classifier realizes linear classification of feature distribution by combining first-order and second-order statistics.

[0023] The shared classifier described in step S3: The shared classifier performs category prediction based on the [EOS] token from the text encoder and the category tag from the image encoder.

[0024] The global covariance pooling classifier described in step S3 is used for class prediction of the visual part. There is a class label and multiple visual labels in the CLIP model. These visual labels come from the output of the block before the pre-trained CLIP model classifier. The linear detection takes the class label or the average pooling of the visual label as input to generate a prediction. For the global covariance pooling classifier, it is necessary to calculate the covariance pool of the visual label for second-order statistical modeling. The second-order prediction is then added to the first-order prediction represented by a single class label to form the final prediction of the visual component;

[0025] Furthermore, the covariance pooling of the calculated visual markers refers to replacing the original average pooling operation with a feature function, which is obtained by modeling the feature distribution of the visual markers and can depict a complete image of the features, and the covariance pooling is standardized by Newton-Schultz iteration to obtain a square root standardized covariance matrix;

[0026] S4, only updates the learnable cues of CoOp and the parameters of the classifier during the training phase. Only predicts new medical images during the inference phase.

[0027] During the training phase, text labels are the same as image labels, and text prompts serve as auxiliary samples to enhance the interaction between text and vision.

[0028] During the inference phase, the shared classifier makes category predictions based on the [EOS] tokens from the text encoder and the category labels from the image encoder.

[0029] S5, output the medical image classification results.

[0030] As a further implementation, in order to optimize the network, the AdamW algorithm was used, combined with a warm-up phase followed by a cosine annealing learning rate schedule. The warm-up phase consisted of 50 iterations, and the entire training spanned 12,800 iterations.

[0031] As a further implementation, it is important to emphasize the rigor of the present invention, specifically: using three different random seeds to sample a specified number of small instances from the training set as training samples for the model.

[0032] A second aspect of the present invention provides a few-sample medical image classification system based on multimodal cue learning, comprising:

[0033] Data acquisition module: used to obtain small sample medical image data;

[0034] Text prompt generation module: used to generate text prompts, that is, to select a prompt method from the prompt pool to generate text prompts. The different prompt methods in the prompt pool include class name, common prompt, manually made prompt, prompt generated by GPT, and CoOp learnable prompt method;

[0035] Model building module: The model uses CLIP as the backbone network, including an image encoder and a text encoder. Medical images and category descriptions are used as visual input and text input respectively, and features are extracted through the image encoder and text encoder;

[0036] Visual classifiers, including shared classifiers and global covariance pooling classifiers: used to make classification predictions of visual features;

[0037] Training module: During the training phase, text prompts are used as auxiliary samples to enhance the interaction between text and vision and perform classification predictions;

[0038] Reasoning module: In the reasoning stage, only predictions are made for medical images to simplify the reasoning process;

[0039] Output module: Outputs the few-sample medical image classification results.

[0040] The third aspect of the present invention provides a medium having a program stored thereon, which, when executed by a processor, implements the steps in the second-order few-sample medical image classification method based on multimodal cue learning described in the first aspect of the present invention.

[0041] The fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the second-order few-sample medical image classification method based on multimodal cue learning described in the first aspect of the present invention are implemented.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The proposed second-order few-sample medical image classification method based on multimodal prompt learning enhances the model's ability to understand and classify data by introducing auxiliary text input and multimodal learning methods. The covariance pooling and feature distribution modeling techniques are used to further improve the model's prediction accuracy and generalization ability, and it performs well in medical image classification tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0045] Figure 1 It is a flow chart of the second-order few-sample medical image classification method based on multimodal cue learning of the present invention;

[0046] Figure 2 It is a framework diagram of the second-order few-sample medical image classification method based on multimodal cue learning of the present invention;

[0047] Figure 3 It is a framework diagram of the visual classifier of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0049] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0050] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0051] Embodiment 1

[0052] like Figure 1 As shown, the first embodiment of the present invention provides a flowchart of a second-order few-sample medical image classification method based on multimodal cue learning, including the following steps:

[0053] S1. Obtain a few-sample medical image data for model training, introduce auxiliary text input for each image to further describe the image or concept category, promote cross-modal learning, and help understand medical image data.

[0054] S2. Build a model. The model uses the CLIP model as the backbone network. The model contains an image encoder and a text encoder.

[0055] S3. Use a shared classifier to predict categories based on text features and visual class tokens. Construct a global covariance pooling classifier that achieves linear classification of feature distribution by combining first-order and second-order statistics.

[0056] S4. Only the learnable cues of CoOp and the parameters of the classifier are updated during the training phase. Only new medical images are predicted during the inference phase.

[0057] S5. Output the medical image classification results.

[0058] Among them, S1 is specifically:

[0059] The present invention utilizes three public medical image datasets and introduces detailed auxiliary text prompts for each image to further describe the image or concept category and promote cross-modal learning.

[0060] The BACH (Breast Cancer Histology) dataset contains 400 histopathology images of breast cancer biopsies, which are divided into four categories: normal, benign, carcinoma in situ, and invasive cancer, with 100 images in each category. A detailed text description is introduced for each image, such as "This is a histopathology image from a breast cancer biopsy, showing normal tissue" or "This is a histopathology image showing a benign tumor."

[0061] Figshare MRI brain tumor dataset contains 3064 T1-weighted contrast-enhanced images from 233 patients, divided into meningiomas, gliomas, and pituitary tumors, with the number of images being 708, 1426, and 930, respectively. Add detailed text descriptions to each image, such as "This is a T1-weighted contrast-enhanced image showing meningioma" or "This is an MRI image showing glioma".

[0062] The diabetic retinopathy (DR) dataset contains 3662 high-resolution color retinal images, which are divided into five categories according to the severity of the lesions: non-proliferative, mild, moderate, severe and proliferative diabetic retinopathy. Descriptive text is added to each image, such as "This is a high-resolution color image showing moderate non-proliferative diabetic retinopathy" or "This is an image showing mild non-proliferative diabetic retinopathy".

[0063] When using the above data sets, the present invention divides them into a training set and a test set with a split ratio of 7:3.

[0064] Based on existing research, this paper explores five different prompting methods, namely class name, vanilla, hand-crafted, category-specific descriptions generated by GPT, and a method that can learn CoOp.

[0065] Using the BACH dataset as an example, these methods are described in detail in Table 1. The first three rows of the table summarize the specifics of the first three methods. Next, we elaborate on how to generate prompts using GPT and the learnable CoOp method.

[0066] GPT generates prompts: GPT adopts the Transformer architecture and is extensively pre-trained on huge text datasets to achieve coherent text generation and understanding, which makes it very valuable in various NLP tasks. In this paper, an interactive tool ChatGPT based on GPT 3.5 is used to create prompts of different categories. Two methods are used to generate these prompts. First, before requesting prompts for categories in the dataset, confirm whether ChatGPT understands the specific dataset, as shown in the fifth row in the table. Second, a structured template is used to formulate prompts.

[0067] CoOp's learnable prompts: The CoOp method innovatively solves the problem of large amounts of prompt engineering time by customizing a CLIP-like visual-language model for image recognition tasks and using learnable vectors as placeholders for context words in prompts. In this invention, following the CoOp framework, the input formula of the text encoder is designed as:

[0068] T=[V]1[V]2...[V] M [cls]

[0069] In this setup, each [V] m (where m∈{1,2,...,M}) matches the dimensionality of the word embeddings (typically 1024 or 512 for CLIP) and is shared across all classes [cls]. M is a hyperparameter that determines the number of context tokens. We adopt a consistent context approach throughout the process based on the findings of the CoOp evaluation.

[0070] S2 is specifically:

[0071] The preprocessed few-sample medical image data and its corresponding category prompts are input into the pre-trained multimodal model for feature extraction. The model infrastructure is composed of the CLIP model; CLIP is a multimodal structure with two input branches: text and image; the encoding process of the two encoders is as follows.

[0072] The text encoding process is:

[0073] The CLIP text encoder aims to generate feature representations of text descriptions T. First, each token in the text description is encoded into a unique numeric ID through the Byte Pair Encoding (BPE) algorithm. Then, these numeric IDs are mapped to 512-dimensional word embedding vectors, which are input into the text encoder L with stacked transformer layers. Note that before token embedding, each text description is surrounded by [SOS] and [EOS] tokens. [SOS] means "Start Of Sentence" and [EOS] means "End Of Sentence". Finally, the features at the [EOS] token position are used as the global sentence representation, followed by a linear projection (Projtxt), which is expressed as follows:

[0074] [SOS,...,EOS]=L(T)

[0075] T c =Proj txt (T idx ([EOS]))

[0076] Among them, T idx ([EOS]) indicates the index of the [EOS] tag, and T c is the global sentence representation, Proj txt (·) denotes linear projection.

[0077] The image encoding process is:

[0078] CLIP's image encoder can use architectures such as ResNet or ViT. Let the visual encoder be V. When using ViT as V, the image is first split into N non-overlapping image patches and projected into the patch embedding At the same time, these embedded P v and a learnable class label P c are input into K transformer blocks (e.g., for ViT-B, K=12) for sequential processing. When ResNet is used as V, the image I first passes through the convolutional layer and then through multi-head attention pooling at the end of the network to combine the learnable category labels with the image block (visual) features. Therefore, the image encoding process in CLIP is formulated as:

[0079] [X c ,X v ]=V(I)

[0080] According to this design, no matter which visual encoder is chosen, two types of tags will be generated in the end: category tag X c and visual marker Xv .

[0081] S3 is specifically:

[0082] The classifiers include a shared classifier and a global covariance pooling classifier. The shared classifier predicts categories based on text features and visual class tokens, while the global covariance pooling classifier implements linear classification of feature distribution by combining first-order and second-order statistics. The shared classifier predicts categories based on the [EOS] token from the text encoder and the category tag from the image encoder. The global covariance pooling classifier predicts categories based on X c and Xv as input. On top of the image encoder, this classifier primarily captures the second-order statistics inherent in the visual tags (i.e., covariance) and makes predictions based on these statistics, which have been neglected in previous work. In addition, the classifier also takes into account the class label, which is the first-order statistic of the global image representation (the class label itself). Therefore, such a visual classifier is achieved by considering both the first-order and second-order moments of the features simultaneously.

[0083] Figure 3 is the framework diagram of the visual classifier of the present invention. In the CLIP model, there is a class token and multiple visual tokens. Given a class token X c ∈R d and a set X containing N d-dimensional visual labels v ∈R d×N , these visual labels come from the output of the previous block of the pre-trained CLIP model classifier. Linear probing usually converts the class label X c or visual mark X v The average pooling of is taken as input and the prediction is generated in the following way, which is expressed as:

[0084]

[0085] Where W is the weight of the linear layer, is some transformation of the input data (probably the identity transformation in this case, since we already have an appropriate representation), X is X c or X v . W∈R C×s is the weight of the classifier, C and S are the number and representation of categories respectively. Dimension. Equal to X c Or for X v Perform an average pooling operation. The prediction y generated by linear detection pred Depth depends on representation However, whether the category label Xc It is also a visual mark X v The average pooling of is the first-order feature statistics, that is, the simple mean point (such as Figure 3 This limits its modeling capabilities.

[0086] Therefore, in order to make the prediction more accurate, the present invention performs v Model the feature distribution on to generate a stronger representation θ(X v ), which can depict the complete image of the feature. The formula of this process is:

[0087] y pred =Wθ(X v )

[0088] In practice, the function θ(Xv) is usually unknown. According to classical probability theory, θ(Xv) v ) can be obtained through its characteristic function To define, its formula is expressed as:

[0089]

[0090] where i and t are the imaginary unit and independent variable of the characteristic function, respectively. refers to the kth moment of Xv. For simplicity, It can be rewritten as:

[0091]

[0092] Where ω k is the coefficient of the k-th moment (i.e. and So far, the feature distribution θ(X v ) can be expressed by the characteristic function in, and the formula is expressed as:

[0093]

[0094] The constant term is omitted in the formula. It can be seen that based on the equation, linear detection makes predictions based on different order moments, which is better than X v The average pooling (mean point) of more reasonably describes the feature distribution. Therefore, this equation predicts more accurately. For efficiency, the present invention considers X again. v The first and second order moments (M1 and M2) of the . Then the prediction is finally achieved in the following way, the formula is expressed as:

[0095]

[0096] In the formula and is a parameter related to the classifier. Since the class label Xc can represent the whole image. To make better use of it, M1 in equation (11) is replaced by it (M1 = X c ),like Figure 3 M2 is calculated by covariance pooling, and the formula is expressed as:

[0097]

[0098] In the formula I is the N×N identity matrix, and 1 is an N×N matrix whose elements are all 1.

[0099] In addition, the present invention further uses the matrix square root to standardize ∑ (i.e., M2) to enhance its modeling capability, which is actually an element-by-element square root operation on its eigenvalues. Therefore, it is necessary to calculate the eigenvalues ​​of the covariance matrix ∑. However, the precise calculation of the eigenvalues ​​is highly dependent on eigenvalue decomposition (EIG) or singular value decomposition (SVD), which has limited support for GPU parallel processing. As an alternative, the present invention adopts a fast, approximate matrix square root normalization method, called Newton-Schulz iteration, to achieve the same purpose. The Newton-Schulz iteration only involves matrix multiplication, which naturally avoids GPU-unfriendly calculations and helps to efficiently achieve parallel computing. Since the Newton-Schulz iteration only converges locally, the covariance matrix ∑ needs to be pre-standardized by its trace, and the formula is expressed as:

[0100]

[0101] Then, given J0 = Q and P0 = I, the Newton-Schultz iteration estimates the square root J of Q by the following form, which is expressed as:

[0102]

[0103] After a small number of iterations (k=3), J k Converges roughly to the square root of ∑ Since pre-standardization is introduced before iteration, post-compensation is also required to offset its adverse effects, thereby obtaining a square root standardized covariance matrix, which is expressed as follows:

[0104]

[0105] In the formula is a symmetric matrix. Therefore, its upper triangular elements are concatenated to form the final feature representation of the patch labels, which are submitted to the subsequent classifier.

[0106] S4 is specifically:

[0107] During the training phase, we set the text labels to be the same as the corresponding image labels, or create text labels that are equal to the image labels. This approach is not feasible during the inference phase because the image labels are not actually available. Text cues as auxiliary samples are used to enhance the interaction between text and visual elements. We do this by assigning the class label X to the image encoder. c and the [EOS] tag of the text encoder T c This collaboration is achieved using a shared linear classifier that simultaneously processes the class labels X from the image encoder. c and the [EOS] tag T from the text encoder c , to make separate predictions. In this way, in the inference phase, predictions are only made for medical images, because the text information has been absorbed into the shared linear classifier in the training phase, thus simplifying the inference process.

[0108] S5 is as follows:

[0109] Our model performs well in few-shot medical image classification tasks. On the BACH dataset, the classification accuracies in 1-shot, 2-shot, 4-shot, 8-shot, and 16-shot settings are 47.5%, 55.00%, 57.78%, 64.72%, and 76.11%, respectively. On the brain tumor dataset, the accuracies in the same settings are 54.69%, 55.28%, 64.99%, 78.87%, and 84.58%, respectively. On the DR dataset, the accuracies in each setting are 66.58%, 64.38%, 68.77%, 68.28%, and 72.17%, respectively. These results show the great potential and excellent performance of our invention in few-shot problems by combining textual cues and visual features.

[0110] Embodiment 2

[0111] Embodiment 2 of the present invention provides a second-order few-sample medical image classification system based on multimodal cue learning, including:

[0112] Data acquisition module: obtains a small number of sample medical image data;

[0113] Text prompt generation module: Generates different prompt methods, including class name, general prompt, manual prompt, GPT generated description and learnable prompt;

[0114] Model building module: The model uses CLIP as the backbone network, including an image encoder and a text encoder. Medical images and category descriptions are used as visual input and text input, respectively, and features are extracted through the image encoder and text encoder.

[0115] Visual classifiers, including shared classifiers and global covariance pooling classifiers: perform visual feature classification prediction;

[0116] Training module: During the training phase, text cues are used to enhance text and visual interaction for classification prediction;

[0117] Reasoning module: In the reasoning stage, only medical images are predicted to simplify the reasoning process;

[0118] The output module is configured to: output a few-sample medical image classification result.

[0119] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0120] Embodiment 3

[0121] Embodiment 3 of the present invention provides a medium on which a program is stored. When the program is executed by a processor, the steps in the second-order few-sample medical image classification method based on multimodal prompt learning as described in Embodiment 1 of the present invention are implemented.

[0122] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0123] Embodiment 4

[0124] Embodiment 4 of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the second-order few-sample medical image classification based on multimodal prompt learning as described in Embodiment 1 of the present invention is implemented.

[0125] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0127] Table 1: Details about classname, vanilla, hand-crafted, GPT, and CoOp prompt methods

[0128]

[0129]

[0130]

Claims

1. A second-order few-shot medical image classification method with multimodal cue learning, characterized in that: Here are the steps: S1, obtain a few sample medical image data for model training; introduce text input for each image to further describe the image or concept category; S2, model construction: The model uses CLIP as the backbone network, including an image encoder and a text encoder; the medical image and category description are used as visual input and text input respectively, and the image encoder and text encoder extract features, and the image encoder generates a category label X c and visual marker X v , the text encoder generates a global sentence representation; S3, constructing a visual classifier: the classifier includes a shared classifier and a global covariance pooling classifier, the shared classifier performs category prediction based on text features and visual class tokens, and the global covariance pooling classifier realizes linear classification of feature distribution by combining first-order and second-order statistics; S4, only updates the learnable cues of CoOp and the parameters of the classifier during the training phase; only predicts new medical images during the inference phase; S5, output the medical image classification results.

2. The method of second-order few-shot medical image classification with multimodal cue learning according to claim 1, characterized in that: In the step S1, before introducing text input into each image, data enhancement processing is performed on the acquired few-sample medical image data.

3. The method of second-order few-shot medical image classification with multimodal cue learning according to claim 1, characterized in that: Before model training, a prompt method needs to be selected from the prompt pool to generate prompts for each category; then, the visual and text features are extracted using their respective encoders; The prompt pool includes five different prompt methods, including class name, general prompt, manual prompt, prompt generated by GPT, and CoOp learnable prompt method; The GPT generated prompts described above: GPT uses the Transformer architecture and is extensively pre-trained on a large text dataset to achieve coherent text generation and understanding; and uses ChatGPT, an interactive tool based on GPT 3.5, to create different categories of prompts; Specifically, two methods are used to generate these prompts: Method 1 is to confirm whether ChatGPT knows about a dataset before requesting a prompt about a category in the dataset, and then ask GPT to give a detailed description of the dataset category as a prompt; Method 2 is to use a template customized for the dataset to formulate prompts, that is, to ask GPT questions based on the template and specific categories, and ask it to give category-related content; The CoOp learning tip: Using the CoOp framework, the input formula of the text encoder is designed as: T=[V]1[V]2...[V] M [cls] Among them, each [V] m They all match the dimension of the word embedding and are shared among all categories [cls], m∈{1,2,...,M}, where M is a hyperparameter that determines the number of context tokens.

4. The method of second-order few-shot medical image classification with multimodal cue learning according to claim 1, characterized in that: In the step S2, the text encoder is used to generate a feature representation of the text description. The specific process is as follows: each tag in the text description is encoded into a unique digital ID through a byte pair encoding algorithm, and these digital IDs are mapped to 512-dimensional word embedding vectors and input into a text encoder stacked with a Transformer layer; each text description is surrounded by [SOS] and [EOS] tags, [SOS] means "start sentence", and [EOS] means "end sentence", and the feature at the [EOS] tag position is finally linearly projected as a global sentence representation; In step S2, the image encoder of CLIP can adopt ResNet or ViT architecture; when ViT is used as the image encoder, the image is divided into N non-overlapping image blocks and projected into the patch embedding P v These embeddings P v and a learnable class label P c It is input into K Transformer blocks for sequence processing; When using ResNet as an image encoder, the image goes through convolutional layers and multi-head attention pooling in sequence to combine learnable category labels with image patch visual features; The text encoder and image encoder are frozen during training; only the learnable cues and parameters of the classifiers of the involved CoOp methods are updated.

5. The method of second-order few-shot medical image classification with multimodal cue learning according to claim 1, characterized in that: The shared classifier in step S3: the shared classifier performs category prediction based on the [EOS] token from the text encoder and the category tag from the image encoder; The global covariance pooling classifier described in step S3 is used to predict the category of the visual part. The CLIP model has a category label and multiple visual labels, and these visual labels come from the output of the block before the pre-trained CLIP model classifier. The linear detection takes the average pooling of the category label or the visual label as input to generate a prediction; For the global covariance pooling classifier, it is necessary to calculate the covariance pool of visual tags for second-order statistical modeling; The second-order prediction is then added to the first-order prediction represented by a single class label to form the final prediction of the visual component; The covariance pooling of the calculated visual markers refers to replacing the original average pooling operation with a feature function, which is obtained by modeling the feature distribution of the visual markers and can depict a complete image of the features. The covariance pooling is standardized through Newton-Schultz iteration to obtain a square root standardized covariance matrix.

6. The method of second-order few-shot medical image classification with multimodal cue learning according to claim 1, characterized in that: In step S4: in the training phase, the text label is the same as the image label, and the text prompt is used as an auxiliary sample to enhance the interaction between text and vision; in the inference phase, the shared classifier makes category predictions based on the [EOS] token from the text encoder and the category label from the image encoder.

7. The method of second-order few-shot medical image classification with multimodal cue learning according to claim 1, characterized in that: The AdamW algorithm is used, combined with a warm-up phase followed by a cosine annealing learning rate schedule; the warm-up phase consists of 50 iterations and the entire training spans 12,800 iterations; a specified number of small instances are sampled from the training set using three different random seeds as training samples for the model.

8. A second-order few-shot medical image classification system with multimodal cue learning, characterized in that: include: Data acquisition module: used to obtain small sample medical image data; Text prompt generation module: used to generate text prompts, that is, to select a prompt method from the prompt pool to generate text prompts. The different prompt methods in the prompt pool include class name, common prompt, manually made prompt, prompt generated by GPT, and CoOp learnable prompt method; Model building module: The model uses CLIP as the backbone network, including an image encoder and a text encoder. Medical images and category descriptions are used as visual input and text input, respectively, and features are extracted through the image encoder and text encoder. Visual classifiers, including shared classifiers and global covariance pooling classifiers: used to make classification predictions of visual features; Training module: During the training phase, text prompts are used as auxiliary samples to enhance the interaction between text and vision and perform classification predictions; Reasoning module: In the reasoning stage, only predictions are made for medical images to simplify the reasoning process; Output module: Outputs the few-sample medical image classification results.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the second-order few-sample medical image classification method of multimodal prompt learning as described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the second-order few-sample medical image classification method of multimodal prompt learning as described in any one of claims 1-7 are implemented.

Images