A second-order few-shot medical image classification method, system, storage medium and equipment of multi-modal prompt learning
By combining multimodal cue learning and second-order pooling techniques with CLIP models and CoOp learnable cueing, the problem of insufficient accuracy and generalization ability in medical image classification under data scarcity is solved, and efficient classification is achieved in cases with few samples.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- DALIAN UNIV OF TECH
- Filing Date
- 2024-08-13
- Publication Date
- 2026-05-12
AI Technical Summary
Existing medical image classification methods lack accuracy and generalization ability when data is scarce, traditional supervised learning methods face challenges, and the application of vision-language pre-trained models in medical image analysis is limited.
A second-order few-shot medical image classification method employing multimodal cue learning is proposed. By introducing text input and second-order pooling techniques, combined with CLIP model and CoOp learnable cueing, the model's ability to understand and classify medical images is enhanced.
It improves the accuracy and generalization ability of medical image classification, especially performing well in cases with few samples, and significantly improves classification accuracy.
Smart Images
Figure CN119091197B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical image processing technology, specifically to a second-order few-shot medical image classification method, system, storage medium, and device based on multimodal cue learning. Background Technology
[0002] Multimodal recognition technology integrates data from multiple modalities, including text, images, and sound, enhancing the performance of tasks such as image recognition. The visual-language pre-trained model CLIP expands its applicability by constructing a multimodal framework and learning transferable visual models from natural language supervision. By introducing learnable text cue vectors, it achieves lossless tuning of the original model, adapting to diverse dataset challenges.
[0003] Medical image classification methods rely on recent advances in deep learning. With the development of deep learning technology, especially the introduction of Convolutional Neural Networks (CNNs) and Transformers, the accuracy of medical image classification has significantly improved. However, due to the scarcity of medical image data and the difficulty of annotation, traditional supervised learning methods face challenges. Few-shot learning, by training high-performance models with limited sample data, provides a solution for medical image classification in situations of data scarcity.
[0004] Visual-language pre-trained models perform multimodal learning through the correspondence between images and text, providing excellent zero-shot and few-shot performance. CLIP models, through large-scale pre-training, achieve alignment between images and text. However, the application of these models in medical image analysis is limited. Developing visual-language models specifically for medical images is of great significance. This patent combines visual-language models with medical image classification, improving the accuracy and generalization ability of medical image classification by introducing multimodal learning and cue learning techniques.
[0005] Cue learning guides models to adapt to new tasks by designing or learning cues. In medical image classification, cue learning helps models better understand and identify targets in medical images. This patent explores various cueing methods, including manually designed class names, general methods, and handcrafted methods; using GPT to generate descriptions related to data categories as cues to provide additional information; and using learnable cues to improve classification performance.
[0006] Furthermore, CLIP uses only first-order information as the overall representation, ignoring potential higher-order correlations between features. Second-order pooling captures global information by calculating the covariance matrix between feature maps. In medical image classification, second-order pooling helps the model utilize higher-order statistical information, improving classification accuracy. This patent applies second-order pooling to medical image classification, combining it with multimodal learning and cue learning techniques to further improve model performance. It explores the characteristics and applicability of different second-order pooling methods, providing new ideas for large-scale vision-language pre-trained models in medical image analysis tasks. Summary of the Invention
[0007] To address the shortcomings of existing technologies, this invention proposes a second-order few-shot medical image classification method based on multimodal cue learning. By introducing text input to further describe the image or concept category, it promotes few-shot learning of medical images and improves the accuracy and generalization ability of medical image classification.
[0008] To achieve the above objectives, the present invention adopts the following technical solution:
[0009] A second-order few-shot medical image classification method based on multimodal cue learning, comprising the following steps:
[0010] S1: Obtain a small sample of medical image data for model training. Introduce text input for each image to further describe the image or concept category.
[0011] Furthermore, in step S1, before introducing text input into each image, data augmentation processing is performed on the acquired small sample medical image data.
[0012] S2, Model Construction: The model uses CLIP as the backbone network, containing an image encoder and a text encoder. Medical images and category descriptions are taken as visual input and text input, respectively. After feature extraction by the image encoder and text encoder, the image encoder generates the category label X. c and visual marker X v The text encoder generates a global sentence representation;
[0013] Furthermore, before training the model, a prompting method needs to be selected from the prompt pool to generate prompts for each category. Then, visual and textual features are extracted using their respective encoders.
[0014] Furthermore, the suggestion pool includes five different suggestion methods, including class name suggestions, normal suggestions, manual suggestions, suggestions generated by GPT, and CoOp learnable suggestion methods.
[0015] The GPT-generated hints are as follows: GPT employs the Transformer architecture and has been extensively pre-trained on a large text dataset to achieve coherent text generation and understanding. Furthermore, ChatGPT, an interactive tool based on GPT 3.5, is used to create hints for different categories. Specifically, two methods are employed to generate these hints: Method one: Before requesting hints about categories within the dataset, it is confirmed whether ChatGPT is familiar with the dataset, and then GPT provides a detailed description of the dataset's categories as a hint. Method two: A template customized for the dataset is used to formulate hints; that is, by combining the template with specific categories, questions are asked to GPT, prompting it to provide category-related content.
[0016] The learnable hints from CoOp: Using the CoOp framework, the input formula for a text encoder can be expressed as follows:
[0017] T = [V]1[V]2...[V] M [cls]
[0018] Among them, each [V] m All of them match the dimensions of the word embedding and are shared across all categories [cls], m∈{1,2,...,M}, where M is a hyperparameter that determines the number of context tags.
[0019] Further, in step S2, the text encoder is used to generate feature representations of the text description. Specifically, each tag in the text description is encoded into a unique numeric ID using a byte-pair encoding algorithm. These numeric IDs are mapped to 512-dimensional word embedding vectors and input into a text encoder with stacked Transformer layers. Each text description is surrounded by [SOS] and [EOS] tags, where [SOS] represents the "start sentence" and [EOS] represents the "end sentence." Finally, the features located at the [EOS] tag positions are linearly projected as a global sentence representation.
[0020] Furthermore, in step S2, the CLIP image encoder can employ a ResNet or ViT architecture. When using ViT as the image encoder, the image is segmented into N non-overlapping image patches and projected onto a patch embedding P. v In, these embedded P v and learnable category labels P c The images are fed into K Transformer blocks for sequential processing. When ResNet is used as the image encoder, the images pass through convolutional layers and multi-head attention pooling in sequence, combining learnable class labels with visual features of image patches.
[0021] Furthermore, to improve efficiency, the text encoder and image encoder are frozen during training; only the learnable hints for the CoOp method involved and the parameters of the classifier are updated.
[0022] S3, Construct a visual classifier: The classifier includes a shared classifier and a global covariance pooling classifier. The shared classifier predicts the category based on text features and visual class tokens. The global covariance pooling classifier achieves linear classification of feature distribution by combining first-order and second-order statistics.
[0023] The shared classifier described in step S3: The shared classifier performs category prediction based on the [EOS] token from the text encoder and the category tag from the image encoder.
[0024] The global covariance pooling classifier described in step S3 is used for category prediction of the visual component. The CLIP model has one class label and multiple visual labels, which are derived from the output of a block preceding the pre-trained CLIP model classifier. Linear probing takes the average pooling of the class label or visual labels as input to generate predictions. For the global covariance pooling classifier, the covariance pooling of the visual labels needs to be calculated for second-order statistical modeling. Then, the second-order prediction is added to the first-order prediction represented by a single class label to form the final prediction for the visual component.
[0025] Furthermore, the covariance pooling of the visual markers refers to replacing the original average pooling operation with a feature function. This feature function is obtained by modeling the feature distribution of the visual markers, which can depict a complete image of the features. The covariance pooling is standardized through Newton-Schultz iteration to obtain a square root standardized covariance matrix.
[0026] S4 updates only the learnable hints for CoOp and the parameters of the classifier during the training phase. During the inference phase, it only predicts new medical images.
[0027] During the training phase, the text labels are the same as the image labels, and the text prompts serve as auxiliary samples to enhance the interaction between text and vision.
[0028] During the inference phase, the shared classifier makes category predictions based on the [EOS] token from the text encoder and the category tag from the image encoder.
[0029] S5 outputs the medical image classification results.
[0030] As a further implementation, to optimize the network, specifically, the AdamW algorithm is used, combined with a warm-up phase, followed by a cosine annealing learning rate schedule. The warm-up phase consists of 50 iterations, and the entire training spans 12,800 iterations.
[0031] As a further implementation, it is important to emphasize the rigor of the invention, specifically by using three different random seeds to sample a specified number of small instances from the training set as training samples for the model.
[0032] A second aspect of the present invention provides a few-shot medical image classification system based on multimodal cue learning, comprising:
[0033] Data acquisition module: used to acquire small sample medical image data;
[0034] Text hint generation module: Used to generate text hints, that is, to select a hint method from the hint pool to generate text hints. Different hint methods in the hint pool include class name, normal hints, manually created hints, hints generated by GPT, and CoOp learnable hint methods;
[0035] Model building module: The model uses CLIP as the backbone network, which includes an image encoder and a text encoder. Medical images and category descriptions are used as visual input and text input, respectively, and features are extracted by the image encoder and text encoder.
[0036] Visual classifiers, including shared classifiers and global covariance pooling classifiers, are used for classification and prediction of visual features.
[0037] Training module: During the training phase, text prompts are used as auxiliary samples to enhance the interaction between text and vision for classification prediction;
[0038] Inference module: During the inference phase, predictions are made only for medical images, simplifying the inference process;
[0039] Output module: Outputs the classification results of medical images with a small number of samples.
[0040] A third aspect of the present invention provides a medium having a program stored thereon, which, when executed by a processor, implements the steps of the second-order few-shot medical image classification method based on multimodal cue learning described in the first aspect of the present invention.
[0041] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the second-order few-shot medical image classification method based on multimodal cue learning described in the first aspect of the present invention.
[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0043] The proposed second-order few-shot medical image classification method based on multimodal cue learning enhances the model's understanding and classification capabilities by introducing auxiliary text input and multimodal learning methods. Furthermore, by utilizing covariance pooling and feature distribution modeling techniques, the model's prediction accuracy and generalization ability are further improved, resulting in outstanding performance in medical image classification tasks. Attached Figure Description
[0044] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0045] Figure 1 This is a flowchart of the second-order few-shot medical image classification method based on multimodal cue learning according to the present invention;
[0046] Figure 2 This is a framework diagram of the second-order few-shot medical image classification method based on multimodal cue learning of this invention;
[0047] Figure 3 This is a framework diagram of the visual classifier of the present invention. Detailed Implementation
[0048] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0049] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0050] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.
[0051] Example 1
[0052] like Figure 1 As shown, Embodiment 1 of the present invention provides a flowchart of a second-order few-shot medical image classification method based on multimodal cue learning, including the following steps:
[0053] S1. Acquire a small number of medical image data for model training, introduce auxiliary text input for each image to further describe the image or concept category, promote cross-modal learning, and help understand the medical image data.
[0054] S2. Construct the model, which uses the CLIP model as the backbone network. This model contains an image encoder and a text encoder.
[0055] S3. Use a shared classifier to predict categories based on text features and visual class tokens. Construct a global covariance pooling classifier that achieves linear classification of feature distributions by combining first-order and second-order statistics.
[0056] S4. During the training phase, only the learnable hints for CoOp and the parameters of the classifier are updated. During the inference phase, only new medical images are predicted.
[0057] S5. Output medical image classification results.
[0058] Specifically, S1 is:
[0059] This invention utilizes three publicly available medical image datasets and introduces detailed auxiliary text prompts for each image to further describe the image or concept category, thereby facilitating cross-modal learning.
[0060] The BACH (Breast Cancer Histology) dataset contains 400 histopathological images from breast cancer biopsies, categorized into four groups: normal, benign, carcinoma in situ, and invasive carcinoma, with 100 images in each group. Each image is accompanied by a detailed text description, such as "This is a histopathological image from a breast cancer biopsy showing normal tissue" or "This is a histopathological image showing a benign tumor."
[0061] The Figshare MRI brain tumor dataset contains 3064 T1-weighted contrast-enhanced images from 233 patients, categorized into meningiomas (708), gliomas (1426), and pituitary tumors (930 images). Detailed text descriptions are added to each image, such as "This is a T1-weighted contrast-enhanced image showing a meningioma" or "This is an MRI image showing a glioma."
[0062] The Diabetic Retinopathy (DR) dataset contains 3,662 high-resolution color retinal images, categorized by severity into five types: non-proliferative, mild, moderate, severe, and proliferative diabetic retinopathy. Descriptive text is added to each image, such as "This is a high-resolution color image showing moderate non-proliferative diabetic retinopathy" or "This is an image showing mild non-proliferative diabetic retinopathy."
[0063] When using the above dataset, this invention divides them into training and test sets with a 7:3 split ratio.
[0064] This invention explores five different suggestion methods based on existing research. These methods are classname suggestions, vanilla suggestions, hand-crafted suggestions, category-specific descriptions generated by GPT, and a learnable CoOp method.
[0065] Using the BACH dataset as an example, these methods are described in detail in Table 1. The first three rows of the table summarize the specifics of the first three methods. Next, the methods of generating cues using GPT and the learnable CoOp approach are explained in detail.
[0066] GPT Hint Generation: GPT employs the Transformer architecture and is extensively pre-trained on massive text datasets to achieve coherent text generation and understanding, making it highly valuable in various NLP tasks. In this invention, the interactive tool ChatGPT, based on GPT 3.5, is used to create hints for different categories. Two methods are employed to generate these hints. First, before requesting hints for categories in the dataset, it is confirmed whether ChatGPT understands the specific dataset, as shown in the fifth row of the table. Second, a structured template is used to formulate the hints.
[0067] CoOp's Learnable Hints: The CoOp method innovatively solves the problem of excessive hint engineering time by customizing a CLIP-like visual-language model for image recognition tasks and using learnable vectors as placeholders for contextual words in the hints. In this invention, following the CoOp framework, the input formula for the text encoder is designed as follows:
[0068] T = [V]1[V]2...[V] M [cls]
[0069] In this setting, each [V] m (where m∈{1,2,...,M}) all match the dimensions of the word embeddings (typically 1024 or 512 for CLIP) and are shared across all categories [cls]. M is a hyperparameter that determines the number of context tokens. This invention, based on findings from CoOp evaluation, employs a consistent contextual approach throughout the process.
[0070] S2 specifically refers to:
[0071] Preprocessed small-sample medical image data and their corresponding category prompts are input into a pre-trained multimodal model for feature extraction. The basic model structure consists of a CLIP model; CLIP is a multimodal structure with two input branches: text and image; the encoding process of the two encoders is as follows.
[0072] The text encoding process is as follows:
[0073] The CLIP text encoder aims to generate feature representations of text descriptions T. First, each token in the text description is encoded into a unique numeric ID using a byte-pair encoding (BPE) algorithm. These numeric IDs are then mapped to 512-dimensional word embedding vectors, which are fed into a text encoder L with stacked transformer layers. Note that before token embedding, each text description is surrounded by [SOS] and [EOS] tokens. [SOS] represents the “Start Of Sentence”, and [EOS] represents the “End Of Sentence”. Finally, the features located at the [EOS] token positions are used as a global sentence representation, followed by a linear projection (Projtxt), whose formula is:
[0074] [SOS,...,EOS]=L(T)
[0075] T c =Proj txt (T idx ([EOS]))
[0076] Among them, T idx ([EOS]) represents the index of the [EOS] token, while T c It is a global sentence representation, Proj txt (·) indicates a linear projection.
[0077] The image encoding process is as follows:
[0078] CLIP's image encoder can employ architectures such as ResNet or ViT. Let V be the visual encoder. When using ViT as V, the image is first segmented into N non-overlapping image patches and projected onto the patch embedding. In the middle. Meanwhile, these embedded P v And a learnable category label P c Together, they are fed into K transformer blocks (e.g., K=12 for ViT-B) for sequence processing. When using ResNet as V, image I first passes through convolutional layers, and then at the end of the network, multi-head attention pooling combines learnable class labels with image patch (visual) features. Therefore, the image encoding process in CLIP can be expressed as:
[0079] [X c ,X v ] = V(I)
[0080] According to this design, regardless of the visual encoder chosen, two types of tags will ultimately be generated: category tags X. c and visual marker Xv .
[0081] S3 specifically refers to:
[0082] The classifiers include a shared classifier and a global covariance pooling classifier. The shared classifier predicts categories based on text features and visual class tokens, while the global covariance pooling classifier achieves linear classification of feature distributions by combining first-order and second-order statistics. The shared classifier predicts categories based on [EOS] tokens from the text encoder and category labels from the image encoder. The global covariance pooling classifier uses X... c Xv is taken as input. At the top of the image encoder, this classifier primarily captures the second-order statistics (i.e., covariance) inherent in the visual labels and makes predictions based on these statistics, which have been neglected in previous work. Furthermore, the classifier also considers the class label, which is the first-order statistic of the global image representation (the class label itself). Thus, such a visual classifier is achieved by simultaneously considering the first and second moments of the features.
[0083] Figure 3 This is a framework diagram of the visual classifier of this invention. In the CLIP model, there is one class token and multiple visual tokens. Given a class token X... c ∈R d And a set X containing N d-dimensional visual markers. v ∈R d×N These visual labels come from the output of a block preceding the pre-trained CLIP model classifier. Linear probing typically assigns the class label X... c Or visual marker X v The mean pooling is used as input, and predictions are generated as follows, expressed by the formula:
[0084]
[0085] In the formula, W is the weight of the linear layer. It is some kind of transformation of the input data (in this case, it might be an identity transformation, since there is already an appropriate representation), X is X c or X v W∈R C×s These are the weights of the classifier, where C and S are the number and representation of the classes, respectively. Dimensions. equals X c Or for X v Perform average pooling. Linear probing generates the predicted y. pred Deeply dependent on representation However, regardless of the category label Xc It is also a visual marker X v Average pooling always uses first-order feature statistics, i.e., simple mean points (such as...). Figure 3 (As shown in the upper part), this limits its modeling capabilities.
[0086] Therefore, in order to make the prediction more accurate, this invention addresses X. v Modeling the feature distribution on the surface to generate a stronger representation θ(X) v This indicates that a complete image of the features can be depicted. The process is expressed by the following formula:
[0087] y pred =Wθ(X) v )
[0088] In practice, the function θ(Xv) is usually unknown. According to classical probability theory, θ(Xv) v ) can be obtained through its characteristic function To define it, its formula is expressed as:
[0089]
[0090] Where i and t are the imaginary unit and independent variable of the characteristic function, respectively. This refers to the k-th moment of Xv. For simplicity, The formula can be rewritten as follows:
[0091]
[0092] In the formula ω k The coefficients of the k-th moment (i.e. and So far, the characteristic distribution θ(X) v The characteristic function in the formula can be used to represent the characteristic function, which is expressed as:
[0093]
[0094] The constant term is omitted in the formula. It can be seen that, based on the equation, linear detection predicts based on different moments, which is better than X. v Average pooling (mean point) more reasonably describes the feature distribution. Therefore, the equation predicts more accurately. For efficiency reasons, this invention again considers X. v The first and second moments (M1 and M2) are then used. The final prediction is achieved through the following formula:
[0095]
[0096] In the formula and These are parameters related to the classifier. Due to the category label X...c It can represent the whole image. To make better use of it, M1 in equation (11) is replaced with it (M1 = X). c ),like Figure 3 (As shown in the lower half). M2 is calculated using covariance pooling, and the formula is as follows:
[0097]
[0098] In the formula I is an N×N identity matrix, and 1 is an N×N matrix in which all elements are 1.
[0099] Furthermore, this invention uses the square root of the matrix to normalize ∑ (i.e., M2) to enhance its modeling capabilities, which is essentially performing an element-wise square root operation on its eigenvalues. Therefore, it is necessary to compute the eigenvalues of the covariance matrix ∑. However, the accurate computation of eigenvalues heavily relies on eigenvalue decomposition (EIG) or singular value decomposition (SVD), which has limited support for GPU parallel processing. As an alternative, this invention employs a fast, approximate matrix square root normalization method called the Newton-Schulz iteration to achieve the same purpose. The Newton-Schulz iteration involves only matrix multiplication, naturally avoiding GPU-unfriendly computations and facilitating efficient parallel computing. Since the Newton-Schulz iteration only converges locally, the covariance matrix ∑ needs to be pre-normalized using its trace, expressed as:
[0100]
[0101] Then, given J0 = Q and P0 = I, the Newton-Schultz iteration estimates the square root J of Q in the following form, expressed as:
[0102]
[0103] After a few iterations (k=3), J k It roughly converges to the square root of ∑ Since pre-standardization is introduced before iteration, post-compensation is also needed to offset its adverse effects, thus obtaining the square root-standardized covariance matrix, which is expressed by the formula:
[0104]
[0105] In the formula It is a symmetric matrix. Therefore, its upper triangular elements are concatenated to form the final feature representation of the patch labels, which are then submitted to the subsequent classifier.
[0106] S4 specifically refers to:
[0107] During training, text labels are either set to be the same as the corresponding image labels, or text labels are created to be equal to the image labels. This approach is not feasible during inference because image labels are practically unavailable. Text cues, used as auxiliary samples, are employed to enhance the interaction between text and visual elements. We use class labels X from the image encoder to... c and the [EOS] tag T of the text encoder c This collaboration is achieved using a shared linear classifier that simultaneously processes class labels X from the image encoder. c and the [EOS] tag T from the text encoder c This allows for separate predictions, where predictions are made only for medical images during the inference phase, since textual information has already been incorporated into a shared linear classifier during the training phase, thus simplifying the inference process.
[0108] S5 specifically refers to:
[0109] Our model demonstrates superior performance in few-shot medical image classification tasks. On the BACH dataset, the classification accuracies are 47.5%, 55.00%, 57.78%, 64.72%, and 76.11% in 1-shot, 2-shot, 4-shot, 8-shot, and 16-shot settings, respectively. On the brain tumor dataset, the accuracies are 54.69%, 55.28%, 64.99%, 78.87%, and 84.58% in the same settings. On the DR dataset, the accuracies are 66.58%, 64.38%, 68.77%, 68.28%, and 72.17% in each setting. These results demonstrate the significant potential and superior performance of our invention in few-shot problems by combining textual cues and visual features.
[0110] Example 2
[0111] Embodiment 2 of the present invention provides a second-order few-shot medical image classification system based on multimodal cue learning, comprising:
[0112] Data acquisition module: Acquires small sample medical image data;
[0113] Text hint generation module: Generates different hint methods, including class names, normal hints, manual hints, GPT-generated descriptions, and learnable hints;
[0114] Model building module: The model uses CLIP as the backbone network, which includes an image encoder and a text encoder. Medical images and category descriptions are used as visual input and text input, respectively. Features are extracted by the image encoder and text encoder.
[0115] Visual classifiers, including shared classifiers and global covariance pooling classifiers, are used to classify and predict visual features.
[0116] Training module: During the training phase, text prompts are used to enhance text and visual interaction for classification prediction;
[0117] Inference module: During the inference phase, predictions are made only for medical images, simplifying the inference process;
[0118] The output module is configured to output the classification results of medical images with few samples.
[0119] The more detailed steps are the same as in Example 1, and will not be repeated here.
[0120] Example 3
[0121] Embodiment 3 of the present invention provides a medium on which a program is stored. When the program is executed by a processor, it implements the steps in the second-order few-shot medical image classification method based on multimodal cue learning as described in Embodiment 1 of the present invention.
[0122] The more detailed steps are the same as in Example 1, and will not be repeated here.
[0123] Example 4
[0124] Embodiment 4 of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements second-order few-shot medical image classification based on multimodal cue learning as described in Embodiment 1 of the present invention.
[0125] The more detailed steps are the same as in Example 1, and will not be repeated here.
[0126] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
[0127] Table 1: Detailed information on class name, vanilla hints, hand-crafted hints, GPT, and CoOp prompt methods.
[0128]
[0129]
[0130]
Claims
1. A second-order few-shot medical image classification method based on multimodal cue learning, characterized in that, The steps are as follows: S1: Obtain a small sample of medical image data for model training; introduce text input for each image to further describe the image or concept category; S2, Model Construction: The model uses CLIP as the backbone network, containing an image encoder and a text encoder; medical images and category descriptions are used as visual input and text input, respectively. After feature extraction by the image encoder and text encoder, the image encoder generates category labels. and visual markers The text encoder generates a global sentence representation; S3, Construct a visual classifier: The classifier includes a shared classifier and a global covariance pooling classifier. The shared classifier predicts the category based on text features and visual class tokens. The global covariance pooling classifier achieves linear classification of feature distribution by combining first-order and second-order statistics. The shared classifier described in step S3: The shared classifier predicts the category based on the [EOS] token from the text encoder and the category label from the image encoder; The global covariance pooling classifier described in step S3 is used for class prediction of the visual part. The CLIP model has one class label and multiple visual labels, which are derived from the output of the block before the pre-trained CLIP model classifier. Linear probing takes the average pooling of the class label or visual labels as input to generate predictions. For global covariance pooling classifiers, it is necessary to calculate the covariance pool of visual labels in order to perform second-order statistical modeling. The second-order prediction is then added to the first-order prediction, which is represented by a single category label, to form the final prediction for the visual component; The covariance pooling of the computational visual markers refers to replacing the original average pooling operation with a feature function. This feature function is obtained by modeling the feature distribution of the visual markers, which can depict a complete image of the features. Furthermore, the covariance pooling is standardized through Newton-Schultz iteration to obtain a square root standardized covariance matrix. S4 updates only the learnable hints for CoOp and the parameters of the classifier during the training phase; and only predicts new medical images during the inference phase. S5 outputs the medical image classification results.
2. The second-order few-shot medical image classification method based on multimodal cue learning according to claim 1, characterized in that, In step S1, before introducing text input into each image, data augmentation processing is performed on the acquired small sample medical image data.
3. The second-order few-shot medical image classification method based on multimodal cue learning according to claim 1, characterized in that, Before training the model, a cue method needs to be selected from the cue pool to generate cue for each category; then, visual and textual features are extracted using their respective encoders. The aforementioned suggestion pool includes five different suggestion methods, including class name, normal suggestions, manual suggestions, suggestions generated by GPT, and CoOp learnable suggestion methods; The GPT-generated hints are as follows: GPT employs the Transformer architecture and has been extensively pre-trained on a large text dataset to achieve coherent text generation and understanding; furthermore, it utilizes interactive tools based on GPT 3.
5. To create different categories of tips; Specifically, two methods are used to generate these hints: Method one is to first confirm the categories in the dataset before requesting hints. One approach is to ask GPT for a detailed description of the dataset's categories as a hint, based on whether the dataset is familiar to the user. Another approach is to use a template tailored to the dataset to create a hint, that is, to ask GPT questions based on the template and specific categories, prompting it to provide category-related information. The learnable hints from CoOp: Using the CoOp framework, the input formula for a text encoder can be expressed as follows: Each of them All match the dimensions of word embeddings, and in all categories Shared between , It is a hyperparameter that determines the number of context tags.
4. The second-order few-shot medical image classification method based on multimodal cue learning according to claim 1, characterized in that, In step S2, the text encoder is used to generate feature representations of the text description. Specifically, each tag in the text description is encoded into a unique numeric ID using a byte-pair encoding algorithm. These numeric IDs are mapped to 512-dimensional word embedding vectors and input into a text encoder with stacked Transformer layers. Each text description is surrounded by [SOS] and [EOS] tags. Indicates the beginning of a sentence. The feature at the [EOS] marker position, representing the end of the sentence, is linearly projected to become the global sentence representation. In step S2, the CLIP image encoder can employ a ResNet or ViT architecture; when using When used as an image encoder, the image is segmented into N non-overlapping image patches and projected onto the patch embedding. In, these embedded and learnable category tags Input to Sequence processing is performed within a Transformer block; When ResNet is used as an image encoder, the image passes through convolutional layers and multi-head attention pooling in sequence, combining learnable class labels with visual features of image patches; The text encoder and image encoder are frozen during training; only the learnable hints for the CoOp methods involved and the parameters of the classifier are updated.
5. The second-order few-shot medical image classification method based on multimodal cue learning according to claim 1, characterized in that, In step S4: During the training phase, the text labels are the same as the image labels, and the text cues serve as auxiliary samples to enhance the interaction between text and vision; during the inference phase, the shared classifier makes category predictions based on the [EOS] token from the text encoder and the category label from the image encoder.
6. The second-order few-shot medical image classification method based on multimodal cue learning according to claim 1, characterized in that, The AdamW algorithm is used, combined with a warm-up phase followed by a cosine annealing learning rate schedule; the warm-up phase consists of 50 iterations, and the entire training spans 12,800 iterations; a specified number of small instances are sampled from the training set using three different random seeds as training samples for the model.
7. A second-order few-shot medical image classification system based on multimodal cue learning, characterized in that, include: Data acquisition module: used to acquire small sample medical image data; Text hint generation module: Used to generate text hints, that is, to select a hint method from the hint pool to generate text hints. Different hint methods in the hint pool include class name, normal hints, manually created hints, hints generated by GPT, and CoOp learnable hint methods; Model building module: The model uses CLIP as the backbone network, which includes an image encoder and a text encoder; medical images and category descriptions are used as visual input and text input, respectively, and features are extracted by the image encoder and text encoder. Visual classifiers, including shared classifiers and global covariance pooling classifiers, are used for classification and prediction of visual features. Training module: During the training phase, text prompts are used as auxiliary samples to enhance the interaction between text and vision for classification prediction; Inference module: During the inference phase, predictions are made only for medical images, simplifying the inference process; Output module: Outputs classification results for medical images with few samples; The shared classifier described in step S3: The shared classifier predicts the category based on the [EOS] token from the text encoder and the category label from the image encoder; The global covariance pooling classifier described in step S3 is used for class prediction of the visual part. The CLIP model has one class label and multiple visual labels, which are derived from the output of the block before the pre-trained CLIP model classifier. Linear probing takes the average pooling of the class label or visual labels as input to generate predictions. For global covariance pooling classifiers, it is necessary to calculate the covariance pool of visual labels in order to perform second-order statistical modeling. The second-order prediction is then added to the first-order prediction, which is represented by a single category label, to form the final prediction for the visual component; The covariance pooling of the computational visual markers refers to replacing the original average pooling operation with a feature function. This feature function is obtained by modeling the feature distribution of the visual markers, which can depict a complete image of the features. Furthermore, the covariance pooling is standardized through Newton-Schultz iteration to obtain a square root standardized covariance matrix.
8. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by a processor, the program implements the steps in the second-order few-shot medical image classification method based on multimodal cue learning as described in any one of claims 1-6.
9. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the second-order few-shot medical image classification method with multimodal cue learning as described in any one of claims 1-6.