Disease classification method and system based on multilevel prompt learning

Abstract

The invention discloses a disease classification method and system based on multilevel prompt learning, and belongs to the technical field. For the problem that a traditional meta-learning method has key defects in a small sample skin disease classification scene, complementary semantic clues are provided by using multi-level image features and text information, and the accuracy of small sample disease classification is improved. Meanwhile, through a learning prompt mode, base class knowledge and new class fine tuning are fused, the generalization ability of the model is enhanced, and an effective solution is provided for clinical diagnosis. Compared with a traditional method, the disease classification method based on multi-level prompt learning provided by the invention can better utilize limited data and improve the robustness of a model, can realize fusion of cross-modal information and enhance the interpretability of a diagnosis result, provides an effective solution for solving the problem of small samples, and has a good application prospect. The method has a remarkable application prospect in the fields of skin disease diagnosis and the like.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and digital medical technology, and specifically relates to a disease classification method and system based on multi-level prompt learning. Background Art

[0002] With the widespread application of machine learning technology in the field of computer vision, it has made significant progress in tasks such as image classification and object detection. However, these advances mainly rely on large-scale annotated datasets, such as ImageNet and COCO. However, in the field of medical classification, especially skin disease classification, it is difficult to obtain such a large-scale annotated dataset due to the limited number of patients and the high cost of data collection and annotation. This leads to the long-tail characteristic of data distribution, that is, most skin disease categories contain only a small number of samples, and traditional machine learning models are difficult to effectively model and generalize.

[0003] To solve this small sample problem, researchers have proposed methods based on data augmentation and transfer learning. Data augmentation technology is to artificially synthesize more training samples by performing a series of transformations on the original training samples, such as rotation, flipping, scaling, adding noise, etc., such as image-level contrastive learning, generative adversarial networks, etc., to generate more realistic and effective synthetic samples. For example, adversarial data augmentation uses generative adversarial networks (GANs) to generate high-quality synthetic images. These samples can not only increase the size of the training set, but also better cover the distribution of the original data. This method can alleviate the small sample problem to a certain extent and increase the diversity and size of the training set. However, the key to data augmentation is whether the generated synthetic samples can truly reflect the original data distribution, thereby contributing to the generalization performance of the model. If the quality of the generated samples is poor, it may introduce noise and reduce the performance of the model. Transfer learning, on the other hand, uses the model parameters pre-trained on related tasks and fine-tunes the parameters on the target task to improve the efficiency of sample utilization. The key to this method is the relevance and generalization ability of the pre-trained model. If the pre-training task is highly relevant to the target task and the pre-trained model has good generalization performance, then transfer learning can significantly improve the performance of the target task. However, if the pre-training task is very different from the target task, or the pre-training model itself has problems such as overfitting, the effect of transfer learning may be greatly reduced, so it is not always feasible in practical applications.

[0004] Meta-learning methods have shown significant advantages in the task of small sample disease classification. Traditional supervised learning methods require a large amount of labeled data to train high-performance models, but in fields such as medical imaging, it is usually very difficult to obtain a large amount of labeled data. Common methods in the medical field (such as CN116797848A) lack sufficient consideration of the fine-grained alignment relationship and multi-level feature information between medical images and texts, which limits their ability to capture complex disease characteristics and micro-lesion characteristics. They are not effective in overall diagnostic effects and feature utilization in small sample cases, which may lead to an increased risk of misdiagnosis or missed diagnosis. Therefore, meta-learning methods have greater application potential in this field. In contrast, meta-learning can make more effective use of limited training data by pre-training on common categories (base classes) and then quickly adapting using a small number of target class samples.

[0005] Traditional meta-learning methods mainly include representative algorithms such as the Model-Agnostic Meta-Learning (MAML) algorithm, the Memory-Augmented Neural Networks (MANN) algorithm, and the Relation Network algorithm. Specifically, the MAML algorithm attempts to learn a universal parameter initialization so that it can quickly converge to a good model when fine-tuning on a small number of target task samples. However, MAML is better at capturing the global semantic features of images, but its performance may be limited for skin disease classification tasks that require fine-grained features. At the same time, MAML has poor interpretability and it is difficult to explain its internal working mechanism. The MANN algorithm introduces an external memory module that can store and quickly retrieve previously learned knowledge, thereby helping the model to quickly adapt to new tasks. Although MANN has advantages in small sample learning tasks, its memory module may be difficult to effectively capture the subtle lesion features required for skin disease classification. The relational network algorithm quickly determines the category of new samples by learning the correlation between samples, and belongs to a metric-based meta-learning method. The relational network mainly focuses on the overall similarity between samples and has difficulty in modeling local detail features, which may be limited in the skin disease classification task that needs to capture the characteristics of small lesions. At the same time, its generalization ability may also be limited.

[0006] In summary, traditional meta-learning methods such as MAML, MANN, and relational networks have some key defects in small-sample skin disease classification scenarios. First, these algorithms are better at capturing the global semantic features of images, and their performance may be limited for skin disease classification tasks that need to capture the characteristics of subtle lesions. Secondly, although these methods have advantages in small-sample learning tasks, their generalization ability may be affected if the target task is significantly different from the base class. In addition, these "black box" models often lack interpretability and are difficult to provide reliable support for clinical classification. Therefore, how to design a more fine-grained feature extraction mechanism, improve the ability of the meta-learning model in modeling subtle features, and take into account the generalization and interpretability of the model, is a key issue in improving the accuracy of medical image disease classification. Summary of the invention

[0007] In view of the problem that traditional meta-learning methods have key defects in small sample skin disease classification scenarios, the present invention provides a disease classification method and system based on multi-level prompt learning. First, multi-level features are extracted from the base class image, the base class clustering centers of different levels are learned, and a multi-level prompt template containing learnable prompts is constructed. Then, using a small number of new class samples, the model is fine-tuned through prompt learning, and the base class knowledge is integrated to improve the generalization ability. Finally, the method can use the trained multi-level prompt learning model to extract more comprehensive disease representations and realize small sample disease classification. This method effectively solves the problem of small sample learning and provides an effective solution for clinical diagnosis applications.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] Step 1: Preprocess the base class image and text datasets. The images are disease images of a certain modality, and the texts are the category labels of the images. Design a hierarchical prototype training model to extract cluster centers of different levels in the base class image data, and train an image encoder to learn the discriminative features of the base class images.

[0010] Step 1.1: Preprocess the base class text dataset; add text descriptions to each category label, starting from the definition, when a hierarchical definition is formed, extract the corresponding text description as the category description; when a hierarchical definition is not formed, truncate the definition as the category description;

[0011] Step 1.2: Preprocess the base class image dataset; design a hierarchical prototype training model to extract the base class cluster centers at different levels in the base class image data, and train an image encoder that can learn the discriminative features of the image, and extract the output features of the base class images in turn as the multi-level base class image feature representation from low level to high level of the base class images;

[0012] Step 1.3: Joint iterative training to obtain image encoders and cluster centers at different levels; based on the extracted multi-level base class image feature representation, at each level, first cluster and initialize K cluster centers q i , (i=1, 2...K), use Student's t distribution to measure the similarity between the sample and the cluster center, and calculate the sample assigned to each cluster center q i The sum of the probabilities f i =∑ i q ij ; Use additional target distributions to refine cluster centers from the current high confidence distribution and improve clustering results. The target distributions are as follows:

[0013]

[0014] Among them, p ij represents the probability that sample i belongs to cluster j; f j represents the weight of cluster j; q ij Represents the similarity score between sample i and cluster j; ∑ j′ q ij′ / f j′ The denominator of is the normalized calculation of the similarity scores of all clusters j′, ensuring that p ij The value of is relative in all clusters, indicating the probability of sample i belonging to all clusters;

[0015] Next, the hierarchical prototype training model uses a loss function based on KL divergence to measure the gap between the soft distribution and the target distribution. The formula is as follows:

[0016]

[0017] Among them, P represents the true distribution; Q represents the approximate or model distribution;

[0018] The clustering effect is improved by alternating the representation of the cluster center and the probability of each sample to each cluster center. When the change between two consecutive iterations is less than the preset value, the optimization is stopped, and the optimized image encoder F is finally obtained. x (·).

[0019] Step 2: Construct a prompt template based on the image encoder F in step 1 x (·) Obtain image category features and image context features, combine learnable prompts, disease category text information, and similar prototypes at different levels to form prompt templates. The constructed prompt templates can effectively serve the base class and the new class, ensuring the effective transfer and application of knowledge between different categories. The specific operations are as follows:

[0020] Step 2.1: For the input image sample I i Perform image augmentation;

[0021] Step 2.2: Perform text augmentation on the text information T of the category in the input;

[0022] Step 2.3: Use the text encoder to build a text encoder network;

[0023] Step 2.4: Select the pre-trained language model vit_base_patch16_224 as the backbone model, freeze the backbone model, and only adjust the input prompts during training;

[0024] Step 2.5: Build the text projection head network g x (·), ensure the dimensionality consistency of images and texts;

[0025] Step 2.6: Extract hint information, which includes the text category feature x of the image cls , Learnable Tips x learn , disease category text information x class , similar prototypes at different levels x pro And the image context feature x context ;

[0026] Step 2.7: Prompt template construction, mapping all prompt information to the prompt template Obtain a hint representation of the input image sample, which can flexibly adapt to the input image sample, whether it comes from the base class or the new class, and ensure that the model effectively uses existing knowledge when dealing with different categories to improve classification performance. ;

[0027] Step 2.8: Use the prompts to build a prompt learning network, concatenate the prompts obtained in step 2.7 with the representation of the input image sample to form an enhanced input image feature as the input of the final Transformer model.

[0028] The step 3: Use the prompt template in step 2 to construct the required prompt learning model, randomly select a small number of base class image data samples, freeze the backbone network for training, and obtain the initialization value of the learnable prompt. The prompt obtained in this process is based on the base class data, so that it can effectively capture the characteristics and knowledge of the base class, which provides a solid foundation for the subsequent learning of new classes, so that the model can better transfer and apply the knowledge learned in the base class, thereby improving the classification performance of the new class, which specifically includes the following steps:

[0029] Step 3.1: Build a hint learning model for base class image data, where hints include text category features of base class images Learning tips Disease category text information Similar prototypes at different levels And the base class image context features All prompt information is mapped to the prompt template obtained in step 2 Then, the obtained hints are concatenated with the representation of the base class image samples and input into the backbone model in step 2.4 to construct the hint learning model;

[0030] Step 3.2: Use the base class image sample I from step 2 i The text information T of the category is trained by the Adam optimizer. The Adam optimizer is used in a stochastic gradient descent manner. The backbone model is frozen during training. Only the learnable prompts are optimized during training. The final learnable prompts As the initialization value of the learnable prompt in subsequent training; the specific operation is:

[0031] Step 3.2.1: First, take the base class image sample I i Input into the image encoding network in step 1;

[0032] Step 3.2.2: Next, input the text information T of the base class image sample category into the text encoder network in step 2, and obtain the disease category text description representation corresponding to the image. Use the Euclidean distance to calculate the similarity of the disease category text descriptions of different samples. The formula is as follows:

[0033] s ij =||e i -e j ||

[0034] where e i 、e j is the representation of the disease category text description of different samples in the same batch; for each image sample Ii, its true label is Find and label according to similarity formula The closest category label is recorded as Among them, the image sample I i The predicted label output after the image encoder is

[0035] Step 3.2.3: Calculate the cross entropy loss using the predicted labels. The cross entropy loss is defined as follows:

[0036]

[0037] Step 3.2.4: For each medical image sample I i Each layer of features Sample an image from the same layer I i As negative examples, there are two types of negative examples. One is the same layer and has the same image sample I. iThe closest different real label samples are denoted as The other is the same layer and belongs to the image sample I i The nearest Class samples, i.e. text semantics and image samples I i The sample of the most similar class and image sample I i The closest sample is denoted by Difficult example Randomly select one of the two, positive example Then it is the cluster center of the current layer, and the formula is as follows:

[0038]

[0039] Step 3.2.5: Calculate the triplet loss using anchor samples, positive samples, and negative samples. The triplet loss formula is as follows:

[0040]

[0041] Among them, f i represents the anchor point sample; represents a positive sample; represents a negative sample; α represents the threshold; Represents the sum of the Euclidean distances from the positive sample and the negative sample to the anchor point sample;

[0042] Step 3.2.6: Calculate the total training loss for a single step To train the network parameters, the total training loss The formula is as follows:

[0043]

[0044] Step 3.2.7: Repeat steps 3.2.1 to 3.2.6 in an iterative manner to train the cue learning model and use the Adam optimizer to update the parameters of the disease localization network through stochastic gradient descent. After the training is completed, the parameters of the determined learnable cue are recorded as As the subsequent initialization value.

[0045] Randomly select a small number of new class data samples for fine-tuning, use the prompt template in step 2 and the initialization value of the learnable prompt in step 3, freeze the backbone network, and train it in a stochastic gradient descent manner through the Adam optimizer to obtain a multi-level prompt learning model; the prompts obtained in this process are based on the new class data, which helps the model to better adapt to the new class characteristics and distribution, and the model can make full use of the knowledge learned based on the base class. At the same time, through fine-tuning of the new class samples, its performance on the new task is enhanced, and the classification accuracy and generalization ability are improved; the specific operations are:

[0046] Step 4.1: Select some image data from the new class as the support set. This is a part of the samples carefully selected from the new class data to provide sufficient information for the model so that the model can effectively learn the new class features during the fine-tuning stage, and can smoothly adapt and perform effective classification; build a fine-tuning prompt learning model for the support set of the new class, including the category features of the new class image Learnable hints from step 3 Disease category text information Similar base class prototypes at different levels And new class image context features

[0047] Step 4.2: Sample from the support set and use the cue learning model obtained in step 3.1 for training. Use the Adam optimizer to train in a stochastic gradient descent manner. The losses are cross entropy loss and triplet loss. Freeze the backbone model during training and only optimize the learnable cue during training. After training, determine the parameters of the learnable cue as the initialization value for subsequent fine-tuning. Training is performed on the basis of the base class to obtain a multi-level prompt learning model; the learnable prompts obtained in this way make full use of the representative samples provided by the support set, allowing the model to better understand the characteristics of the new class, thereby being more targeted in training, which not only speeds up the speed at which the model adapts to the new class, but also improves its generalization ability on new tasks.

[0048] Step 5: Use the multi-level cue learning model trained in step 4 to extract disease representations and perform disease classification;

[0049] Step 5.1: Select some data from the new class as the test set and build a test prompt learning model for the new class. The prompt includes the category features of the new class image. Learnable hints from step 4 Similar base class prototypes at different levels And new class image context features Tip: The constructed data comes from the test set, which is a sample independently drawn from the new class. It aims to evaluate the performance of the model in practical applications and effectively verify the generalization ability and classification accuracy of the model.

[0050] Step 5.2: Use the prompt model obtained in step 3.1 to test the disease classification results by sampling multiple times from the test set. During the test, not only the backbone model but also all prompts including learnable prompts are frozen to ensure that the model is in the evaluation stage without being disturbed by other parameter updates, maintaining its stability and consistency, and more truly reflecting the actual performance and generalization ability of the model in new types of tasks.

[0051] A disease classification system based on multi-level prompt learning, the system is used to implement the above-mentioned disease classification method based on multi-level prompt learning, the system includes a computer processor, an image text data storage unit, an image text data preprocessing unit, a hierarchical prototype processing unit, a prompt model training unit, and a disease classification output unit;

[0052] The image text data storage unit stores the image data and category text description of the base class and the new class; the image text data preprocessing unit executes step 1 to obtain the features of different levels of the base class image and stores them in the computer memory; the hierarchical prototype processing unit executes step 1 to obtain the clustering centers of the base classes at different levels; the prompt model training unit executes steps 2-4 to obtain the prompt model suitable for the new class; the disease classification output unit executes step 5 to obtain the result of the disease classification; the specific data processing and calculation work in all units is completed by the computer processor.

[0053] Compared with the prior art, the present invention has the following advantages:

[0054] 1. A disease classification method based on multi-level cue learning designed by the present invention makes full use of base class knowledge. The method adopts a multi-level feature extraction method to learn feature representations at different semantic levels from base class images. This includes visual features such as texture and shape at the bottom layer, structural features such as position at the middle layer, and semantic features of diseases at the high level. Through a learnable cue fusion mechanism, these base class knowledge of different granularities are effectively transferred to the learning process of new class samples. This method can effectively make up for the lack of knowledge of new categories in small sample cases, enhance the model's understanding and representation capabilities of new categories, thereby effectively improving the quality of vector representation of medical image data and improving the utilization rate of medical image data;

[0055] 2. A disease classification method based on multi-level prompt learning designed by the present invention enhances the model's understanding ability. This method enables the model to better understand the relationship between images and texts in medical image classification tasks through multi-level feature representation and prompt learning. The location, size, severity and other information of the lesion in the image are closely corresponding to the content described in the text. Multi-level feature representation helps to capture this fine-grained correspondence, and prompt learning further strengthens this association, improves the quality and utilization of the overall representation, and enhances the model's understanding ability.

[0056] 3. A disease classification method based on multi-level cue learning designed by the present invention proposes a new hard negative example sampling strategy, which can better distinguish two types of "hard negative examples" in disease classification tasks. The first category is samples clustered at the same feature level. They are highly similar in some feature dimensions, but actually belong to different disease categories, which can easily cause confusion in the model. The second category is the most semantically similar samples, that is, the most similar disease categories. Although the overall feature differences are not large, they still constitute "hard negative examples" for tasks that require precise positioning of lesions. The hard negative example sampling strategy of this method can effectively identify and distinguish these two types of "hard negative examples", improve the accuracy of the model in disease classification tasks, and provide more accurate results for subsequent treatments.

[0057] 4. The disease classification method and system designed by the present invention based on multi-level prompt learning achieves improved generalization performance, fully integrates the prompt learning mechanism of base class knowledge, and effectively makes up for the lack of model generalization ability under small sample conditions. By extracting multi-level features, learning the rich knowledge of the base class, and integrating it into the learning process of new class samples, the model can be better migrated to new categories, improving the classification performance on new class sample data sets. This is of great significance for the diagnosis of new diseases faced in clinical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is an overall framework diagram of a disease classification method based on multi-level prompt learning;

[0059] Figure 2 This is a structural diagram of a prompt model training module in a disease classification method based on multi-level prompt learning;

[0060] Figure 3 This is a structural diagram of a disease classification system based on multi-level prompt learning;

[0061] Figure 4 Flowchart of a disease classification method based on multi-level cue learning. DETAILED DESCRIPTION

[0062] In order to gain a deeper understanding of the present invention, we will provide a comprehensive and detailed description of the present invention. However, the present invention has multiple implementations and is not limited to the specific examples listed herein. The presentation of these examples is intended to deepen the comprehensive understanding of the disclosure of the present invention.

[0063] The present invention is implemented by a computer program. Figure 3 The following is a diagram of the system structure implemented by the computer. Figure 1 , 2 The model structure diagram shown and Figure 4The method flow chart in the figure uses the image dataset MedMNIST in 10 medical fields collected and collated by Shanghai Jiaotong University to explain the technical solution of the present invention in detail. This dataset is a commonly used medical dataset, covering common medical imaging data sources: pathological images, chest X-rays, dermatoscopes, ultrasound, fundus photography, OCT, CT, etc. The implementation method mainly includes the following key contents:

[0064] Step 1: Preprocess the base class image and text datasets. The images are disease images of a certain modality, and the texts are the category labels of the images. Design a hierarchical prototype training model to extract the base class clustering centers of different levels in the base class image data, and train an image encoder to learn the discriminative features of the base class images.

[0065] Step 1.1: Process the medical text dataset. In this medical image text dataset, medical images are disease images of a certain modality of patients, and text is the category label of the image. In order to improve the quality of text information, a text description is added to each category label, mainly starting from the definition. When the definition can be clearly divided into multiple levels, such as the bottom-level visual features such as texture and shape, the middle-level structural features such as the location, and the high-level semantic features of the disease, the corresponding text description is extracted as the category description. If there is no definition of a hierarchical structure, the definition will be truncated as the category description.

[0066] Step 1.2: Process the medical image dataset. For image data, in order to extract multi-level and multi-scale image features, the PLIP model is used as the image encoder to extract the output features of the base class image at the 5th, 9th, and 12th transformer layers in turn as the multi-level base class image feature representation from low to high levels of the base class image. The output features of these three layers are recorded as The average feature is denoted as f i ; The input data shape of the image encoder is [64, 3, 224, 224], where 64 represents the batch size, 3 represents the RGB three channels, and 224×224 is the resolution of each image. After encoding, the network output shape is [64, 50, 768], where the batch size remains unchanged, 50 represents the maximum number of tokens contained in each image, and 768 represents that each token is encoded into a 768-dimensional vector representation;

[0067] Step 1.3: Joint iterative training obtains the image encoder and base class cluster centers at different levels. Specifically, the Deep Embedded Clustering (DEC) algorithm is used. DEC clusters data by simultaneously learning K cluster centers in the feature space and the parameters of the image encoder that maps data points to the feature space. According to the extracted multi-level base class image feature representation, at each level, the K cluster centers are first clustered and initialized as ui , (i=1, 2...K), use Student's t distribution to measure the similarity between the sample and the cluster center, and calculate the similarity between the sample and the cluster center q i and the soft class frequency f i =∑ i q ij , (i=1, 2...n). In order to improve the clustering effect and give more importance to data points with high confidence distribution, an additional target distribution is used to refine the cluster center from the current high confidence distribution. The target distribution is as follows:

[0068]

[0069] Among them, p ij represents the probability that sample i belongs to cluster j; f j represents the weight of cluster j; q ij Represents the similarity score between sample i and cluster j; ∑ j′ q ij′ / f j′ The denominator of is the normalized calculation of the similarity scores of all clusters j′, ensuring that p ij The value of is relative in all clusters, indicating the probability of sample i belonging to all clusters.

[0070] Next, the hierarchical prototype training model uses a loss function based on KL divergence to measure the gap between the soft distribution and the target distribution. The formula is as follows:

[0071]

[0072] Among them, P represents the true distribution; Q represents the approximate or model distribution;

[0073] The clustering effect is improved by alternating the representation of the cluster center and the probability of each sample to each cluster center. When the change between two consecutive iterations is less than the preset value, the optimization is stopped, and the optimized image encoder F is finally obtained. x (·).

[0074] Step 2: Construct a prompt template based on the image encoder F obtained in step 1 x (·) Obtain image category features and image context features, and combine learnable prompts, disease category text information, and similar prototypes at different levels to form a prompt model, which specifically includes the following steps:

[0075] Step 2.1: For the input image sample I iPerform image augmentation: A series of data augmentation operations are performed in the preprocessing stage of medical image model training. They mainly include scaling according to the specific data set, center cropping the image to 28×28 pixels, horizontal flipping with a probability of 20%, and random adjustment of color and affine transformations. Finally, the image tensor is normalized. Overall, these data processing methods are aimed at improving the performance and generalization ability of the model in complex downstream applications such as medical classification.

[0076] Step 2.2: Perform text augmentation on the text information T of the category in the input; for the text information T, truncate the sentences with a length of more than 100 tokens; assume that the maximum length of the text information in the same data set is M max , the length is less than M max The sentences are padded with 0 to unify their length;

[0077] Step 2.3: Use the text encoder of the PLIP model to build a text encoder network with input of dimension [C, M max ] category text information, where M max is the maximum length of the class description text, C is the number of data set categories; the output is a text tensor of dimension [C, 512], where C represents the number of categories of the specific data set, and 512 represents the vector dimension of each category text token in the output tensor;

[0078] Step 2.4: Select the pre-trained language model vit_base_patch16_224 as the backbone model, which is a transformer encoder model (similar to BERT) pre-trained on a large number of images in a supervised manner at 224×224 pixels. Subsequently, the vit model fine-tunes the model on ImageNet, a dataset containing 1 million images and 1000 classes with a resolution of 224×224.

[0079] During training, by freezing the backbone model, only the input prompt is adjusted to achieve prompt learning. The focus of this method is to utilize the feature extraction ability of the pre-trained model to avoid updating the parameters of the backbone network, thereby speeding up training and reducing the risk of overfitting. Specifically, the input of the image encoder network is a small batch of image samples of dimension [64, 3, 28, 28], where 64 is the batch size, 3 is the color channel of the image, and [28, 28] is the image size. Its output is an image tensor representation of dimension [64, 3, 50, 768], where 768 is the vector dimension of each image token, 3 represents the number of intermediate layers, and 50 is the total number of image block tokens and [CLS] tokens.

[0080] Step 2.5: Build the text projection head network gx (·). Text Projection Head Network g x (·) is a single-layer linear layer with an input dimension of [32, 512] and an output dimension of [32, 768]. This is because the output dimensions of the image and text are inconsistent. The image output dimension is 768, while the text dimension is 512. In order to achieve effective splicing and form the final prompt, the text needs to be projected. This projection not only ensures dimensional consistency, but also enhances the expressiveness of the model and promotes the fusion of cross-modal features, thereby improving task performance;

[0081] Step 2.6: Hint Information Extraction: Hints include text category features x of the input image cls , Learnable Tips x learn , disease category text information x class , similar prototypes at different levels x pro And the image context feature x context ;

[0082] For each image sample I i , after the image encoder F x (·) After encoding, a structured, flexible and efficient feature representation is obtained;

[0083] Among them, the cls token, which is a global feature representation and gathers the overall information of the image, is denoted by x cls ;

[0084] The remaining part of feature x is recorded as context feature x context , contains detailed local information of the image, provides rich contextual information, and supports more fine-grained analysis;

[0085] Learnable Tips x learn It is a randomly initialized learnable variable with a length of 5 tokens. As one of the traditional components of prompts, learnable prompts can be randomly initialized and optimized during training to enhance the adaptability and performance of the model, so that it can better capture the characteristics of specific tasks;

[0086] Disease category text information x class The feature representation obtained by inputting the text description corresponding to the sample disease category into the text encoder network in step 2.3 indicates that the semantic features of the disease category have been extracted, providing knowledge related to the sample, promoting cross-modal learning, and helping the model to more accurately understand and classify the relationship between the image and the corresponding disease category;

[0087] Similar prototype x proIt is a feature representation extracted from the hierarchical structure of the image. These features are matched with the cluster centers of the corresponding layers in step 1 at different levels, and the cluster centers that are most similar to the image features at each level are recorded. The hierarchical structure features can more comprehensively reflect the image content, including details and global information, and improve the accuracy of feature representation. By matching similar cluster centers, the model can recognize and utilize the similarities between features, thereby enhancing the reliability and consistency of decisions in classification tasks. Similar prototypes improve the performance of the model in complex tasks by combining hierarchical features with clustering information.

[0088] Step 2.7: Prompt template construction, mapping all prompt information to prompt template x prompt =[x cls ;x learn ;x class ;x pro ;x context ], get the hint representation of the input image sample, the input image can be from either the base class or the new class;

[0089] Step 2.8: Use the prompts to build a prompt learning network, concatenate the prompts obtained in step 2.7 with the representation of the input image sample to form an enhanced input image feature as the input of the final Transformer model.

[0090] Step 3: Use the prompt template in step 2 to construct the required prompt learning model, randomly select a small number of base class image data samples, freeze the backbone network for training, and then obtain the initialization value of the learnable prompt. Only the prompt is updated to reduce the complexity of parameter update and speed up the training. The overall structure is as follows: Figure 1 As shown, the specific steps include:

[0091] Step 3.1: Build a hint learning model for base class image data, where hints include text category features of base class images Learning tips Disease category text information Similar prototypes at different levels And the base class image context features All information is mapped to the prompt template Then, the obtained hints are concatenated with the representation of the base class image samples and input into the backbone model in step 2.4 to construct the hint learning model;

[0092] Step 3.2: Use the base class image sample I from step 2 i The prompt learning model is trained 100 times by Adam optimizer with the text information T of the category, and each training step uses 64 pairs of base class image samples I i, T augmented samples of text information of the category. Taking single-step training as an example, the following specific steps are included:

[0093] Step 3.2.1: Figure 1 As shown, for a batch of medical images (64 images), they are first input into the image encoding network in step 1 to obtain image representations with tensor dimensions of [64, 50, 768];

[0094] Step 3.2.2: Next, input the text information T of the image sample category into the text encoder network in step 2, and obtain the text description representation of the disease category corresponding to the image. Use the Euclidean distance to calculate the similarity of the disease category text descriptions of different samples. The formula is as follows:

[0095] s ij =||e i -e j ||

[0096] where e i 、e j is the representation of the disease category text description of different samples in the same batch. For each image sample, its true label is Find and label according to similarity formula The closest category label is recorded as Among them, the image sample I i The predicted label output after the image encoder is

[0097] Step 3.2.3: Calculate the cross entropy loss using the predicted labels. The loss definition formula is as follows:

[0098]

[0099] Step 3.2.4: For each medical image I i Each layer of features Sampling an image sample from the same layer as a negative example, there are two types of negative examples, one is the same layer and I i The closest different real label samples are denoted as The other is the same layer and belongs to I i The nearest Class samples, that is, text semantics and I i The samples of the most similar class are i The closest sample is denoted by Difficult example Randomly select one of these two, the positive example Then it is the cluster center of the current layer, and the formula is as follows:

[0100]

[0101] Step 3.2.5: Calculate the triplet loss using anchor samples, positive samples, and negative samples. The formula is as follows:

[0102]

[0103] Step 3.2.6: Calculate the total training loss for a single step To train the network parameters, the total training loss The formula is as follows:

[0104]

[0105] Step 3.2.7: Repeat steps 3.2.1 to 3.2.6 in an iterative manner, train the cue learning model for 100 rounds, and use the Adam optimizer to update the parameters of the disease localization network through stochastic gradient descent. After the training is completed, the parameters of the determined learnable cue are recorded as As the subsequent initialization value.

[0106] Step 4: Randomly select a small number of new class data samples for fine-tuning, use the prompt template in step 2 and the initialization value of the learnable prompt in step 3, freeze the backbone network for training, and obtain a multi-level prompt learning model; through the learnable prompt, the knowledge obtained on the base class is effectively transferred to the new class, which improves the learning efficiency of the model on the new task, reduces the dependence on a large amount of labeled data, and reduces the cost of data collection and labeling. Specifically, it includes the following steps:

[0107] Step 4.1: Select some image data from the new class as the support set. This is a part of the samples carefully selected from the new class data to provide sufficient information for the model so that the model can effectively learn the new class features during the fine-tuning stage, and can smoothly adapt and perform effective classification; construct a fine-tuning prompt learning model for the new class support set, which includes the category features of the new class images. Learnable hints from step 3 Disease category text information Similar base class prototypes at different levels And new class image context features

[0108] Step 4.2: Select some data from the new class as the support set, and use the prompt model obtained in step 3.1 to train from the support set. This is similar to step 3.2.7, except that all image data comes from the support set. Repeat steps 3.2.1 to 3.2.6 iteratively, train the prompt learning model for 100 rounds, and train it with the Adam optimizer using stochastic gradient descent. The losses are cross entropy loss and triplet loss. Freeze the backbone during training, and only adjust the input prompts during training, that is, only the learnable prompts are optimized. After training, determine the parameters of the learnable prompts as the initialization values ​​for subsequent fine-tuning. The final learnable prompts It is trained on the basis of the base class. Through training on the support set, the model can retain and utilize the knowledge learned on the base class to strengthen the basis for learning the new class. At the same time, the model can optimize the features of the new class to improve its performance on the new class. This method combines the knowledge of the base class with the features of the new class to enhance the generalization ability and accuracy of the model. Secondly, through effective initialization values ​​and knowledge transfer, the need for a large amount of new class data is reduced, thereby reducing training costs and time.

[0109] Step 5: Figure 1 As shown, the multi-level prompt learning model trained in step 4 is used to extract disease representations and classify diseases, which specifically includes the following steps:

[0110] Step 5.1: Scale the medical image of the new patient to 28×28, input the processed image into the image encoder to obtain the image representation; input the disease category text information corresponding to the sample into the text encoder to obtain the disease category text information representation; select part of the data from the new class as the test set, and build a test prompt learning model for the new class, which includes the category features of the new class image Learnable hints from step 4 Similar base class prototypes at different levels And new class image context features Tip: The constructed data comes from the test set, which is a sample independently drawn from the new class. It aims to evaluate the performance of the model in practical applications and effectively verify the generalization ability and classification accuracy of the model.

[0111] Step 5.2: Use the prompt model obtained in step 3.1 to test the disease classification results by sampling multiple times from the test set. During the test, not only the backbone model but also all prompts including learnable prompts are frozen to ensure that the model is in the evaluation stage without being disturbed by other parameter updates, maintain its stability and consistency, and more truly reflect the actual performance and generalization ability of the model in new types of tasks. The patient's medical image representation and the category text information representation are input into the prompt learning model to obtain the disease diagnosis results. In order to measure the effectiveness of disease diagnosis, the accuracy rate is used as an indicator to compare with other baseline methods. The accuracy rate can reflect the average classification effect of the method on different test samples. By comparing the classification accuracy rates of different methods on the test set, the classification efficiency of the multi-level prompt learning method in small sample tasks can be intuitively judged. The results are shown in Table 1.

[0112] Table 1 Disease classification results

[0113]

[0114] like Figure 3 As shown, a disease classification system based on multi-level prompt learning, the system is used to implement the above-mentioned disease classification method based on multi-level prompt learning, including a computer processor, an image text data storage unit, an image text data preprocessing unit, a hierarchical prototype processing unit, a prompt model training unit, and a disease classification output unit.

[0115] The image text data storage unit stores the image data and category text description of the base class and the new class; the image text data preprocessing unit executes step 1 to obtain the features of different levels of the base class image and stores them in the computer memory; the hierarchical prototype processing unit executes step 1 to obtain the clustering centers of the base classes at different levels; the prompt model training unit executes steps 2-4 to obtain the prompt model suitable for the new class; the disease classification output unit executes step 5 to obtain the result of the disease classification; the specific data processing and calculation work in all units is completed by the computer processor.

[0116] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in the field. Although the illustrative specific embodiments of the present invention are described above to facilitate the understanding of the present invention by the technical personnel in the field, it should be clear that the present invention is not limited to the scope of the specific embodiments. For the ordinary technical personnel in the field, as long as various changes are within the spirit and scope of the present invention defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A disease classification method based on multi-level prompt learning, characterized in that: The method comprises the following steps: Step 1: Preprocess the base class image and text datasets. The images are disease images of a certain modality, and the texts are the category labels of the images. Design a hierarchical prototype training model to extract cluster centers of different levels in the base class image data, and train an image encoder to learn the discriminative features of the base class images. Step 2: Construct a prompt template based on the image encoder F in step 1 x (·) Obtain image category features and image context features, and combine them with learnable prompts, disease category text information, and similar prototypes at different levels to form prompt templates; Step 3: Use the prompt template in step 2 to construct the required prompt learning model, randomly select a small number of base class image data samples, freeze the backbone network for training, and obtain the initialization value of the learnable prompt with base class knowledge; Step 4: Randomly select a small number of new class data samples for fine-tuning, use the prompt template in step 2 and the initialization value of the learnable prompt in step 3, freeze the backbone network for training, and obtain a multi-level prompt learning model; Step 5: Use the multi-level cue learning model trained in step 4 to extract disease representations and perform disease classification.

2. A disease classification method based on multi-level prompt learning according to claim 1, characterized in that: The step 1: preprocessing base class images and text data sets, where the images are disease images of a certain modality and the texts are category labels of the images; designing a hierarchical prototype training model to extract base class clustering centers of different levels in the base class image data, and training an image encoder to learn the discriminative features of the base class images. The specific operations are: Step 1.1: Preprocess the base class text dataset; add text descriptions to each category label, starting from the definition, when a hierarchical definition is formed, extract the corresponding text description as the category description; when a hierarchical definition is not formed, truncate the definition as the category description; Step 1.2: Preprocess the base class image dataset; design a hierarchical prototype training model to extract the base class cluster centers at different levels in the base class image data, and train an image encoder that can learn the discriminative features of the image, and extract the output features of the base class images in turn as the multi-level base class image feature representation from low level to high level of the base class images; Step 1.3: Joint iterative training to obtain image encoders and cluster centers at different levels; based on the extracted multi-level base class image feature representation, at each level, first cluster and initialize K cluster centers q i , (i=1,2…K), use Student t distribution to measure the similarity between the sample and the cluster center, and calculate the sample assigned to each cluster center q i The sum of the probabilities of i =∑ i q ij ; Use additional target distributions to refine cluster centers from the current high confidence distribution and improve clustering results. The target distributions are as follows: Among them, p ij represents the probability that sample i belongs to cluster j; f j represents the weight of cluster j; q ij Represents the similarity score between sample i and cluster j; ∑ j' q ij' / f j' The denominator of is the normalized calculation of the similarity scores of all clusters j', ensuring that p ij The value of is relative in all clusters, indicating the probability of sample i belonging to all clusters; Next, the hierarchical prototype training model uses a loss function based on KL divergence to measure the gap between the soft distribution and the target distribution. The formula is as follows: Among them, P represents the true distribution; Q represents the approximate or model distribution; The clustering effect is improved by alternating the representation of the cluster center and the probability of each sample to each cluster center. When the change between two consecutive iterations is less than the preset value, the optimization is stopped, and the optimized image encoder F is finally obtained. x (·).

3. A disease classification method based on multi-level prompt learning according to claim 2, characterized in that: Step 2: Construct a prompt template according to the image encoder F in step 1 x (·) Obtain image category features and image context features, and combine learnable prompts, disease category text information, and similar prototypes at different levels to form prompt templates. The specific operations are as follows: Step 2.1: For the input image sample I i Perform image augmentation; Step 2.2: Perform text augmentation on the text information T of the category in the input; Step 2.3: Use the text encoder to build a text encoder network; Step 2.4: Select the pre-trained language model vit_base_patch16_224 as the backbone model, freeze the backbone model, and only adjust the input prompts during training; Step 2.5: Build the text projection head network g x (·), ensure the dimensionality consistency of images and texts; Step 2.6: Extract hint information, which includes the text category feature x of the image cls , Learnable Tips x learn , disease category text information x class , similar prototypes at different levels x pro And the image context feature x context ; Step 2.7: Prompt template construction, mapping all prompt information to the prompt template Get a hint representation of the input image sample; Step 2.8: Use the prompts to build a prompt learning network, concatenate the prompts obtained in step 2.7 with the representation of the input image sample to form an enhanced input image feature as the input of the final Transformer model.

4. A disease classification method based on multi-level prompt learning according to claim 3, characterized in that: The step 3: constructing the required prompt learning model using the prompt template in step 2, randomly selecting a small number of base class image data samples, freezing the backbone network for training, and obtaining the initialization value of the learnable prompt, specifically includes the following steps: Step 3.1: Build a hint learning model for base class image data, where hints include text category features of base class images Learning tips Disease category text information Similar prototypes at different levels And the base class image context features All prompt information is mapped to the prompt template obtained in step 2 Then, the obtained hints are concatenated with the representation of the base class image samples and input into the backbone model in step 2.4 to construct the hint learning model; Step 3.2: Use the base class image sample I from step 2 i The text information T of the category is trained by the Adam optimizer. The Adam optimizer is used in a stochastic gradient descent manner. The backbone model is frozen during training. Only the learnable prompts are optimized during training. The final learnable prompts As the initialization value of the learnable prompt in subsequent training; the specific operation is: Step 3.2.1: First, take the base class image sample I i Input into the image encoding network in step 1; Step 3.2.2: Next, input the text information T of the base class image sample category into the text encoder network in step 2, and obtain the disease category text description representation corresponding to the image. Use the Euclidean distance to calculate the similarity of the disease category text descriptions of different samples. The formula is as follows: s ij =||e i -e j || where e i 、e j is the representation of the disease category text description of different samples in the same batch; for each image sample I i , whose true label is Find and label according to similarity formula The closest category label is recorded as Among them, the image sample I i The predicted label output after the image encoder is Step 3.2.3: Calculate the cross entropy loss using the predicted labels. The cross entropy loss is defined as follows: Step 3.2.4: For each medical image sample I i Each layer of features Sample an image from the same layer I i As negative examples, there are two types of negative examples. One is the same layer and has the same image sample I. i The closest different real label samples are denoted as The other is the same layer and belongs to the image sample I i The nearest Class samples, i.e. text semantics and image samples I i The sample of the most similar class and image sample I i The closest sample is denoted by Difficult example Randomly select one of the two, positive example Then it is the cluster center of the current layer, and the formula is as follows: Step 3.2.5: Calculate the triplet loss using anchor samples, positive samples, and negative samples. The triplet loss formula is as follows: Among them, f i represents the anchor point sample; represents a positive sample; represents a negative sample; α represents the threshold; Represents the sum of the Euclidean distances from the positive sample and the negative sample to the anchor point sample; Step 3.2.6: Calculate the total training loss for a single step To train the network parameters, the total training loss The formula is as follows: Step 3.2.7: Repeat steps 3.2.1 to 3.2.6 in an iterative manner to train the cue learning model and use the Adam optimizer to update the parameters of the disease localization network through stochastic gradient descent. After the training is completed, the parameters of the determined learnable cue are recorded as As the subsequent initialization value.

5. A disease classification method based on multi-level prompt learning according to claim 4, characterized in that: Step 4: Randomly select a small number of new class data samples for fine-tuning, use the prompt template in step 2 and the initialization value of the learnable prompt in step 3, freeze the backbone network, and train it in a stochastic gradient descent manner through the Adam optimizer to obtain a multi-level prompt learning model. The specific operations are: Step 4.1: Select some image data from the new class as the support set, and build a fine-tuned prompt learning model for the support set of the new class, where the prompt includes the category features of the new class image Learnable hints from step 3 Disease category text information Similar base class prototypes at different levels And new class image context features Step 4.2: Sample from the support set and use the cue learning model obtained in step 3.1 for training. Use the Adam optimizer to train in a stochastic gradient descent manner. The losses are cross entropy loss and triplet loss. Freeze the backbone model during training and only optimize the learnable cue during training. After training, determine the parameters of the learnable cue as the initialization value for subsequent fine-tuning. Training is performed on the basis of the base class to obtain a multi-level prompt learning model.

6. A disease classification method based on multi-level prompt learning according to claim 5, characterized in that: Step 5: Use the multi-level prompt learning model trained in step 4 to extract disease representations and classify diseases. The specific operations are as follows: Step 5.1: Select some data from the new class as the test set and build a test prompt learning model for the new class. The prompt includes the category features of the new class image. Learnable hints from step 4 Similar base class prototypes at different levels And new class image context features Step 5.2: Use multiple samplings from the test set to test the prompt model obtained in step 3.1 to obtain the disease classification results.

7. A disease classification system based on multi-level prompt learning, characterized in that: The system is used to implement a disease classification method based on multi-level prompt learning as described in any one of claims 1-6, and the system includes a computer processor, an image text data storage unit, an image text data preprocessing unit, a hierarchical prototype processing unit, a prompt model training unit, and a disease classification output unit.

8. A disease classification system based on multi-level prompt learning according to claim 7, characterized in that: The image text data storage unit stores the image data and category text description of the base class and the new class; the image text data preprocessing unit executes step 1 to obtain the features of different levels of the base class image and stores them in the computer memory; the hierarchical prototype processing unit executes step 1 to obtain the clustering centers of the base classes at different levels; the prompt model training unit executes steps 2-4 to obtain the prompt model suitable for the new class; the disease classification output unit executes step 5 to obtain the result of the disease classification; the specific data processing and calculation work in all units is completed by the computer processor.

Images