Medical image classification method based on condition category specific prompt

Abstract

The invention discloses a medical image classification method based on condition category specific prompt, which comprises the following steps of: 1, acquiring a data set, constructing a pre-training model, a feature memory bank and a lightweight network of a comparison text-image pair, adding prompt into an image encoder, and extracting visual features of a medical image; 2, according to the visual features of the medical image, obtaining a category average feature, inputting the category average feature into a lightweight network to obtain a category specific prompt, splicing the category specific prompt and the image category, and inputting the spliced category specific prompt and the image category into a text encoder to obtain a text feature; and 3, constructing a loss function, training the network by using a gradient descent method, updating prompts in an image encoder and parameter weights of the lightweight network, and obtaining an optimal medical image classification model for classifying medical images. According to the method, specific prompts of each category can be generated according to the features of all the images to represent feature attributes of different categories, so that the perception of text prompts on image details can be enhanced, and more efficient training and reasoning are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence deep learning medical image classification, and in particular to a medical image classification method based on conditional category specific prompts. Background Art

[0002] Large-scale visual language models, such as the contrastive text-image pre-training model CLIP, have excellent performance in downstream tasks. Their strong visual and language understanding capabilities enable them to handle complex multimodal information. Their huge training data enables the model to acquire diverse and comprehensive knowledge, thus having strong zero-shot capabilities. Due to the scarcity of medical data, the contrastive language image pre-training model has attracted attention in the field of medical imaging.

[0003] The pre-trained model CLIP for contrasting text-image pairs contains a visual branch and a language branch. After obtaining the visual and text features of the two branches, the two types of features are matched to predict the image category. A commonly used text prompt is "a photo of [category name]". After filling in the category name, it is input into the text encoder to obtain the text feature representation. Existing methods have shown that adding attribute names describing medical images, such as color, texture, and shape information, to the text prompt can better drive the CLIP model to focus on specific features of medical images, thereby improving the model's ability to understand medical images. The attribute information contained in these medical prompts is obtained from pre-trained language models and visual question-answering models in the medical field based on the category name.

[0004] However, relying solely on text descriptions of attributes to enhance the model's perception of medical images has certain limitations. On the one hand, medical images of different categories may have similar attributes. For example, brain tumors "ependymoma" and "medulloblastoma" are both located in the posterior cranial fossa and may show irregular shapes and lobed edges. On the other hand, the similarity between medical images is high. The pre-trained model CLIP for contrastive text-image pairs only focuses on the attribute features obtained based on the category prior and does not necessarily distinguish the differences between images. It also misses the possibility of extracting other discriminative features. These problems significantly affect the generalization ability of the pre-trained model for contrastive text-image pairs in medical image classification tasks. Summary of the invention

[0005] The present invention aims to address the deficiencies of the above-mentioned prior art and proposes a medical image classification method based on conditional category-specific prompts, in order to generate specific prompts for each category according to the characteristics of all medical images to characterize the characteristic attributes of different categories, thereby enhancing the perception of medical image details by text prompts and achieving more efficient medical image recognition and reasoning.

[0006] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0007] The medical image classification method based on conditional category specific prompts of the present invention is characterized in that it includes the following steps:

[0008] Step S1: Obtain a medical image dataset including K categories ,in, Indicates Medical images, represents the total number of medical images; let The true category label is ,and ∈{1,2,…,K};

[0009] The first Medical images Divide A fixed-size image block is formed, and each image block is flattened and then projected onto a fixed dimension, thereby The projected one-dimensional vectors are concatenated into image blocks for embedding ;

[0010] Constructing Hints to Learn ,in, Indicates Tips to learn; Indicates the total number of prompts;

[0011] Step S2: Construct a pre-trained model for comparing text-image pairs, including: image encoder , Text Encoder , wherein the image encoder is used to prompt and image patch embedding Process it and get Medical images The visual feature vector ;

[0012] Step S3: Use formula (1) to obtain the average image feature vector of the kth category: :

[0013] (1)

[0014] In formula (1), represents the total number of categories, is the first Medical images The visual feature vector of is the total number of medical images in the kth category;

[0015] Step S4: Build a lightweight network and Processing to get specific tips for the kth category ;

[0016] Step S5: Category-specific tips |k=1,2,…,K} and the name of each category |k=1,2,…,K} after corresponding concatenation, input text encoder Processed in, get the text features { |k=1,2,…,K}; where represents the name of the k-th category, Represents the text features of the kth category;

[0017] Step S6: and{ |k=1,2,…,K} to match, and select the category corresponding to the maximum probability as the first Medical images The prediction category of

[0018] Step S7: Construct loss function , and train the pre-trained model and lightweight network of the contrasting text-image pair using the gradient descent method, and update the image encoder by gradient backpropagation Tips in and the parameter weights of the lightweight network until the loss function Until convergence, the optimal medical image classification model is obtained, which is used to classify medical images and output the predicted category.

[0019] The medical image classification method based on conditional category specific prompts of the present invention is also characterized in that the image encoder in step S2 Includes: Category Embedding , Position Embedding , Layer Image Self-Attention Model , where each layer of the image self-attention model consists of a multi-head image self-attention layer and a multi-layer image perception machine block;

[0020] Step S21: Initialization =1; define and initialize the -1 layer image self-attention model Output visual feature vector ;

[0021] Initialize the self-attention layer of the input i-th layer Tips = ; Embed the image block As the i-th layer of self-attention Image patch embedding ; Initialize input to the first Self-attention layer Category embedding = ;

[0022] Step S22: Prompt , Image Patch Embedding With category embedding After concatenation, embed with position Add and enter Self-attention layer In order to process it, the formula (2) is used to output the Layer image patch embedding With category embedding ;

[0023] (2)

[0024] In formula (2), is the first self-attention layer, and It is Category embedding and image patch embedding output by the self-attention layer;

[0025] Step S221: Input the prompt of the i-th layer , Image Patch Embedding With category embedding After concatenation, layer normalization is performed to obtain the embedding vector sequence using formula (3): ;

[0026] (3)

[0027] Step S222, self-attention layer of the i-th layer The h self-attention heads of Processing is performed to obtain the self-attention layer of the i-th layer h multi-head attention representations , and then use formula (5) to get the self-attention layer of the i-th layer Comprehensive characterization of output :

[0028] (4)

[0029] (5)

[0030] In formula (4)-formula (5), , , is the self-attention layer of the i-th layer No. The projection parameter matrix of the self-attention head, is also the projection parameter matrix, is the activation function, represents transpose, Represents a serial operation, Represents the self-attention layer of the i-th layer No. Multi-head attention representation of the output of the attention heads , is the number of parallel self-attention heads;

[0031] Step S223: Use formula (6) to obtain the self-attention layer of the i-th layer Residual representation in :

[0032] (6)

[0033] In formula (6), represents residual connection;

[0034] Step S224: Use formula (7) to obtain Self-attention layer Output image patch embedding With category embedding :

[0035] (7)

[0036] In formula (7), and are the two transformation matrices of the two linear layers, and are the two biases of the two linear layers, is the activation function; Representation layer normalization operation; Indicates Self-attention layer Output prompts;

[0037] Step S225, after copying i+1 to i, return to step 221 and execute sequentially until i>J, thereby obtaining the J-layer self-attention layer { |i=1,2,…,J} The output image patch embedding { |i=1,2,…,J} and category embedding { |i=1,2,…,J};

[0038] Step S23: Use formula (8) to obtain Self-attention layer To Self-attention layer The self-attention layer of any kth layer in Output image patch embedding , Category Embedding and tips :

[0039] (8)

[0040] Step S24: Self-attention layer Output category embedding After projection, the medical image is obtained Visual features .

[0041] Further, the step S4 includes:

[0042] Step S4.1: Use formula (9) to obtain the intermediate features of the kth category :

[0043] (9)

[0044] In the formula, is the transformation matrix of the third linear layer, is the bias of the third linear layer;

[0045] Step S4.2: Use formula (10) to obtain the specific prompt of the kth category :

[0046] (10)

[0047] In formula (10), is the transformation matrix of the fourth linear layer, is the bias of the fourth linear layer, is the activation function.

[0048] Furthermore, the step S5 comprises:

[0049] Step S5.1: Create a text encoder include: Layer text self-attention model, where each layer of text self-attention model consists of a multi-head text self-attention layer and a multi-layer text perception machine block;

[0050] Step S5.2: The corresponding category name After concatenation, the k-th sentence text representation is obtained, thereby obtaining the K-sentence text representation;

[0051] Step S5.3: After segmenting the K sentences, project the segmentation results onto a fixed dimension to obtain text embedding. ;

[0052] Step S5.4: Initialization =1; define and initialize the -1 layer text self-attention model Output text embedding = ;

[0053] Step S5.5: Input to Layer Text Self-Attention Model Processed in, output text embedding feature vector ;

[0054] Step S5.5.1, Layer Text Self-Attention Model The h-th head text self-attention layer in the multi-head text self-attention layer uses formula (11) to obtain the h-th head text self-attention layer Layer Text Self-Attention Model The h-th head text attention representation , and then use formula (12) to get the Layer Text Self-Attention Model Comprehensive feature representation of the output of multi-head text self-attention layer :

[0055] (11)

[0056] (12)

[0057] In formulas (11) and (12), , , It is Layer Text Self-Attention Model The three projection parameter matrices of the h-th head text self-attention layer, is the global projection parameter matrix, is the activation function, represents transpose, Represents a serial operation, is the number of parallel text self-attention heads;

[0058] Step S5.5.2: Use formula (13) to get Layer Text Self-Attention Model The residual representation of :

[0059] (13)

[0060] In formula (13), represents residual connection;

[0061] Step S5.5.3: Use formula (14) to obtain Layer Text Self-Attention Model The text embedding output by the multi-layer text perceptron block :

[0062] (14)

[0063] In formula (14), and They are the text transformation matrices of the two linear layers, and They are the text biases of the two linear layers, is the activation function;

[0064] Step S5.5.4: +1 assigned to Then, return to step S5.5.2 and execute sequentially until >L, thus obtaining the Lth text self-attention layer Output text embedding ;

[0065] Step S5.6: Embed text The last token of is projected onto a fixed-dimensional vector space to obtain the text feature vector { |k=1,2,…,K}.

[0066] Furthermore, in step S6, formula (15) is used to obtain Medical images The probability of belonging to the kth category :

[0067] (15)

[0068] In formula (15), represents the cosine similarity, represents the exponential function, Represents the temperature parameter.

[0069] Furthermore, in step S7, the loss function is constructed using formula (16): :

[0070] (16)

[0071] In formula (16), Represents a logarithmic function.

[0072] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the medical image classification method, and the processor is configured to execute the program stored in the memory.

[0073] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the medical image classification method when executed by a processor.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] 1. The method of category-specific prompting of the present invention utilizes a pre-trained model of contrasting text-image pairs to extract image features that may contain subtle feature information that can distinguish medical images, thereby enhancing the text encoder's perception of image features and improving the generalization ability of the pre-trained model of contrasting text-image pairs for medical images.

[0076] 2. The image encoder of the present invention captures the detailed features in the medical image, and these detailed features are stored in the memory bank. The features in the memory bank are converted into prompts matching the text description by category to better align the semantic information and context of the image.

[0077] 3. A small amount of learnable parameters are inserted into the image encoder of the present invention to better enhance the ability of the CLIP model, a pre-trained model for contrasting text-image pairs, to understand medical images, so that the model can more accurately identify and distinguish different medical image categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0079] In this embodiment, a medical image classification method based on conditional category-specific prompts is mainly to capture the detailed features in the medical image by comparing the image encoder of the pre-trained model of the text-image pair. These detailed features are stored in the memory bank, and the features in the memory bank are converted into prompts matching the text description by category to better align the semantic information and context of the image; at the same time, a small amount of learnable parameters are inserted into the image encoder to better enhance the CLIP model's ability to understand medical images; Figure 1 As shown, the following steps are included:

[0080] Step S1: Obtain a medical image dataset including K categories ,in, Indicates Medical images, represents the total number of medical images; let The true category label is ,and ∈{1,2,…,K};

[0081] The first Medical images Divide A fixed-size image block is formed, and each image block is flattened and then projected onto a fixed dimension, thereby The projected one-dimensional vectors are concatenated into image blocks for embedding ;

[0082] Constructing Hints to Learn ,in, Indicates Tips to learn; Indicates the total number of prompts.

[0083] Step S2: Construct an image encoder of a pre-trained model for comparing text-image pairs , including: Category Embedding , Position Embedding , Layer Image Self-Attention Model , where each layer of the image self-attention model consists of a multi-head image self-attention layer and a multi-layer image perception machine block; the image encoder is used to prompt and image patch embedding Process it and get Medical images The visual feature vector ;

[0084] Step S21: Initialization =1; define and initialize the -1 layer image self-attention model Output visual feature vector ;

[0085] Initialize the self-attention layer of the input i-th layer Tips = ; Embed the image block As the i-th layer of self-attention Image patch embedding ; Initialize input to the first Self-attention layer Category embedding = ;

[0086] Step S22: Prompt , Image Patch Embedding With category embedding After concatenation, embed with position Add and enter Self-attention layer In order to process it, the formula (1) is used to output the Layer image patch embedding With category embedding ;

[0087] (1)

[0088] In formula (2), is the first self-attention layer, and It is Category embedding and image patch embedding output by the self-attention layer;

[0089] Step S221: Input the prompt of the i-th layer , Image Patch Embedding With category embedding After concatenation, layer normalization is performed to obtain the embedding vector sequence using formula (2): ;

[0090] (2)

[0091] Step S222, self-attention layer of the i-th layer The h self-attention heads of Processing is performed to obtain the self-attention layer of the i-th layer h multi-head attention representations , and then use formula (4) to get the self-attention layer of the i-th layer Comprehensive characterization of output :

[0092] (3)

[0093] (4)

[0094] In formula (4)-formula (5), , , is the self-attention layer of the i-th layer No. The projection parameter matrix of the self-attention head, is also the projection parameter matrix, is the activation function, represents transpose, Represents a serial operation, Represents the self-attention layer of the i-th layer No. Multi-head attention representation of the output of the attention heads , is the number of parallel self-attention heads.

[0095] Step S223: Use formula (5) to obtain the self-attention layer of the i-th layer Residual representation in :

[0096] (5)

[0097] In formula (6), represents residual connection;

[0098] Step S224: Use formula (6) to obtain Self-attention layer Output image patch embedding With category embedding :

[0099] (6)

[0100] In formula (6), and are the two transformation matrices of the two linear layers, and are the two biases of the two linear layers, is the activation function; Representation layer normalization operation; Indicates Self-attention layer Output prompts;

[0101] Step S225, after copying i+1 to i, return to step 221 and execute sequentially until i>J, thereby obtaining the J-layer self-attention layer { |i=1,2,…,J} The output image patch embedding { |i=1,2,…,J} and category embedding { |i=1,2,…,J}.

[0102] Step S23: Use formula (7) to obtain Self-attention layer To Self-attention layer The self-attention layer of any kth layer in Output image patch embedding , Category Embedding and tips :

[0103] (7)

[0104] Step S24: Self-attention layer Output category embedding After projection, the medical image is obtained Visual features .

[0105] Step S3: Use formula (8) to obtain the average image feature vector of the kth category: :

[0106] (8)

[0107] In formula (8), represents the total number of categories, is the first Medical images The visual feature vector of is the total number of medical images in the kth category.

[0108] Step S4: Build a lightweight network and Processing to get specific tips for the kth category ;

[0109] Step S4.1: Use formula (9) to obtain the intermediate features of the kth category :

[0110] (9)

[0111] In the formula, is the transformation matrix of the third linear layer, is the bias of the third linear layer;

[0112] Step S4.2: Use formula (10) to obtain the specific prompt of the kth category :

[0113] (10)

[0114] In formula (10), is the transformation matrix of the fourth linear layer, is the bias of the fourth linear layer, is the activation function.

[0115] Step S5: Category-specific tips |k=1,2,…,K} and the name of each category |k=1,2,…,K} after corresponding concatenation, input text encoder Processed in, get the text features { |k=1,2,…,K}; where represents the name of the k-th category, Represents the text features of the kth category;

[0116] Step S5.1: Create a text encoder include: Layer text self-attention model, where each layer of text self-attention model consists of a multi-head text self-attention layer and a multi-layer text perception machine block;

[0117] Step S5.2: The corresponding category name After concatenation, the k-th sentence text representation is obtained, thereby obtaining K sentence text representations.

[0118] Step S5.3: After segmenting the K sentences, project the segmentation results onto a fixed dimension to obtain text embedding. ;

[0119] Step S5.4: Initialization =1; define and initialize the -1 layer text self-attention model Output text embedding = ;

[0120] Step S5.5: Input to Layer Text Self-Attention Model Processed in, output text embedding feature vector .

[0121] Step S5.5.1, Layer Text Self-Attention Model The h-th head text self-attention layer in the multi-head text self-attention layer uses formula (11) to obtain the h-th head text self-attention layer Layer Text Self-Attention Model The h-th head text attention representation , and then use formula (12) to get the Layer Text Self-Attention Model Comprehensive feature representation of the output of multi-head text self-attention layer :

[0122] (11)

[0123] (12)

[0124] In formulas (11) and (12), , , It is Layer Text Self-Attention Model The three projection parameter matrices of the h-th head text self-attention layer, is the global projection parameter matrix, is the activation function, represents transpose, Represents a serial operation, is the number of parallel text self-attention heads.

[0125] Step S5.5.2: Use formula (13) to get Layer Text Self-Attention Model The residual representation of :

[0126] (13)

[0127] In formula (13), represents residual connection;

[0128] Step S5.5.3: Use formula (14) to obtain Layer Text Self-Attention Model The text embedding output by the multi-layer text perceptron block :

[0129] (14)

[0130] In formula (14), and They are the text transformation matrices of the two linear layers, and They are the text biases of the two linear layers, is the activation function.

[0131] Step S5.5.4: +1 assigned to Then, return to step S5.5.2 and execute sequentially until >L, thus obtaining the Lth text self-attention layer Output text embedding ;

[0132] Step S5.6: Embed text The last token of is projected onto a fixed-dimensional vector space to obtain the text feature vector { |k=1,2,…,K}.

[0133] Step S6: Use formula (15) to obtain Medical images The probability of belonging to the kth category , select the category corresponding to the maximum probability as the Medical images The predicted category,

[0134] (15)

[0135] In formula (15), represents the cosine similarity, represents the exponential function, Represents the temperature parameter.

[0136] Step S7: Use formula (16) to construct the loss function :

[0137] (16)

[0138] In formula (16), The pre-trained model and lightweight network of the contrasting text-image pair are trained using the gradient descent method, and the image encoder is updated by gradient backpropagation. Tips in and the parameter weights of the lightweight network until the loss function Until convergence, the optimal medical image classification model is obtained, which is used to classify medical images and output the predicted category.

[0139] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the medical image classification method, and the processor is configured to execute the program stored in the memory.

[0140] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the medical image classification method are executed.

[0141] In order to verify the effectiveness of the present invention, this implementation selected five advanced prompt learning classification methods, namely COOP (Context Optimization), COCOOP (Conditional Context Optimization), VPT (Visual Prompt Tuning), MAPLE (Multi-modal Prompt Learning), and COOPLVT (Context Optimization with Learnable Visual Tokens), and trained them on three medical image datasets to evaluate the performance of different method models, namely, brain MRI dataset, melanoma dataset, and tuberculosis detection dataset. The accuracy is used as the evaluation index for the brain MRI dataset and the tuberculosis detection dataset. Due to the long-tail distribution problem of the melanoma dataset, the average category accuracy is used as the evaluation index on this dataset to ensure that the final result will not be biased due to the difference in the number of samples of different categories. The experimental results are shown in Table 1.

[0142] Table 1 Experimental results of the proposed method and the selected comparative methods on three medical image datasets

[0143]

[0144] It can be seen from the experimental results that after the model is processed by the method (Ours) process of the present invention, the pre-trained model of the comparative text-image pair can capture the subtle differences between different categories in the medical image, enhance the perception of image details by the text prompts, and improve the understanding of medical images by the comparative language image pre-trained model, so that the model can more accurately identify and distinguish different medical image categories.

Claims

1. A method for medical image classification based on conditional category specific prompts, characterized in that: The following steps are involved: Step S1: Obtain a medical image dataset including K categories ,in, Indicates Medical images, represents the total number of medical images; let The true category label is ,and ∈{1,2,…,K}; The first Medical images Divide A fixed-size image block is formed, and each image block is flattened and then projected onto a fixed dimension, thereby The projected one-dimensional vectors are concatenated into image blocks for embedding ; Constructing Hints to Learn ,in, Indicates Tips to learn; Indicates the total number of prompts; Step S2: Construct a pre-trained model for comparing text-image pairs, including: image encoder , Text Encoder , wherein the image encoder is used to prompt and image patch embedding Process it and get Medical images The visual feature vector ; Step S3: Use formula (1) to obtain the average image feature vector of the kth category: : (1) In formula (1), represents the total number of categories, is the first Medical images The visual feature vector of is the total number of medical images in the kth category; Step S4: Build a lightweight network and Processing to get specific tips for the kth category ; Step S5: Category-specific tips |k=1,2,…,K} and the name of each category |k=1,2,…,K} after corresponding concatenation, input text encoder Processed in, get the text features { |k=1,2,…,K}; where represents the name of the k-th category, Represents the text features of the kth category; Step S6: and{ |k=1,2,…,K} to match, and select the category corresponding to the maximum probability as the first Medical images The prediction category of Step S7: Construct loss function , and train the pre-trained model and lightweight network of the contrasting text-image pair using the gradient descent method, and update the image encoder by gradient backpropagation Tips in and the parameter weights of the lightweight network until the loss function Until convergence, the optimal medical image classification model is obtained, which is used to classify medical images and output the predicted category.

2. The medical image classification method based on conditional category specific prompts according to claim 1, characterized in that: Image encoder in step S2 Includes: Category Embedding , Position Embedding , Layer Image Self-Attention Model , where each layer of the image self-attention model consists of a multi-head image self-attention layer and a multi-layer image perception machine block; Step S21: Initialization =1; define and initialize the -1 layer image self-attention model Output visual feature vector ; Initialize the self-attention layer of the input i-th layer Tips = ; Embed the image block As the i-th layer of self-attention Image patch embedding ; Initialize input to the first Self-attention layer Category embedding = ; Step S22: Prompt , Image Patch Embedding With category embedding After concatenation, embed with position Add and enter Self-attention layer In order to process it, the formula (2) is used to output the Layer image patch embedding With category embedding ; (2) In formula (2), is the first self-attention layer, and It is Category embedding and image patch embedding output by the self-attention layer; Step S221: Input the prompt of the i-th layer , Image Patch Embedding With category embedding After concatenation, layer normalization is performed to obtain the embedding vector sequence using formula (3): ; (3) Step S222, self-attention layer of the i-th layer The h self-attention heads of Processing is performed to obtain the self-attention layer of the i-th layer h multi-head attention representations , and then use formula (5) to get the self-attention layer of the i-th layer Comprehensive characterization of output : (4) (5) In formula (4)-formula (5), , , is the self-attention layer of the i-th layer No. The projection parameter matrix of the self-attention head, is also the projection parameter matrix, is the activation function, represents transpose, Represents a serial operation, Represents the self-attention layer of the i-th layer No. Multi-head attention representation of the output of the attention heads , is the number of parallel self-attention heads; Step S223: Use formula (6) to obtain the self-attention layer of the i-th layer Residual representation in : (6) In formula (6), represents residual connection; Step S224: Use formula (7) to obtain Self-attention layer Output image patch embedding With category embedding : (7) In formula (7), and are the two transformation matrices of the two linear layers, and are the two biases of the two linear layers, is the activation function; Representation layer normalization operation; Indicates Self-attention layer Output prompts; Step S225, after copying i+1 to i, return to step 221 and execute sequentially until i>J, thereby obtaining the J-layer self-attention layer { |i=1,2,…,J} The output image patch embedding { |i=1,2,…,J} and category embedding { |i=1,2,…,J}; Step S23: Use formula (8) to obtain Self-attention layer To Self-attention layer The self-attention layer of any kth layer in Output image patch embedding , Category Embedding and tips : (8) Step S24: Self-attention layer Output category embedding After projection, the medical image is obtained Visual features .

3. The medical image classification method based on conditional category specific prompts according to claim 1, characterized in that: The step S4 comprises: Step S4.1: Use formula (9) to obtain the intermediate features of the kth category : (9) In the formula, is the transformation matrix of the third linear layer, is the bias of the third linear layer; Step S4.2: Use formula (10) to obtain the specific prompt of the kth category : (10) In formula (10), is the transformation matrix of the fourth linear layer, is the bias of the fourth linear layer, is the activation function.

4. The medical image classification method based on conditional category specific prompts according to claim 1, characterized in that: The step S5 comprises: Step S5.1: Create a text encoder include: Layer text self-attention model, where each layer of text self-attention model consists of a multi-head text self-attention layer and a multi-layer text perception machine block; Step S5.2: The corresponding category name After concatenation, the k-th sentence text representation is obtained, thereby obtaining the K-sentence text representation; Step S5.3: After segmenting the K sentences, project the segmentation results onto a fixed dimension to obtain text embedding. ; Step S5.4: Initialization =1; define and initialize the -1 layer text self-attention model Output text embedding = ; Step S5.5: Input to Layer Text Self-Attention Model Processed in, output text embedding feature vector ; Step S5.5.1, Layer Text Self-Attention Model The h-th head text self-attention layer in the multi-head text self-attention layer uses formula (11) to obtain the h-th head text self-attention layer Layer Text Self-Attention Model The h-th head text attention representation , and then use formula (12) to get the Layer Text Self-Attention Model Comprehensive feature representation of the output of multi-head text self-attention layer : (11) (12) In formulas (11) and (12), , , It is Layer Text Self-Attention Model The three projection parameter matrices of the h-th head text self-attention layer, is the global projection parameter matrix, is the activation function, represents transpose, Represents a serial operation, is the number of parallel text self-attention heads; Step S5.5.2: Use formula (13) to get Layer Text Self-Attention Model The residual representation of : (13) In formula (13), represents residual connection; Step S5.5.3: Use formula (14) to obtain Layer Text Self-Attention Model The text embedding output by the multi-layer text perceptron block : (14) In formula (14), and They are the text transformation matrices of the two linear layers, and They are the text biases of the two linear layers, is the activation function; Step S5.5.4: +1 assigned to Then, return to step S5.5.2 and execute sequentially until >L, thus obtaining the Lth text self-attention layer Output text embedding ; Step S5.6: Embed text The last token of is projected onto a fixed-dimensional vector space to obtain the text feature vector { |k=1,2,…,K}.

5. The medical image classification method based on conditional category specific prompts according to claim 1, characterized in that: In step S6, equation (15) is used to obtain Medical images The probability of belonging to the kth category : (15) In formula (15), represents the cosine similarity, represents the exponential function, Represents the temperature parameter.

6. The medical image classification method based on conditional category specific prompts according to claim 1, characterized in that: In step S7, the loss function is constructed using formula (16): : (16) In formula (16), Represents a logarithmic function.

7. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the medical image classification method according to any one of claims 1 to 6, and the processor is configured to execute the program stored in the memory.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the medical image classification method according to any one of claims 1 to 6 are performed.

Images