A medical image classification method, device and medium based on prompt reverse learning

Abstract

The present invention discloses a medical image classification method, device and medium based on prompt reverse learning, including obtaining a data set, defining a mapping relationship between images and categories; constructing a pre-trained model for contrasting text-image pairs, extracting visual features of natural images; extracting visual features of medical images after adding prompts to an image encoder; constructing a reverse learning loss function; inputting the prompts into a text encoder through a pre-trained model to obtain text features; constructing a loss function to obtain a task-based objective function; using a gradient descent method to train the prompts in the image encoder of the pre-trained model, and calculating and updating the prompt weights, iterating multiple times to obtain the trained prompts, classifying medical images, and completing the medical image classification task. The present invention improves the ability and robustness of the natural image pre-trained model to generalize to the medical image classification task by establishing a connection between medical images and natural images.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence deep learning medical image classification, and in particular to a medical image classification method, device and medium based on prompt reverse learning. Background Art

[0002] As an emerging natural image and text semantic alignment model, the Contrastive Language Image Pre-training Model is trained on a massive dataset consisting of 400 million text-image pairs from the Internet. This extensive data corpus enables the model to acquire diverse and comprehensive knowledge, thus having strong zero-shot capabilities. Due to the scarcity of medical data, the Contrastive Language Image Pre-training Model has attracted attention in the field of medical imaging.

[0003] However, there are obvious differences between the field of medical images and the field of natural images. First of all, there are differences in imaging methods and image modes. Natural images are usually color three-channel images captured by cameras, while medical images are mostly single-channel grayscale images of radiography. Secondly, the similarity between medical images is high. This is because images in the same medical dataset often come from the same anatomical location of different patients, and human tissues are highly similar, which poses a challenge for pre-trained models to distinguish the differences between medical images. Finally, medical terminology is highly professional, and pre-trained models lack an understanding of clinical concepts.

[0004] These differences have a negative impact on the generalization of the contrastive language image pre-training model to medical image classification, making it difficult to directly apply large models pre-trained on natural images to the field of medical images. In theory, in order to solve this problem, a large amount of medical image data can be collected to train a large medical model. However, medical image data is relatively scarce, the cost of obtaining it is high, and it is often provided by professional medical institutions, which raises issues of personal privacy. This makes it challenging to train a large medical image model. Therefore, how to improve the robustness of the contrastive language image pre-training model to generalize to the field of medical images with limited medical data is an issue to be solved. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art. To achieve the above purpose, a medical image classification method, device and medium based on prompt reverse learning are adopted to solve the problems raised in the above background technology.

[0006] A medical image classification method based on prompt reverse learning comprises the following steps:

[0007] Step S1, obtaining a data set of natural images and medical images, and defining a mapping relationship between images and categories;

[0008] Step S2: construct a pre-trained model for comparing text-image pairs, and extract visual features of natural images through the image encoder of the pre-trained model;

[0009] Step S3, extracting visual features of the medical image after adding the prompt to the image encoder;

[0010] Step S4, constructing a reverse learning loss function according to the obtained visual features of the natural image and the medical image;

[0011] Step S5, setting the image text description as a prompt, and inputting the prompt into the text encoder of the pre-trained model to obtain text features;

[0012] Step S6: construct a loss function based on the obtained reverse learning loss function to obtain a task-based objective function;

[0013] Step S7, use the gradient descent method to train the prompts in the image encoder of the pre-trained model, and calculate the loss function to update the prompt weights through gradient backpropagation, iterate multiple times to obtain the trained prompts, classify the medical images, and complete the medical image classification task.

[0014] As a further technical solution of the present invention: the specific steps in step S1 include:

[0015] Step S11: categories of medical image datasets, and randomly assign categories from natural image datasets to represent the categories of medical images; define represents a medical image dataset, Represents a natural image dataset; defines a category mapping function , we get the mapping relationship between the categories of the medical image dataset and the categories in the natural image dataset. The formula is:

[0016] ;

[0017] In the formula, Represents the first Category names, Represents the natural image dataset Category name;

[0018] Step S12: for each medical image sample belonging to a specific class, a random sample in a natural image of the corresponding class is assigned to obtain a mapping pair between the images; an image mapping function is defined: , the formula is:

[0019]

[0020] In the formula, Represents the first Category images, Represents the natural image dataset Category images;

[0021] The mapping between medical images and natural images is obtained through image mapping function. .

[0022] As a further technical solution of the present invention: the specific steps in step S2 include:

[0023] Step S21: Establish an image encoder of a pre-trained model for comparing text-image pairs , the image encoder includes image block embedding, category embedding, position embedding, and The self-attention model of each layer consists of a multi-head self-attention layer and a multi-layer perceptron block;

[0024] Step S22: Natural image Divided into A fixed-size image block is flattened and projected to a fixed dimension to get the image block embedding ;

[0025] Step S23: Image block embedding With category embedding Concatenation, and positional embedding Add, input to The self-attention model of the layer The output of the self-attention layer is fed into the The self-attention layer is:

[0026] ;

[0027] In the formula, is the first self-attention layer, and It is Category embedding and image patch embedding output by the self-attention layer;

[0028] Step S231: Concatenated image block embedding and category embedding Perform layer normalization, the formula is:

[0029]

[0030] Step S232: The multi-head self-attention layer uses the following two formulas to Processing, the formula is:

[0031]

[0032]

[0033] In the formula, , , and is the projection parameter matrix, yes Activation function, represents transpose, is the vector dimension, Represents a serial operation, Indicates A head of attention, is the number of parallel self-attention heads, is the output of the multi-head self-attention layer;

[0034] Step S233: perform residual connection according to the following formula:

[0035]

[0036] Step S234, the multilayer perceptron block uses the following formula to obtain the first Output of the self-attention layer , the formula is:

[0037]

[0038] In the formula, and is the transformation matrix of the linear layer, and is the bias of the linear layer, yes Activation function;

[0039] Step S24: Category embedding of the last layer output of the self-attention model Visual features of natural images obtained through projection .

[0040] As a further technical solution of the present invention: the specific steps of adding prompts to the image encoder in step S3 include:

[0041] Step S31: The front of the self-attention model Layers build learnable hints ,in, Indicates Learnable hints for self-attention layers;

[0042] Step S32: Prompt and image patch embedding With category embedding Connect in series and use the following formula to get The output of the self-attention layer is:

[0043]

[0044] Step S33, use the following formula to get From the self-attention layer to the The output of the self-attention layer is:

[0045]

[0046] Step S34: Category embedding of the last layer output of the self-attention model Visual features of medical images obtained through projection .

[0047] As a further technical solution of the present invention: the reverse learning loss function in step S4 The specific formula is:

[0048]

[0049] In the formula, is the batch size, represents the cosine similarity, is the visual feature of medical images, It is the visual feature of natural images.

[0050] As a further technical solution of the present invention: the specific steps in step S5 include:

[0051] Step S51: Establish a text encoder for a pre-trained model of contrasting text-image pairs , the text encoder Includes text embedding and Layer-wise self-attention model, where each layer of self-attention model consists of a multi-head self-attention layer and a multi-layer perceptron block;

[0052] Use "an image of <category name>" as the prompt, fill in the space where the prompt needs to be supplemented with the natural image category corresponding to the medical image category, and input the text encoder of the pre-trained model for comparing text-image pairs , get text features , Represents the total number of categories in the medical image dataset;

[0053] Step S52: Fill in the natural image category name corresponding to the medical image category in the category name of the image text description. The total number of categories in the medical image dataset is ,get Sentence text representation;

[0054] Step S53: Segment the text expression and project the segmentation result to a fixed dimension to obtain text embedding ;

[0055] Step S54: Text embedding Input to The self-attention model of the layer The output of the self-attention layer is input to the Self-attention layer:

[0056]

[0057] In the formula, It is the first self-attention layer, It is The self-attention layer outputs text embedding;

[0058] Step S55: Text embedding of the output of the last self-attention layer The last token of is projected into a fixed-dimensional vector space to obtain text features .

[0059] As a further technical solution of the present invention: the specific steps in step S54 include:

[0060] Step S541, the multi-head self-attention layer uses the following two formulas to Processing, the formula is:

[0061]

[0062]

[0063] In the formula, , , and is the projection parameter matrix, yes Activation function, represents transpose, is the vector dimension, Represents a serial operation, Indicates A head of attention, is the number of parallel self-attention heads, is the output of the multi-head self-attention layer;

[0064] Step S542: Perform residual connection according to the following formula:

[0065]

[0066] Step S543, the multilayer perceptron block uses the following formula to obtain the first Output of the self-attention layer , the formula is:

[0067]

[0068] In the formula, and is the transformation matrix of the linear layer, and is the bias of the linear layer, yes Activation function.

[0069] As a further technical solution of the present invention: in step S6, a task-based objective function is constructed The specific formula is:

[0070]

[0071] In the formula, represents the cosine similarity, represents the exponential function, Represents the visual features of medical images, Indicates The text features of categories, is the text feature of the natural image category corresponding to the medical image, represents the temperature parameter, is the size of each batch, is the number of categories.

[0072] The technical solution of the second aspect: a device, further comprising:

[0073] at least one processor;

[0074] at least one memory for storing at least one program;

[0075] When the at least one program is executed by the at least one processor, the at least one processor implements a medical image classification method based on prompt reverse learning as described in any one of the above items.

[0076] The technical solution of the third aspect: a storage medium storing processor-executable instructions, wherein the processor-executable instructions are used to implement a medical image classification method based on prompt reverse learning as described in any one of the above items when executed by the processor.

[0077] Compared with the prior art, the present invention has the following technical effects:

[0078] By adopting the above technical solution, a method of prompt reverse learning is proposed to adapt the feature distribution of medical images to fixed upstream features, establish the connection between medical images and natural images, and make full use of the advantages of the contrastive language image pre-training model on natural images to improve its generalization performance to medical image classification tasks. The parameters of the contrastive language image pre-training model are maintained, and only the weights of the prompts are updated. When the amount of training parameters is small, the robustness of the pre-training model is enhanced; by mapping the specified natural image and its class name to represent the medical image and its class name of the specified class, the visual features of the medical image are aligned with the visual and text features of the natural image, reducing the negative impact caused by the huge difference between medical images and natural images. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings:

[0080] Figure 1 A schematic diagram of the steps of the medical image classification method disclosed in the present application;

[0081] Figure 2 This is an overall flow chart of the embodiments disclosed in this application. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] Please refer to Figure 1 and Figure 2 In an embodiment of the present invention, a medical image classification method based on prompt reverse learning includes the following steps:

[0084] Step S1: Obtain a data set of natural images and medical images, define a mapping relationship between images and categories, and establish a connection between natural images and medical images. The specific steps include:

[0085] In this embodiment, by designing a set of prompt reverse learning methods, the feature distribution of medical images is aligned with the fixed upstream feature distribution, so that the performance and robustness of the pre-trained model generalized to the task of medical image classification are significantly improved;

[0086] Specifically, natural images and their class names are specified to represent specific classes of medical images and class names:

[0087] Step S11: categories of medical image datasets, and randomly assign categories from natural image datasets to represent the categories of medical images; define represents a medical image dataset, Represents a natural image dataset; defines a category mapping function , we get the mapping relationship between the categories of the medical image dataset and the categories in the natural image dataset. The formula is:

[0088] ;

[0089] In the formula, Represents the first Category names, Represents the natural image dataset Category name;

[0090] In this embodiment, the natural image dataset used is the ImageNet dataset.

[0091] Step S12: for each medical image sample belonging to a specific class, a random sample in a natural image of the corresponding class is assigned to obtain a mapping pair between the images; an image mapping function is defined: , the formula is:

[0092]

[0093] In the formula, Represents the first Category images, Represents the natural image dataset Category images;

[0094] The mapping between medical images and natural images is obtained through image mapping function. .

[0095] Step S2: construct a pre-trained model for comparing text-image pairs, and extract visual features of natural images through the image encoder of the pre-trained model as anchor points for visual features of medical images. The specific steps include:

[0096] Step S21: Establish an image encoder of a pre-trained model for comparing text-image pairs , the image encoder includes image block embedding, category embedding, position embedding, and The self-attention model of each layer consists of a multi-head self-attention layer and a multi-layer perceptron block;

[0097] Step S22: Natural image Divided into A fixed-size image block is flattened and projected to a fixed dimension to get the image block embedding ;

[0098] Step S23: Image block embedding With category embedding Concatenation, and positional embedding Add, input to The self-attention model of the layer The output of the self-attention layer is fed into the The self-attention layer is:

[0099] ;

[0100] In the formula, is the first self-attention layer, and It is Category embedding and image patch embedding output by the self-attention layer;

[0101] Step S231: Concatenated image block embedding and category embedding Perform layer normalization, the formula is:

[0102]

[0103] Step S232: The multi-head self-attention layer uses the following two formulas to Processing, the formula is:

[0104]

[0105]

[0106] In the formula, , , and is the projection parameter matrix, yes Activation function, represents transpose, is the vector dimension, Represents a serial operation, Indicates A head of attention, is the number of parallel self-attention heads, is the output of the multi-head self-attention layer;

[0107] Step S233: perform residual connection according to the following formula:

[0108]

[0109] Step S234, the multilayer perceptron block uses the following formula to obtain the first Output of the self-attention layer , the formula is:

[0110]

[0111] In the formula, and is the transformation matrix of the linear layer, and is the bias of the linear layer, yes Activation function;

[0112] Step S24: Category embedding of the last layer output of the self-attention model Visual features of natural images obtained through projection .

[0113] Step S3, extracting visual features of the medical image after adding the prompt to the image encoder;

[0114] The specific steps of adding prompts to the image encoder include:

[0115] Step S31: The front of the self-attention model Layers build learnable hints ,in, Indicates Learnable hints for self-attention layers;

[0116] Step S32: Prompt and image patch embedding With category embedding Connect in series and use the following formula to get The output of the self-attention layer is:

[0117]

[0118] Step S33, use the following formula to get From the self-attention layer to the The output of the self-attention layer is:

[0119]

[0120] Step S34: Category embedding of the last layer output of the self-attention model Visual features of medical images obtained through projection .

[0121] Step S4, constructing a reverse learning loss function according to the obtained visual features of the natural image and the medical image, and quantifying the gap between the visual features of the medical image and the visual features of the natural image;

[0122] Among them, the reverse learning loss function The specific formula is:

[0123]

[0124] In the formula, is the batch size, represents the cosine similarity, is the visual feature of medical images, It is the visual feature of natural images.

[0125] Step S5: setting the image text description as a prompt, inputting the prompt into the text encoder of the pre-trained model to obtain text features, and the specific steps include:

[0126] Step S51: Establish a text encoder for a pre-trained model of contrasting text-image pairs , the text encoder Includes text embedding and Layer-wise self-attention model, where each layer of self-attention model consists of a multi-head self-attention layer and a multi-layer perceptron block;

[0127] Use "an image of <category name>" as the prompt, fill in the space where the prompt needs to be supplemented with the natural image category corresponding to the medical image category, and input the text encoder of the pre-trained model for comparing text-image pairs , get text features , Represents the total number of categories in the medical image dataset;

[0128] Step S52: Fill in the natural image category name corresponding to the medical image category in the category name of the image text description. The total number of categories in the medical image dataset is ,get Sentence text representation;

[0129] Step S53: Segment the text expression and project the segmentation result to a fixed dimension to obtain text embedding ;

[0130] Step S54: Text embedding Input to The self-attention model of the layer The output of the self-attention layer is input to the Self-attention layer:

[0131]

[0132] In the formula, It is the first self-attention layer, It is The self-attention layer outputs text embedding;

[0133] In this embodiment, the specific steps in step S54 include:

[0134] Step S541, the multi-head self-attention layer uses the following two formulas to Processing, the formula is:

[0135]

[0136]

[0137] In the formula, , , and is the projection parameter matrix, yes Activation function, represents transpose, is the vector dimension, Represents a serial operation, Indicates A head of attention, is the number of parallel self-attention heads, is the output of the multi-head self-attention layer;

[0138] Step S542: Perform residual connection according to the following formula:

[0139]

[0140] Step S543, the multilayer perceptron block uses the following formula to obtain the first Output of the self-attention layer , the formula is:

[0141]

[0142] In the formula, and is the transformation matrix of the linear layer, and is the bias of the linear layer, yes Activation function.

[0143] Step S55: Text embedding of the output of the last self-attention layer The last token of is projected into a fixed-dimensional vector space to obtain text features .

[0144] Step S6: Based on the obtained reverse learning loss function, a loss function is constructed to obtain a task-based objective function to guide the parameter update process of the model;

[0145] Among them, constructing a task-based objective function The specific formula is:

[0146]

[0147] In the formula, represents the cosine similarity, represents the exponential function, Represents the visual features of medical images, Indicates The text features of categories, is the text feature of the natural image category corresponding to the medical image, represents the temperature parameter, is the size of each batch, is the number of categories.

[0148] Step S7, use the gradient descent method to train the prompts in the image encoder of the pre-trained model, and calculate the loss function to update the prompt weights through gradient backpropagation, iterate multiple times to obtain the trained prompts, classify the medical images, and complete the medical image classification task.

[0149] In this embodiment, in order to verify the effectiveness of the present invention, five advanced prompt learning classification methods, namely COOP (Context Optimization), COCOOP (Conditional Context Optimization), RLM-VP (Random Label Mapping -Visual Prompt), VPT (Visual Prompt Tuning), and MAPLE (Multi-modal Prompt Learning), are selected for training on three medical image datasets to evaluate the performance of different method models. The three medical image datasets are brain MRI dataset, melanoma dataset, and bone abnormality detection dataset. The accuracy is used as the evaluation index for the brain MRI dataset and the bone abnormality detection dataset, and the average category accuracy is used as the evaluation index for the melanoma dataset. The experimental results are shown in Table 1.

[0150] Table 1 Experimental results of the proposed method and the selected flow comparison method on three medical image datasets

[0151]

[0152] It can be seen from the experimental results that after the model is processed by the method (ours) process of the present invention, the feature distribution of medical images can adapt to the fixed upstream features, and the advantages of the contrastive language image pre-training model can be fully utilized, so that the generalization performance and robustness of medical image classification are greatly improved. The present invention has achieved results that exceed the existing prompt learning methods on three medical image datasets.

[0153] The technical solution of the second aspect: a device, further comprising:

[0154] Contains at least one high-performance processor core, which has powerful data processing and computing capabilities and can efficiently execute complex algorithms;

[0155] At least one high-speed, large-capacity memory is provided, which is not only used to store massive amounts of data, but also has enough space to store at least one programmed program;

[0156] When the written programs are accurately executed by at least one of the above high-performance processors, they will activate a series of complex calculation processes, enabling the processor to intelligently implement a medical image classification method based on prompt reverse learning. By deeply mining the subtle features in the image and combining the reverse learning strategy, the classification model is continuously optimized to achieve high-precision classification of medical images such as X-rays, CT scans or MRI images, providing strong auxiliary support for medical diagnosis.

[0157] The technical solution of the third aspect: a storage medium storing processor-executable instructions, wherein the processor-executable instructions are used to implement a medical image classification method based on prompt reverse learning as described in any one of the above items when executed by the processor.

[0158] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0159] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents, and all should be included within the scope of protection of the present invention.

Claims

1. A medical image classification method based on prompt reverse learning, characterized in that: The following steps are involved: Step S1, obtaining a data set of natural images and medical images, and defining a mapping relationship between images and categories; Step S2: construct a pre-trained model for comparing text-image pairs, and extract visual features of natural images through the image encoder of the pre-trained model; Step S3, extracting visual features of the medical image after adding the prompt to the image encoder; Step S4, constructing a reverse learning loss function according to the obtained visual features of the natural image and the medical image; Step S5, setting the image text description as a prompt, and inputting the prompt into the text encoder of the pre-trained model to obtain text features; Step S6: construct a loss function based on the obtained reverse learning loss function to obtain a task-based objective function; Step S7, use the gradient descent method to train the prompts in the image encoder of the pre-trained model, and calculate the loss function to update the prompt weights through gradient backpropagation, iterate multiple times to obtain the trained prompts, classify the medical images, and complete the medical image classification task.

2. According to claim 1, a medical image classification method based on prompt reverse learning is characterized in that: The specific steps in step S1 include: Step S11: categories of medical image datasets, and randomly assign categories from natural image datasets to represent the categories of medical images; define represents a medical image dataset, Represents a natural image dataset; defines a category mapping function , we get the mapping relationship between the categories of the medical image dataset and the categories in the natural image dataset. The formula is: ； In the formula, Represents the first Category names, Represents the natural image dataset Category name; Step S12: for each medical image sample belonging to a specific class, a random sample in a natural image of the corresponding class is assigned to obtain a mapping pair between the images; an image mapping function is defined: , the formula is: In the formula, Represents the first Category images, Represents the natural image dataset Category images; The mapping between medical images and natural images is obtained through image mapping function. .

3. According to claim 1, a medical image classification method based on prompt reverse learning is characterized in that: The specific steps in step S2 include: Step S21: Establish an image encoder of a pre-trained model for comparing text-image pairs , the image encoder includes image block embedding, category embedding, position embedding, and The self-attention model of each layer consists of a multi-head self-attention layer and a multi-layer perceptron block; Step S22: Natural image Divided into A fixed-size image block is flattened and projected to a fixed dimension to get the image block embedding ; Step S23: Image block embedding With category embedding Concatenation, and positional embedding Add, input to The self-attention model of the layer The output of the self-attention layer is fed into the The self-attention layer is: ； In the formula, is the first self-attention layer, and It is Category embedding and image patch embedding output by the self-attention layer; Step S231: Concatenated image block embedding and category embedding Perform layer normalization, the formula is: Step S232: The multi-head self-attention layer uses the following two formulas to Processing, the formula is: In the formula, , , and is the projection parameter matrix, yes Activation function, represents transpose, is the vector dimension, Represents a serial operation, Indicates Attention head, is the number of parallel self-attention heads, is the output of the multi-head self-attention layer; Step S233: perform residual connection according to the following formula: Step S234, the multilayer perceptron block uses the following formula to obtain the first Output of the self-attention layer , the formula is: In the formula, and is the transformation matrix of the linear layer, and is the bias of the linear layer, yes Activation function; Step S24: Category embedding of the last layer output of the self-attention model Visual features of natural images obtained through projection .

4. According to claim 1, a medical image classification method based on prompt reverse learning is characterized in that: The specific steps of adding a prompt to the image encoder in step S3 include: Step S31: The front of the self-attention model Layers build learnable hints ,in, Indicates Learnable hints for self-attention layers; Step S32: Prompt and image patch embedding With category embedding In series, we can use the following formula to get The output of the self-attention layer is: Step S33, use the following formula to get From the self-attention layer to the The output of the self-attention layer is: Step S34: Category embedding of the last layer output of the self-attention model Visual features of medical images obtained through projection .

5. According to claim 1, a medical image classification method based on prompt reverse learning is characterized in that: The reverse learning loss function in step S4 The specific formula is: In the formula, is the batch size, represents the cosine similarity, is the visual feature of medical images, It is the visual feature of natural images.

6. According to claim 1, a medical image classification method based on prompt reverse learning is characterized in that: The specific steps in step S5 include: Step S51: Establish a text encoder for a pre-trained model of contrasting text-image pairs , the text encoder Includes text embedding and Layer-wise self-attention model, where each layer of self-attention model consists of a multi-head self-attention layer and a multi-layer perceptron block; Use "an image of <category name>" as the prompt, fill in the space where the prompt needs to be supplemented with the natural image category corresponding to the medical image category, and input the text encoder of the pre-trained model for comparing text-image pairs , get text features , Represents the total number of categories in the medical image dataset; Step S52: Fill in the natural image category name corresponding to the medical image category in the category name expressed in the image text. The total number of categories in the medical image dataset is ,get Sentence text representation; Step S53: Segment the text expression and project the segmentation result to a fixed dimension to obtain text embedding ; Step S54: Text embedding Input to The self-attention model of the layer The output of the self-attention layer is input to the Self-attention layer: In the formula, It is the first self-attention layer, It is The self-attention layer outputs text embedding; Step S55: Text embedding of the output of the last self-attention layer The last token of is projected into a fixed-dimensional vector space to obtain text features .

7. A medical image classification method based on prompt reverse learning according to claim 6, characterized in that: The specific steps in step S54 include: Step S541, the multi-head self-attention layer uses the following two formulas to Processing, the formula is: In the formula, , , and is the projection parameter matrix, yes Activation function, represents transpose, is the vector dimension, Represents a serial operation, Indicates Attention head, is the number of parallel self-attention heads, is the output of the multi-head self-attention layer; Step S542: Perform residual connection according to the following formula: Step S543, the multilayer perceptron block uses the following formula to obtain the first Output of the self-attention layer , the formula is: In the formula, and is the transformation matrix of the linear layer, and is the bias of the linear layer, yes Activation function.

8. The medical image classification method based on prompt reverse learning according to claim 1, characterized in that: In step S6, a task-based objective function is constructed. The specific formula is: In the formula, represents the cosine similarity, represents the exponential function, Represents the visual features of medical images, Indicates The text features of categories, is the text feature of the natural image category corresponding to the medical image, represents the temperature parameter, is the size of each batch, is the number of categories.

9. A device, characterized in that: Also includes: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the medical image classification method based on prompt reverse learning as described in any one of claims 1 to 8.

10. A storage medium storing instructions executable by a processor, characterized in that: The processor-executable instructions are used to implement a medical image classification method based on prompt reverse learning as described in any one of claims 1 to 8 when executed by the processor.

Images