Medical image classification method based on prior knowledge enhanced mask and alignment modeling

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to a medical image classification method based on priori knowledge enhanced masks and alignment modeling. Comprising the following steps: acquiring an image text pair data set and preprocessing the image text pair data set to obtain a preprocessed image text pair data set; performing preliminary training on a visual encoder and a text encoder according to the preprocessed image text pair data set to obtain a preliminarily trained visual encoder and a preliminarily trained text encoder; training the visual encoder for the second time in combination with the text encoder to obtain a trained visual encoder; classifying the medical images by adopting the trained visual encoder; the model provided by the invention can be better generalized to various downstream medical image recognition tasks, and the accuracy of the downstream medical image recognition tasks is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a medical image classification method based on prior knowledge enhanced mask and alignment modeling. Background Art

[0002] Powered by large-scale annotated natural image datasets, deep learning has achieved great success in the field of computer vision. However, in medical imaging, deep learning models require a large amount of annotated data and usually show poor generalization ability. Obtaining large-scale, high-quality annotated data is a major challenge in the medical field because it is time-consuming and expensive. Medical vision-language pre-training (Med-VLP) has emerged as a promising alternative method, which utilizes relatively easy-to-obtain medical reports as supervision signals to learn universal representations. This method effectively alleviates the problem of insufficient annotated data in downstream tasks and achieves impressive generalization performance.

[0003] Current Med-VLP methods can be divided into two main categories: report-supervised cross-modal alignment pre-training and reconstruction-based self-supervised pre-training. The former uses information-rich medical report-image pairs to align text and image features at the global and local levels. The latter uses joint image-text reconstruction for self-supervised pre-training, usually using methods such as MAE. Each method has its advantages: alignment pre-training captures the intrinsic connection between images and paired reports, enhancing the discriminative ability of representations; while reconstruction pre-training focuses on local details, promotes fine-grained representation learning and captures low-level modality information.

[0004] However, combining the strengths of both paradigms also brings challenges as each approach has its limitations. Alignment pre-training introduces the false negative problem due to traditional contrastive learning, where each X-ray image is forced to correspond to a specific report and other pairs in the same batch are treated as negative samples. This results in some semantically similar image-report pairs being treated as negative samples, resulting in suboptimal image-text representations. On the other hand, reconstruction pre-training treats all image patches equally, but the importance of each image patch is different, especially image patches containing key pathologies, which have richer visual features. This leads to insufficient capture of details of key pathological areas in medical images.

[0005] Therefore, designing a medical vision-language pre-training framework that can effectively utilize the advantages of both the reconstruction pre-training paradigm and the alignment pre-training paradigm is an important issue that needs to be solved. At the same time, this framework also needs to alleviate the shortcomings of these two pre-training paradigms, so as to learn better medical image information and improve the accuracy of downstream tasks. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a medical image classification method based on prior knowledge enhanced mask and alignment modeling, the method comprising:

[0007] S1: Obtain an image-text pair dataset and preprocess it to obtain a preprocessed image-text pair dataset;

[0008] S2: Preliminary training of the visual encoder and the text encoder is performed based on the preprocessed image-text pair dataset to obtain a preliminarily trained visual encoder and text encoder;

[0009] S3: Combine the text encoder to train the visual encoder for the second time to obtain a trained visual encoder;

[0010] S4: Use the trained visual encoder to classify medical images.

[0011] Preferably, the process of performing preliminary training on the visual encoder and the text encoder includes:

[0012] S21: downsample the images in the dataset to obtain high-resolution images and low-resolution images;

[0013] S22: Process the low-resolution image to obtain primary features; use a visual encoder to process the primary features to obtain visual features; input the visual features into a visual decoder for processing to obtain a prediction value of the low-resolution mask block;

[0014] S23: calculating a low-resolution reconstruction weighted loss according to the prediction value of the low-resolution mask block; processing the prediction value of the low-resolution mask block using a multi-layer perceptron to obtain a prediction value of the high-resolution mask block; calculating a high-resolution reconstruction weighted loss according to the prediction value of the high-resolution mask block;

[0015] S24: embedding the text paired with the image in the dataset to obtain text embedding features;

[0016] S25: Fusing visual features and text embedding features to obtain fused features; using a text encoder and a text tag classifier to process the fused features to obtain classification probability;

[0017] S26: Calculate the text reconstruction loss according to the classification probability; sum the low-resolution reconstruction weighted loss, the high-resolution reconstruction weighted loss and the text reconstruction loss to obtain the reconstruction loss;

[0018] S27: Adjust the parameters of the visual encoder and the text encoder according to the reconstruction loss to obtain the preliminarily trained visual encoder and text encoder.

[0019] Furthermore, the process of obtaining primary features includes: dividing the low-resolution image into multiple image patch blocks; randomly selecting 75% of the image patch blocks for masking to obtain a set of unmasked patch blocks and a set of masked patch blocks; performing Flatten operation and linear projection on the unmasked patch blocks to obtain embedded features; adding the embedded features to their corresponding position codes to obtain primary features.

[0020] Furthermore, the formula for calculating the weighted loss of low-resolution reconstruction is expressed as:

[0021]

[0022] Among them, L low represents the weighted loss of low-resolution reconstruction, N m represents the number of low-resolution mask blocks, W i * represents the weight of the i-th low-resolution mask block, I Pred,i Represents the predicted value of the i-th low-resolution mask block, I m,i represents the reconstructed label of the i-th low-resolution mask block.

[0023] Furthermore, the formula for calculating the text reconstruction loss is:

[0024]

[0025] Among them, L MLM represents the text reconstruction loss, M represents the length of the text token sequence, r i represents the i-th mask label, represents the classification probability of the i-th mask.

[0026] Preferably, the process of performing a second training on the initially trained visual encoder and text encoder includes:

[0027] A visual encoder and a text encoder are used to process the image and the text respectively to obtain global visual features and global text features; the contrastive learning loss is calculated based on the global visual features and the global text features;

[0028] Concatenate global visual features and global text features, input the concatenated features into the linear layer for processing, and obtain aggregated features;

[0029] A knowledge graph is constructed based on the medical knowledge base, and a graph attention network is used to process the knowledge graph to obtain the medical entity knowledge representation;

[0030] The medical entity knowledge representation and the aggregated features are integrated to obtain the final integrated features; the final integrated features are clustered to obtain multiple clusters and soft clustering labels of the final integrated features;

[0031] Define the cluster center features of each cluster, and calculate the probability that the global visual features and global text features belong to different clusters based on the cluster center features of each cluster;

[0032] The cross entropy loss is calculated based on the probability that the global visual features and global text features belong to different clusters and the soft cluster labels of the final fused features;

[0033] The parameters of the visual encoder are adjusted according to the contrastive learning loss and the cross entropy loss to obtain a trained visual encoder.

[0034] Furthermore, the formula for calculating contrastive learning loss is:

[0035]

[0036] Among them, L ITC represents contrastive learning loss, N represents the number of samples, represents the cross entropy loss from the i-th sample image to text, Represents the cross entropy loss from the i-th sample text to image.

[0037] Furthermore, the formula for calculating the cross entropy loss is:

[0038]

[0039] Among them, L CA represents the cross entropy loss, N represents the number of samples, C represents the number of clusters, Indicates that the fusion feature of the i-th sample belongs to the soft clustering label of the k-th cluster, p v,i (k) represents the probability that the global image feature of the i-th sample belongs to the k-th cluster, p t,i (k) represents the probability that the global text feature of the i-th sample belongs to the k-th cluster.

[0040] The beneficial effects of the present invention are as follows: the present invention utilizes the weights of the most advanced medical pre-training model to obtain the importance score of each image block, enhances the model's ability to capture details of key pathological areas, and overcomes the limitation of treating image blocks equally. In addition, it is proposed to use medical expert knowledge as an external supervision signal to enhance the recognition of visual and text features of different diseases, improve cross-modal clustering alignment performance, and reduce the negative impact of false negative samples. Through these two methods, the pre-training model fully combines the advantages of the alignment pre-training paradigm and the reconstruction pre-training paradigm, and alleviates the shortcomings of the two paradigms. The model can be better generalized to various downstream medical image recognition tasks, thereby improving the accuracy of downstream medical image recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1This is a flow chart of the image classification method based on prior knowledge enhanced mask and alignment modeling in the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] This paper proposes a medical image classification method based on prior knowledge enhanced mask and alignment modeling, such as Figure 1 As shown, the method includes:

[0044] S1: Obtain an image-text pair dataset and preprocess it to obtain a preprocessed image-text pair dataset.

[0045] Obtain an image-text pair dataset, such as a medical CXR image-text pair dataset; preprocess the dataset, including random cropping, random horizontal flipping, random affine transformation, and randomly changing the brightness and contrast of the image; and use a large language model to expand the text details and clean some insignificant word symbols in the text to obtain a preprocessed image-text pair dataset.

[0046] S2: Perform preliminary training on the visual encoder and the text encoder according to the preprocessed image-text pair dataset to obtain preliminary trained visual encoder and text encoder.

[0047] S21: Downsample the images in the dataset to obtain high-resolution images and low-resolution images.

[0048] S22: Process the low-resolution image to obtain primary features; use a visual encoder to process the primary features to obtain visual features; input the visual features into a visual decoder for processing to obtain a prediction value of the low-resolution mask block.

[0049] Process low-resolution images, specifically:

[0050] Divide the low-resolution image into N p Image patches of size P×P are randomly selected to mask 75% of the image patches to obtain the unmasked patch set. And the masked patch block set N u is the number of unmasked Patch blocks, N mis the number of masked patches, i.e. the number of low-resolution mask patches. Flatten and linearly project the unmasked patches to obtain the embedded feature E. p ; Embed feature E p The corresponding position code E pos Add together to get the primary feature I V .

[0051] Using visual encoder E I (12-layer VIT / B-16) processes the primary features to obtain visual features F v , the process can be expressed as F v =E I (I V ).

[0052] The visual features are input into the visual decoder for processing. Specifically: During the image decoding process, the learnable [MASK] vector is used to restore N p After the original size of the visual feature F v Input visual decoder D I , using the position index table Index of the masked Patch block, the predicted value of the low-resolution mask block is obtained:

[0053] I Pred =D I (F v )[Index]

[0054] S23: calculating a low-resolution reconstruction weighted loss according to the prediction value of the low-resolution mask block; processing the prediction value of the low-resolution mask block using a multi-layer perceptron to obtain a prediction value of the high-resolution mask block; and calculating a high-resolution reconstruction weighted loss according to the prediction value of the high-resolution mask block.

[0055] Traditional reconstruction pre-training treats all image patches equally, but the present invention considers that the importance of each patch is different, especially for patches containing key pathological areas, which have richer visual features. In order to make the model learn the visual features of key pathological areas more effectively, it is necessary to give greater weight to patches containing pathological areas. First, the ScispaCy tool is used to extract disease-related medical terms from the text T paired with the image. If these terms are not available, the vector is set All elements are equal to 0, indicating that all patches are treated equally. If a disease noun is detected, the weight vector A is generated by loading publicly available pre-trained weights using the current state-of-the-art methods. The element values ​​of A are between 0 and 1, and then these values ​​are linearly scaled to generate the final weight map W. The process can be mathematically represented as follows:

[0056] W=α·A+β

[0057] Where α is the scaling factor and β is the offset. This scaling adjusts the weight values ​​to enhance the importance of each patch according to its attention score, thereby improving the reconstruction pre-training performance by paying more attention to important patches.

[0058] The reconstruction weight W is calculated based on the weight map and the index of the masked Patch block. * :

[0059] W * =W[Index]

[0060] The low-resolution reconstruction weighted loss is calculated based on the reconstruction weight, the predicted value of the low-resolution mask block, and the original value of the low-resolution mask block:

[0061]

[0062] Among them, L low represents the weighted loss of low-resolution reconstruction, N m represents the number of low-resolution mask blocks, W i * represents the weight of the i-th low-resolution mask block, I Pred,i Represents the predicted value of the i-th low-resolution mask block, I m,i represents the reconstructed label of the i-th low-resolution mask block.

[0063] Extracting features from low-resolution images often results in severe information loss, especially in small lesion areas, which leads to poor performance on visual tasks. High-resolution images can help the visual encoder extract more comprehensive and accurate feature representations, thus improving the performance of visual tasks. Therefore, not only low-resolution images but also high-resolution images can be reconstructed.

[0064] In order to restore the high-resolution patch, a multi-layer perceptron (MLP) is used, including two fully connected layers and an activation function, to process the predicted value of the low-resolution mask patch to obtain the predicted value of the high-resolution mask patch. The process is expressed as:

[0065] I restored =MLP(I Pred )

[0066] The high-resolution reconstruction weighted loss is calculated based on the predicted value of the high-resolution mask block. Specifically: s Divided into N p Using Index, we get high-resolution patches. The formula for calculating the weighted loss of high-resolution reconstruction is:

[0067]

[0068] Among them, L high represents the weighted loss for high-resolution reconstruction, N′ m represents the number of high-resolution mask blocks, W i ′ represents the weight of the i-th high-resolution mask block, I restored,i Represents the predicted value of the i-th high-resolution mask block, I′ m,i represents the reconstructed label of the i-th high-resolution mask block.

[0069] The overall optimization goal of the image reconstruction process is defined as the sum of the reconstruction losses of the low-resolution and high-resolution images L MIM This comprehensive loss ensures that the model effectively reconstructs features at multiple resolutions. The optimization objective can be expressed as:

[0070] L MIM =L high +L low

[0071] S24: Embed the text paired with the image in the dataset to obtain text embedding features.

[0072] Use WordPiece to convert the text T into a token sequence R = {r 1 ,r 2 ,...,r M}, M represents the length of the token sequence, 70% of R is masked to obtain the masked part Unmasked part The purpose of using a high masking ratio is to make the text reconstruction task more challenging, thereby encouraging the visual encoder to extract more informative representations for text reconstruction.

[0073] Insert [MASK] at the corresponding position of the mask to keep the original size of R, and then calculate the text embedding feature F by projecting the text token and adding the encoded position information t .

[0074] S25: The visual features and the text embedding features are fused to obtain fused features; the text encoder and the text tag classifier process the fused features to obtain classification probabilities.

[0075] Use a multimodal fusion module (containing a multi-head cross attention block and a multi-layer perceptron) to fuse the visual features F v and text features F t Get the fusion feature F fThe fusion features are processed using the text encoder and text tag classifier to obtain the classification probability R * , expressed as:

[0076] R * =D t (E t (F t ))

[0077] S26: Calculate the text reconstruction loss according to the classification probability; sum the low-resolution reconstruction weighted loss, the high-resolution reconstruction weighted loss and the text reconstruction loss to obtain the reconstruction loss.

[0078] The formula for calculating text reconstruction loss is:

[0079]

[0080] Among them, L MLM represents the text reconstruction loss, M represents the length of the text token sequence, r i represents the i-th mask label, represents the classification probability of the i-th mask.

[0081] The reconstruction loss is obtained by summing the low-resolution reconstruction weighted loss, the high-resolution reconstruction weighted loss and the text reconstruction loss:

[0082] L res =L MIM +L MLM

[0083] S27: Adjust the parameters of the visual encoder and the text encoder according to the reconstruction loss to obtain the preliminarily trained visual encoder and text encoder.

[0084] During the entire reconstruction phase, the visual encoder and text encoder are optimized according to the loss function, using the reconstruction loss L res Capture the local details of the image and obtain the preliminarily trained visual encoder and text encoder.

[0085] S3: Combine the text encoder to train the visual encoder for the second time to obtain a trained visual encoder.

[0086] The visual encoder and text encoder are used to process the image and text respectively to obtain the global visual feature v and the global text feature t. Two nonlinear visual and text projection layers P are used respectively. v and P t Convert v and t to the same dimension d, where v = P v (v) and t = P t (t).

[0087] To effectively capture the mutual and internal similarities between paired medical images and reports, we incorporate a contrastive learning objective, which encourages the framework to map correct image-report pairs closer together in the latent space while moving mismatched pairs further apart.

[0088] The contrastive learning loss is calculated based on the global visual features and global text features. Specifically:

[0089]

[0090] where exp(·) represents the natural exponential function, σ is the temperature hyperparameter controlling the sharpness of the distribution, T represents the transpose operation, and L ITC represents the overall contrastive learning loss, N represents the number of samples, represents the cross entropy loss from the i-th sample image to text, represents the cross entropy loss from the i-th sample text to the image, L ITC Minimize the distance between similar image and text pairs, while maximizing the distance between different image and text pairs, v i represents the i-th global visual feature, t j represents the jth global text feature.

[0091] By optimizing L ITC ,The model learns to maximize the mutual information between matching image-text pairs, capturing both mutual and internal similarities between paired medical images and reports.

[0092] For a given pair of X-ray images and reports, traditional contrastive learning methods treat other pairs in the same batch as negative samples, forcing each X-ray image to correspond to a specific text report. This approach inevitably leads to the semantic overlap problem, where pairs sharing highly similar semantic information are treated as negatives. Relying solely on global image-text alignment, cross-modal representations of the same disease may be incorrectly separated, preventing the model from capturing the corresponding image features of the disease. To reduce the impact of these false negatives, MGCA clusters image features and text features separately. It then performs cluster alignment to group samples with very similar meanings together, ensuring that they are close in feature space. Existing clustering methods, although designed differently, all generate supervisory signals internally. Despite their success, this internally guided clustering approach has inherent limitations. For example, pneumonia and lung cancer both show similar shadows on images, so they may be incorrectly grouped together. For example, Pneumonia and pneumthorax have similar text features but different image features, resulting in clustering errors when clustering text. Internal clustering methods introduce noise during cluster alignment, making optimization very difficult.

[0093] In the medical knowledge base, the definitions of different medical entities are different. Using expert knowledge as an external supervisory signal can make the visual and text features of different diseases clearer and improve the clustering performance.

[0094] Specific:

[0095] First, t concatenates the global visual features and global text features along the feature dimension, and inputs the concatenated features into the linear layer for processing to obtain the aggregated features Q. The purpose of this fusion operation is to distinguish samples with similar unimodal features. For example, although pneumonia and lung cancer have similar visual features, their text features are different. Through fusion, the features of pneumonia samples will be distinguished from those of lung cancer.

[0096] In order to ensure that samples with similar bimodal features also remain distinguishable, the present invention uses medical expert knowledge as an external supervisory signal; preferably, a knowledge graph G is constructed based on a medical knowledge base (such as UMLS), and a graph attention network is used to process the knowledge graph, aggregating local information from the neighborhood of each node to obtain a medical entity knowledge representation K.

[0097] The feature fusion module (including a multi-head cross attention mechanism and two Transformer blocks) is used to fuse the medical entity knowledge representation K (K is used as the key and value of the multi-head attention mechanism) and the aggregated feature Q (Q is used as the query of the multi-head attention mechanism) to obtain the final fused feature H. The fusion process can be expressed as follows:

[0098] H=Transformer(MCA(LN(query,key,value)))

[0099] Among them, LN(·) represents layer normalization, MCA is multi-head cross attention, and the MCA mechanism can be expressed as follows:

[0100]

[0101] Among them, T represents the transpose operation of the matrix, and d is the dimension size of the feature.

[0102] The fused feature H has higher significance and better clustering performance. The iterative Sinkhorn-Knopp clustering algorithm is used to cluster the final fused feature H to obtain multiple clusters and the soft clustering label L∈R of the final fused feature. N×C , where C is the number of clusters and N is the number of samples.

[0103] Define the cluster center features of each cluster, and calculate the probability that the global visual features and global text features belong to different clusters based on the cluster center features of each cluster. Specifically: pre-define C trainable cluster center features R = (r1 ,r 2 ,r 3 ,r 4 ........), calculate the cosine similarity between the global image feature v and all cluster center features in R, and after the SoftMax operation, obtain the probability p that the image feature belongs to each cluster cluster v Similarly, we calculate the cosine similarity between the global text feature t and all cluster center features in R, and obtain the probability p that the text feature belongs to each cluster cluster t .

[0104] The cross entropy loss is calculated based on the probability that the global visual features and global text features belong to different clusters and the soft clustering labels of the final fused features, so that the sample features belonging to the same cluster will be pulled closer, while the sample features belonging to different clusters will be pushed apart, thereby guiding the cross-modal cluster alignment; the formula for calculating the cross entropy loss is:

[0105]

[0106] Among them, L CA represents the cross entropy loss, N represents the number of samples, C represents the number of clusters, Indicates that the fusion feature of the i-th sample belongs to the soft clustering label of the k-th cluster, p v,i (k) represents the probability that the global image feature of the i-th sample belongs to the k-th cluster, p t,i (k) represents the probability that the global text feature of the i-th sample belongs to the k-th cluster.

[0107] Contrastive learning loss is added to cross entropy loss to get contrastive loss L con :

[0108] L con =L ITC +L CA

[0109] Using contrastive loss L con Continue to optimize the visual encoder obtained from the reconstruction stage. When the loss function converges or reaches a predetermined number of iterations, save the optimal visual encoder parameters to obtain a trained visual encoder. This helps the visual encoder capture the intrinsic relationship between the image and its paired report and learn high-level semantic information.

[0110] S4: Use the trained visual encoder to classify medical images.

[0111] The visual encoder trained by the above process can be used for various visual tasks, such as target detection, image classification, image segmentation, etc. If the trained visual encoder is used to classify medical images, more accurate medical image classification results can be obtained.

[0112] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A medical image classification method based on prior knowledge enhanced mask and alignment modeling, characterized in that: include: S1: Obtain an image-text pair dataset and preprocess it to obtain a preprocessed image-text pair dataset; S2: Reconstruct and pre-train the visual encoder and text encoder according to the pre-processed image-text pair dataset to obtain the preliminarily trained visual encoder and text encoder; S3: Combine the text encoder to perform comparative pre-training on the visual encoder to obtain a trained visual encoder; S4: Use the trained visual encoder to classify medical images.

2. The medical image classification method based on prior knowledge enhanced mask and alignment modeling according to claim 1, characterized in that: The initial training process for the visual encoder and text encoder includes: S21: downsample the images in the dataset to obtain high-resolution images and low-resolution images; S22: Process the low-resolution image to obtain primary features; use a visual encoder to process the primary features to obtain visual features; input the visual features into a visual decoder for processing to obtain a prediction value of the low-resolution mask block; S23: calculating a low-resolution reconstruction weighted loss according to the prediction value of the low-resolution mask block; processing the prediction value of the low-resolution mask block using a multi-layer perceptron to obtain a prediction value of the high-resolution mask block; calculating a high-resolution reconstruction weighted loss according to the prediction value of the high-resolution mask block; S24: embedding the text paired with the image in the dataset to obtain text embedding features; S25: Fusing visual features and text embedding features to obtain fused features; using a text encoder and a text tag classifier to process the fused features to obtain classification probability; S26: Calculate the text reconstruction loss according to the classification probability; sum the low-resolution reconstruction weighted loss, the high-resolution reconstruction weighted loss and the text reconstruction loss to obtain the reconstruction loss; S27: Adjust the parameters of the visual encoder and the text encoder according to the reconstruction loss to obtain the preliminarily trained visual encoder and text encoder.

3. The medical image classification method based on prior knowledge enhanced mask and alignment modeling according to claim 2, characterized in that: The process of obtaining primary features includes: dividing the low-resolution image into multiple image patches; randomly selecting 75% of the image patches for masking to obtain a set of unmasked patches and a set of masked patches; performing a Flatten operation and linear projection on the unmasked patches to obtain embedded features; and adding the embedded features to their corresponding position codes to obtain primary features.

4. The medical image classification method based on prior knowledge enhanced mask and alignment modeling according to claim 2, characterized in that: The formula for calculating the weighted loss of low-resolution reconstruction is expressed as: Among them, L low represents the weighted loss of low-resolution reconstruction, N m represents the number of low-resolution mask blocks, represents the weight of the i-th low-resolution mask block, I Pred,i Represents the predicted value of the i-th low-resolution mask block, I m,i represents the reconstructed label of the i-th low-resolution mask block.

5. The medical image classification method based on prior knowledge enhanced mask and alignment modeling according to claim 2, characterized in that: The formula for calculating text reconstruction loss is: Among them, L MLM represents the text reconstruction loss, M represents the length of the text token sequence, r i represents the i-th mask label, represents the classification probability of the i-th mask.

6. The medical image classification method based on prior knowledge enhanced mask and alignment modeling according to claim 1, characterized in that: The process of training the preliminarily trained visual encoder and text encoder for the second time includes: A visual encoder and a text encoder are used to process the image and the text respectively to obtain global visual features and global text features; the contrastive learning loss is calculated based on the global visual features and the global text features; Concatenate global visual features and global text features, input the concatenated features into the linear layer for processing, and obtain aggregated features; A knowledge graph is constructed based on the medical knowledge base, and a graph attention network is used to process the knowledge graph to obtain the medical entity knowledge representation; The medical entity knowledge representation and the aggregated features are integrated to obtain the final integrated features; the final integrated features are clustered to obtain multiple clusters and soft clustering labels of the final integrated features; Define the cluster center features of each cluster, and calculate the probability that the global visual features and global text features belong to different clusters based on the cluster center features of each cluster; The cross entropy loss is calculated based on the probability that the global visual features and global text features belong to different clusters and the soft cluster labels of the final fused features; The parameters of the visual encoder are adjusted according to the contrastive learning loss and the cross entropy loss to obtain a trained visual encoder.

7. The medical image classification method based on prior knowledge enhanced mask and alignment modeling according to claim 6, characterized in that: The formula for calculating contrastive learning loss is: Among them, L ITC represents contrastive learning loss, N represents the number of samples, represents the cross entropy loss from the i-th sample image to text, Represents the cross entropy loss from the i-th sample text to image.

8. The medical image classification method based on prior knowledge enhanced mask and alignment modeling according to claim 6, characterized in that: The formula for calculating the cross entropy loss is: Among them, L CA represents the cross entropy loss, N represents the number of samples, C represents the number of clusters, Indicates that the fusion feature of the i-th sample belongs to the soft clustering label of the k-th cluster, p v,i (k) represents the probability that the global image feature of the i-th sample belongs to the k-th cluster, p t,i (k) represents the probability that the global text feature of the i-th sample belongs to the k-th cluster.

Images