A Medical Image Classification Method Based on Prior Knowledge-Enhanced Masking and Alignment Modeling

Abstract

This invention belongs to the field of computer vision technology, specifically relating to a medical image classification method based on prior knowledge-enhanced masking and alignment modeling. The method includes: acquiring an image-text pair dataset and preprocessing it to obtain a preprocessed image-text pair dataset; performing preliminary training on a visual encoder and a text encoder based on the preprocessed image-text pair dataset to obtain a pre-trained visual encoder and a text encoder; performing a second training on the visual encoder using the text encoder to obtain a trained visual encoder; and using the trained visual encoder to classify medical images. The model proposed in this invention can better generalize to various downstream medical image recognition tasks, improving the accuracy of downstream medical image recognition tasks.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a medical image classification method based on prior knowledge-enhanced masking and alignment modeling. Background Technology

[0002] Deep learning has achieved tremendous success in computer vision, supported by large-scale labeled natural image datasets. However, in medical imaging, deep learning models require vast amounts of labeled data and often exhibit poor generalization ability. Obtaining large-scale, high-quality labeled data is a significant challenge in the medical field, as it is both time-consuming and expensive. Medical Vision-Language Pre-training (Med-VLP) has emerged as a promising alternative, utilizing relatively readily available medical reports as supervisory signals to learn general representations. This approach effectively alleviates the problem of insufficient labeled data in downstream tasks and achieves impressive generalization performance.

[0003] Current Med-VLP methods are mainly divided into two categories: report-supervised cross-modal alignment pre-training and reconstruction-based self-supervised pre-training. The former aligns text and image features at both global and local levels by using information-rich medical report-image pairs. The latter employs joint image-text reconstruction for self-supervised pre-training, typically using methods such as MAE. Each method has its advantages: alignment pre-training captures the intrinsic relationship between images and paired reports, enhancing the discriminative power of the representation; while reconstruction pre-training focuses on local details, promoting fine-grained representation learning and capturing low-level modal information.

[0004] However, combining the advantages of both paradigms also presents challenges, as each method has its limitations. Alignment pre-training introduces false negatives due to the inherent limitations of traditional contrastive learning, where each X-ray image is forced to correspond to a specific report, while other pairs in the same batch are treated as negative samples. This results in some semantically similar image-report pairs being treated as negative samples, leading to suboptimal image-text representations. On the other hand, reconstruction pre-training treats all image patches equally, but the importance of each patch varies, especially those containing key pathological areas, which possess richer visual features. This leads to insufficient detail capture of key pathological regions in medical images.

[0005] Therefore, designing a medical vision-language pre-training framework that effectively utilizes the advantages of both the reconstruction and alignment pre-training paradigms is a crucial problem that urgently needs to be solved. Simultaneously, this framework needs to mitigate the shortcomings of these two pre-training paradigms to learn better medical image information and improve the accuracy of downstream tasks. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a medical image classification method based on prior knowledge-enhanced masking and alignment modeling. This method includes:

[0007] S1: Obtain the image-text pair dataset and preprocess it to obtain the preprocessed image-text pair dataset;

[0008] S2: Based on the preprocessed image-text pair dataset, perform preliminary training on the visual encoder and text encoder to obtain the pre-trained visual encoder and text encoder;

[0009] S3: Combine the text encoder to perform a second training on the visual encoder to obtain a trained visual encoder;

[0010] S4: Use a trained visual encoder to classify medical images.

[0011] Preferably, the initial training process for the visual encoder and text encoder includes:

[0012] S21: Downsample the images in the dataset to obtain high-resolution and low-resolution images;

[0013] S22: Process the low-resolution image to obtain primary features; use a visual encoder to process the primary features to obtain visual features; input the visual features into a visual decoder for processing to obtain the predicted value of the low-resolution mask block.

[0014] S23: Calculate the low-resolution reconstruction weighted loss based on the predicted value of the low-resolution mask block; process the predicted value of the low-resolution mask block using a multilayer perceptron to obtain the predicted value of the high-resolution mask block; calculate the high-resolution reconstruction weighted loss based on the predicted value of the high-resolution mask block.

[0015] S24: Embed the text paired with the images in the dataset to obtain text embedding features;

[0016] S25: Combine visual features and text embedding features to obtain fused features; use a text encoder and a text tag classifier to process the fused features to obtain classification probabilities;

[0017] S26: Calculate the text reconstruction loss based on the classification probability; sum the low-resolution reconstruction weighted loss, high-resolution reconstruction weighted loss, and text reconstruction loss to obtain the reconstruction loss;

[0018] S27: Adjust the parameters of the visual encoder and text encoder according to the reconstruction loss to obtain the initially trained visual encoder and text encoder.

[0019] Furthermore, the process of obtaining primary features includes: dividing the low-resolution image into multiple image patches; randomly selecting 75% of the image patches for masking to obtain a set of unmasked patches and a set of masked patches; performing a Flatten operation and linear projection on the unmasked patches to obtain embedded features; and adding the embedded features to their corresponding positional codes to obtain primary features.

[0020] Furthermore, the formula for calculating the weighted loss of low-resolution reconstruction is expressed as follows:

[0021]

[0022] Among them, L low N represents the low-resolution reconstruction weighted loss. m W represents the number of low-resolution mask blocks. i * I represents the weight of the i-th low-resolution mask block. Pred,i I represents the predicted value of the i-th low-resolution mask block. m,i This represents the reconstructed label of the i-th low-resolution mask block.

[0023] Furthermore, the formula for calculating the text reconstruction loss is:

[0024]

[0025] Among them, L MLM Let r represent the text reconstruction loss, M represent the length of the text token sequence, and r represent the text reconstruction loss. i This represents the i-th mask label. This represents the classification probability of the i-th mask.

[0026] Preferably, the process of performing a second training on the initially trained visual encoder and text encoder includes:

[0027] A visual encoder and a text encoder are used to process the image and text respectively to obtain global visual features and global text features; the contrastive learning loss is calculated based on the global visual features and global text features.

[0028] Global visual features and global text features are concatenated, and the concatenated features are input into a linear layer for processing to obtain aggregated features;

[0029] A knowledge graph is constructed based on a medical knowledge base, and a graph attention network is used to process the knowledge graph to obtain a representation of medical entity knowledge.

[0030] The medical entity knowledge representation and aggregation features are integrated to obtain the final fused features; the final fused features are then clustered to obtain multiple clusters and soft clustering labels for the final fused features.

[0031] Define the cluster center features for each cluster, and calculate the probability that the global visual features and global text features belong to different clusters based on the cluster center features of each cluster.

[0032] The cross-entropy loss is calculated based on the probability that global visual features and global text features belong to different clusters and the soft clustering labels of the final fused features.

[0033] The parameters of the visual encoder are adjusted by comparing the contrastive learning loss and the cross-entropy loss to obtain a trained visual encoder.

[0034] Furthermore, the formula for calculating the contrastive learning loss is:

[0035]

[0036] Among them, L ITC This represents the contrastive learning loss, where N represents the number of samples. This represents the cross-entropy loss from the i-th sample image to the text. Let represent the cross-entropy loss from the text to the image for the i-th sample.

[0037] Furthermore, the formula for calculating the cross-entropy loss is:

[0038]

[0039] Among them, L CA The cross-entropy loss represents the number of samples, and C represents the number of clusters. p represents the soft clustering label indicating that the fusion feature of the i-th sample belongs to the k-th cluster. v,i (k) represents the probability that the global image features of the i-th sample belong to the k-th cluster, p t,i (k) represents the probability that the global text features of the i-th sample belong to the k-th cluster.

[0040] The beneficial effects of this invention are as follows: This invention utilizes the weights of a state-of-the-art medical pre-trained model to obtain an importance score for each image patch, enhancing the model's ability to capture details in key pathological regions and overcoming the limitation of treating image patches equally. Furthermore, it proposes using medical expert knowledge as an external supervisory signal to improve the discriminative power of visual and textual features of different diseases, improve cross-modal clustering alignment performance, and reduce the negative impact of false negative samples. Through these two methods, the pre-trained model fully combines the advantages of the alignment pre-training paradigm and the reconstruction pre-training paradigm, while mitigating the shortcomings of both paradigms. The model can better generalize to various downstream medical image recognition tasks, improving the accuracy of downstream medical image recognition tasks. Attached Figure Description

[0041] Figure 1This is a flowchart of the image classification method based on prior knowledge enhancement masking and alignment modeling in this invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] This invention proposes a medical image classification method based on prior knowledge-enhanced masking and alignment modeling, such as... Figure 1 As shown, the method includes:

[0044] S1: Obtain the image-text pair dataset and preprocess it to obtain the preprocessed image-text pair dataset.

[0045] Obtain an image-text pair dataset, such as a medical CXR image-text pair dataset; preprocess the dataset, including random cropping, random horizontal flipping, random affine transformation, and random changes to the brightness and contrast of the images; and use a large language model to expand the text details and clean up some irrelevant words and symbols in the text to obtain a preprocessed image-text pair dataset.

[0046] S2: Based on the preprocessed image-text pair dataset, perform preliminary training on the visual encoder and text encoder to obtain the pre-trained visual encoder and text encoder.

[0047] S21: Downsample the images in the dataset to obtain high-resolution and low-resolution images.

[0048] S22: Process the low-resolution image to obtain primary features; use a visual encoder to process the primary features to obtain visual features; input the visual features into a visual decoder for processing to obtain the predicted value of the low-resolution mask block.

[0049] Processing low-resolution images, specifically:

[0050] Divide the low-resolution image into N p Given a P×P image patch, randomly select 75% of the image patch blocks for masking to obtain a set of unmasked patch blocks. and the set of masked patch blocks N u N is the number of unmasked patch blocks. mThe number of masked patch blocks is the same as the number of low-resolution mask blocks. Flattening and linear projection are performed on the unmasked patch blocks to obtain the embedding features E. p ; embed feature E p Its corresponding position code E pos Adding them together yields the primary feature I. V .

[0051] Using visual encoder E I (12-layer VIT / B-16) processes the primary features to obtain visual features F. v The process can be represented as F v =E I (I V ).

[0052] Visual features are input into a visual decoder for processing. Specifically, during image decoding, learnable [MASK] vectors are used to recover N. p After determining the original size, the visual features F v Input visual decoder D I Using the index table of the masked patch block, the predicted value of the low-resolution mask block is obtained:

[0053] I Pred =D I (F v [Index]

[0054] S23: Calculate the low-resolution reconstruction weighted loss based on the predicted value of the low-resolution mask block; process the predicted value of the low-resolution mask block using a multilayer perceptron to obtain the predicted value of the high-resolution mask block; calculate the high-resolution reconstruction weighted loss based on the predicted value of the high-resolution mask block.

[0055] Traditional reconstruction pre-training processes all image patches equally. However, this invention argues that each patch has different importance, especially those containing key pathological regions, which possess richer visual features. To enable the model to learn the visual features of key pathological regions more effectively, patches containing these regions need to be given greater weight. First, the ScispaCy tool is used to extract disease-related medical terms from the text T of the image pairings. If these terms are unavailable, a vector is set... All elements are equal to 0, indicating that all patches are treated equally. If a disease term is detected, publicly available pre-trained weights are loaded using state-of-the-art methods to generate a weight vector A. The elements of A are between 0 and 1, and these values ​​are then linearly scaled to generate the final weight map W. This process can be represented mathematically as follows:

[0056] W = α·A + β

[0057] Here, α is a scaling factor and β is an offset. This scaling adjusts the weight values, enhancing their importance based on the attention score of each patch, thereby improving reconstruction pre-training performance by giving more attention to important patches.

[0058] The reconstruction weight W is calculated based on the weight map and the index of the masked patch block. * :

[0059] W * =W[Index]

[0060] The low-resolution reconstruction weighted loss is calculated based on the reconstruction weights, the predicted values ​​of the low-resolution mask blocks, and the original values ​​of the low-resolution mask blocks:

[0061]

[0062] Among them, L low N represents the low-resolution reconstruction weighted loss. m W represents the number of low-resolution mask blocks. i * I represents the weight of the i-th low-resolution mask block. Pred,i I represents the predicted value of the i-th low-resolution mask block. m,i This represents the reconstructed label of the i-th low-resolution mask block.

[0063] Extracting features from low-resolution images often leads to significant information loss, especially in small lesion areas, resulting in degraded performance on visual tasks. High-resolution images can help visual encoders extract more comprehensive and accurate feature representations, thereby improving performance on visual tasks. Therefore, it is possible to reconstruct not only low-resolution images but also high-resolution images.

[0064] To recover the high-resolution patch, a multilayer perceptron (MLP) is used, consisting of two fully connected layers and an activation function, to process the predicted values ​​of the low-resolution mask block to obtain the predicted values ​​of the high-resolution mask block. This process is described as follows:

[0065] I restored =MLP(I Pred )

[0066] The high-resolution reconstruction weighted loss is calculated based on the predicted values ​​of the high-resolution mask blocks. Specifically, the high-resolution image I... s Divided into N p High-resolution patches were obtained using the index. The formula for calculating the weighted loss of high-resolution reconstruction is:

[0067]

[0068] Among them, L high N′ represents the weighted loss for high-resolution reconstruction. m W represents the number of high-resolution mask blocks. i ′ I represents the weight of the i-th high-resolution mask block. restored,i I′ represents the predicted value of the i-th high-resolution mask block. m,i This represents the reconstructed label of the i-th high-resolution mask block.

[0069] The overall optimization objective of the image reconstruction process is defined as the sum of the reconstruction losses of the low-resolution and high-resolution images, L. MIM This comprehensive loss ensures that the model effectively reconstructs features at multiple resolutions. The optimization objective can be expressed as:

[0070] L MIM =L high +L low

[0071] S24: Embed the text paired with the images in the dataset to obtain text embedding features.

[0072] Use WordPiece to convert text T into a token sequence R = {r1, r2, ..., r}. M}, where M represents the length of the token sequence, and 70% of R is masked to obtain the masked portion. Unmasked portion The purpose of using a high masking ratio is to make the text reconstruction task more challenging, thereby encouraging the visual encoder to extract more information representations for text reconstruction.

[0073] Insert [MASK] at the appropriate positions in the mask to maintain the original size of R, and then compute the text embedding features F by projecting the text tokens and adding the encoded positional information. t .

[0074] S25: Combine visual features and text embedding features to obtain fused features; the text encoder and text tag classifier process the fused features to obtain classification probabilities.

[0075] Visual features F are fused using a multimodal fusion module (containing a multi-head cross-attention block and a multilayer perceptron). v and text features F t Obtain the fusion feature F f The fused features are processed using a text encoder and a text tag classifier to obtain the classification probability R. *, represented as:

[0076] R * =D t (E t (F t ))

[0077] S26: Calculate the text reconstruction loss based on the classification probability; sum the low-resolution reconstruction weighted loss, high-resolution reconstruction weighted loss, and text reconstruction loss to obtain the reconstruction loss.

[0078] The formula for calculating the text reconstruction loss is:

[0079]

[0080] Among them, L MLM Let r represent the text reconstruction loss, M represent the length of the text token sequence, and r represent the text reconstruction loss. i This represents the i-th mask label. This represents the classification probability of the i-th mask.

[0081] The reconstruction loss is obtained by summing the low-resolution reconstruction weighted loss, the high-resolution reconstruction weighted loss, and the text reconstruction loss:

[0082] L res =L MIM +L MLM

[0083] S27: Adjust the parameters of the visual encoder and text encoder according to the reconstruction loss to obtain the initially trained visual encoder and text encoder.

[0084] Throughout the reconstruction phase, the visual encoder and text encoder are optimized based on the loss function, utilizing the reconstruction loss L. res Capture local details of the image to obtain a pre-trained visual encoder and text encoder.

[0085] S3: Combine the text encoder with the visual encoder for a second training to obtain a trained visual encoder.

[0086] A visual encoder and a text encoder are used to process the image and text respectively, obtaining global visual features v and global text features t. Two nonlinear visual and text projection layers P are used respectively. v and P t Transform v and t to the same dimension d, where v = P v (v) and t = P t (t).

[0087] To effectively capture the mutual and internal similarities between paired medical images and reports, this invention incorporates a contrastive learning objective. This encourages the framework to map correct image-report pairs closer together in the latent space, while further separating mismatched pairs.

[0088] The contrastive learning loss is calculated based on global visual features and global text features, specifically:

[0089]

[0090] Where exp(·) represents the natural exponential function, σ is the sharpness of the temperature hyperparameter control distribution, T represents the transpose operation, and L ITC This represents the overall contrastive learning loss, where N represents the number of samples. This represents the cross-entropy loss from the i-th sample image to the text. L represents the cross-entropy loss from the text to the image for the i-th sample. ITC Minimize the distance between similar image and text pairs, while maximizing the distance between different image and text pairs, v i Let t represent the i-th global visual feature. j This represents the j-th global text feature.

[0091] By optimizing L ITC The model learns to maximize the mutual information between matched image-text pairs, capturing the mutual and internal similarities between pairs of medical images and reports.

[0092] For a given pair of X-ray images and reports, traditional contrastive learning methods treat other pairs in the same batch as negative samples, forcing each X-ray image to correspond to a specific text report. This approach inevitably leads to semantic overlap problems, where pairs sharing highly similar semantic information are treated as negations. Relying solely on global image-text alignment, cross-modal representations of the same disease may be incorrectly separated, preventing the model from capturing the corresponding image features of that disease. To mitigate the impact of these false negatives, MGCA clusters image features and text features separately. It then performs cluster alignment, grouping samples with very similar meanings together, ensuring they are close in the feature space. Existing clustering methods, while designed differently, all internally generate supervisory signals. Despite their success, this internally guided clustering approach has inherent limitations. For example, pneumonia and lung cancer both show similar shadows in images, so they may be incorrectly grouped together. Similarly, pneumonia and pneumoniax have similar text features but different image features, leading to clustering errors when clustering text. Internal clustering methods introduce noise during cluster alignment, making optimization very difficult.

[0093] In medical knowledge bases, the definitions of different medical entities vary. Utilizing expert knowledge as an external supervisory signal can clarify the visual and textual features of different diseases, thereby improving clustering performance.

[0094] Specifically:

[0095] First, the global visual features and global text features are concatenated along the feature dimension. This concatenated feature is then input into a linear layer for processing, yielding the aggregated feature Q. The purpose of this fusion operation is to distinguish samples with similar unimodal features. For example, although pneumonia and lung cancer have similar visual features, their text features differ. Through fusion, the features of a pneumonia sample will be distinguished from those of a lung cancer sample.

[0096] To ensure that samples with similar bimodal features remain distinguishable, this invention uses medical expert knowledge as an external supervision signal; preferably, a knowledge graph G is constructed based on a medical knowledge base (such as UMLS), and a graph attention network is used to process the knowledge graph to aggregate local information from the neighborhood of each node to obtain a medical entity knowledge representation K.

[0097] The feature fusion module (containing a multi-head cross-attention mechanism and two Transformer blocks) fuses the medical entity knowledge representation K (K serving as the key and value of the multi-head attention mechanism) with the aggregated feature Q (Q serving as the query of the multi-head attention mechanism) to obtain the final fused feature H. The fusion process can be represented by the following formula:

[0098] H=Transformer(MCA(LN(query,key,value)))

[0099] Where LN(·) represents layer normalization, MCA is multi-head cross attention, and the MCA mechanism can be expressed by the following formula:

[0100]

[0101] Where T represents the transpose of the matrix, and d is the dimension of the feature.

[0102] The fusion feature H exhibits higher significance and better clustering performance. An iterative Sinkhorn-Knopp clustering algorithm is used to cluster the final fusion feature H, resulting in multiple clusters and soft clustering labels L∈R for the final fusion feature. N×C Where C is the number of clusters and N is the number of samples.

[0103] Define the cluster center features for each cluster. Based on these cluster center features, calculate the probability that the global visual feature and global text feature belong to different clusters. Specifically: Predefine C trainable cluster center features R = (r1, r2, r3, r4...). Calculate the cosine similarity between the global image feature v and all cluster center features in R. After a SoftMax operation, obtain the probability p that the image feature belongs to each cluster. v Similarly, we calculate the cosine similarity between the global text feature t and the features of all cluster centers in R to obtain the probability p that the text feature belongs to each cluster. t .

[0104] The cross-entropy loss is calculated based on the probabilities of global visual features and global text features belonging to different clusters, as well as the soft clustering labels of the final fused features. This causes features belonging to the same cluster to be brought closer together, while features belonging to different clusters are pushed apart, thereby guiding cross-modal cluster alignment. The formula for calculating the cross-entropy loss is:

[0105]

[0106] Among them, L CA The cross-entropy loss represents the number of samples, and C represents the number of clusters. p represents the soft clustering label indicating that the fusion feature of the i-th sample belongs to the k-th cluster. v,i (k) represents the probability that the global image features of the i-th sample belong to the k-th cluster, p t,i (k) represents the probability that the global text features of the i-th sample belong to the k-th cluster.

[0107] The contrastive loss L is obtained by adding the contrastive learning loss and the cross-entropy loss. con :

[0108] L con =L ITC +L CA

[0109] Using contrast loss L con The visual encoder obtained from the reconstruction phase is further optimized. When the loss function converges or a predetermined number of iterations is reached, the optimal visual encoder parameters are saved, resulting in a well-trained visual encoder. This helps the visual encoder capture the intrinsic relationships between images and their paired reports, and learn high-level semantic information.

[0110] S4: Use a trained visual encoder to classify medical images.

[0111] The visual encoder trained using the above process can be used for various visual tasks, such as object detection, image classification, and image segmentation. If the trained visual encoder is used to classify medical images, more accurate medical image classification results can be obtained.

[0112] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A medical image classification method based on prior knowledge-enhanced masking and alignment modeling, characterized in that, include: S1: Obtain the image-text pair dataset and preprocess it to obtain the preprocessed image-text pair dataset; S2: Reconstruct and pre-train the visual encoder and text encoder based on the pre-processed image-text pair dataset to obtain the initially trained visual encoder and text encoder; The training process includes: S21: Downsample the images in the dataset to obtain high-resolution and low-resolution images; S22: Process the low-resolution image to obtain primary features; use a visual encoder to process the primary features to obtain visual features; input the visual features into a visual decoder for processing to obtain the predicted value of the low-resolution mask block. S23: Calculate the low-resolution reconstruction weighted loss based on the predicted value of the low-resolution mask block; process the predicted value of the low-resolution mask block using a multilayer perceptron to obtain the predicted value of the high-resolution mask block; calculate the high-resolution reconstruction weighted loss based on the predicted value of the high-resolution mask block. S24: Embed the text paired with the images in the dataset to obtain text embedding features; S25: Combine visual features and text embedding features to obtain fused features; use a text encoder and a text tag classifier to process the fused features to obtain classification probabilities; S26: Calculate the text reconstruction loss based on the classification probability; sum the low-resolution reconstruction weighted loss, high-resolution reconstruction weighted loss, and text reconstruction loss to obtain the reconstruction loss; S27: Adjust the parameters of the visual encoder and text encoder according to the reconstruction loss to obtain the initially trained visual encoder and text encoder; S3: Combine the text encoder with the visual encoder for comparative pre-training to obtain a well-trained visual encoder; the training process includes: A visual encoder and a text encoder are used to process the image and text respectively to obtain global visual features and global text features; the contrastive learning loss is calculated based on the global visual features and global text features. Global visual features and global text features are concatenated, and the concatenated features are input into a linear layer for processing to obtain aggregated features; A knowledge graph is constructed based on a medical knowledge base, and a graph attention network is used to process the knowledge graph to obtain a representation of medical entity knowledge. The medical entity knowledge representation and aggregation features are integrated to obtain the final fused features; the final fused features are then clustered to obtain multiple clusters and soft clustering labels for the final fused features. Define the cluster center features for each cluster, and calculate the probability that the global visual features and global text features belong to different clusters based on the cluster center features of each cluster. The cross-entropy loss is calculated based on the probability that global visual features and global text features belong to different clusters and the soft clustering labels of the final fused features. The parameters of the visual encoder are adjusted based on the contrastive learning loss and cross-entropy loss to obtain a well-trained visual encoder. S4: Use a trained visual encoder to classify medical images.

2. The medical image classification method based on prior knowledge-enhanced masking and alignment modeling according to claim 1, characterized in that, The process of obtaining primary features includes: dividing the low-resolution image into multiple image patches; randomly selecting 75% of the image patches for masking to obtain a set of unmasked patches and a set of masked patches; performing Flatten operation and linear projection on the unmasked patches to obtain embedded features; and adding the embedded features to their corresponding positional codes to obtain primary features.

3. The medical image classification method based on prior knowledge-enhanced masking and alignment modeling according to claim 1, characterized in that, The formula for calculating the weighted loss of low-resolution reconstruction is expressed as: Among them, L low N represents the low-resolution reconstruction weighted loss. m W represents the number of low-resolution mask blocks. i * I represents the weight of the i-th low-resolution mask block. Pred,i I represents the predicted value of the i-th low-resolution mask block. m,i This represents the reconstructed label of the i-th low-resolution mask block.

4. The medical image classification method based on prior knowledge-enhanced masking and alignment modeling according to claim 1, characterized in that, The formula for calculating the text reconstruction loss is: Among them, L MLM Let r represent the text reconstruction loss, M represent the length of the text token sequence, and r represent the text reconstruction loss. i This represents the i-th mask label. This represents the classification probability of the i-th mask.

5. The medical image classification method based on prior knowledge-enhanced masking and alignment modeling according to claim 1, characterized in that, The formula for calculating the contrastive learning loss is: Among them, L ITC This represents the contrastive learning loss, where N represents the number of samples. This represents the cross-entropy loss from the i-th sample image to the text. Let represent the cross-entropy loss from the text to the image for the i-th sample.

6. The medical image classification method based on prior knowledge-enhanced masking and alignment modeling according to claim 1, characterized in that, The formula for calculating cross-entropy loss is: Among them, L CA The cross-entropy loss represents the number of samples, and C represents the number of clusters. p represents the soft clustering label indicating that the fusion feature of the i-th sample belongs to the k-th cluster. v,i (k) represents the probability that the global image features of the i-th sample belong to the k-th cluster, p t,i (k) represents the probability that the global text features of the i-th sample belong to the k-th cluster.

Images