Medical image classification method and related device

Abstract

The application relates to a medical image classification method and related equipment. The method comprises the following steps: generating an image block based on a sample medical image; inputting the image block into a student network and a teacher network; calculating an image-level loss according to global features output by the student network and the teacher network; calculating an image block-level loss according to local features output by the student network and the teacher network; performing pixel-level feature mapping according to the local features output by the student network and the teacher network, and calculating a pixel-level loss based on a mapping result; cooperatively training the student network and the teacher network according to the losses at all levels; and obtaining a medical image basic model encoder based on the trained student network. The application can construct an encoder for feature extraction under the condition of no labeled data and improve the feature expression capability of the encoder.

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and in particular to a medical image classification method and related equipment. Background Art

[0002] Feature extraction from medical images is a core technology in computer-assisted diagnosis and intelligent pathology analysis. Its goal is to extract discriminative features from medical imaging data to support tasks such as disease classification, lesion detection, and prognosis prediction. However, existing medical image feature extraction methods still have some shortcomings. For example, feature extraction methods such as ResNet (Residual Network) based on the CNN (Convolutional Neural Network) or Dinov2 (a self-supervised computer vision model released by Meta) based on the Vision Transformer (ViT) architecture typically rely on single-level feature extraction (such as global average pooling or fixed-size patch partitioning), making it difficult to simultaneously capture cross-scale information in medical images, from macroscopic tissue distribution to microscopic cellular structure. Furthermore, existing methods often rely on supervised learning, requiring a large amount of annotated data to improve generalization. However, in medical scenarios where annotations are scarce, these models are susceptible to image quality fluctuations, resulting in reduced feature consistency. Summary of the Invention

[0003] In response to the above-mentioned deficiencies or shortcomings, the present application provides a medical image classification method and related equipment. The embodiment of the present application adopts a self-supervised learning method, which can complete the construction of a medical image basic model encoder without labeled data; through the joint optimization of multi-level features at the image level, image block level and pixel level, cross-scale feature capture from macroscopic tissue distribution to microscopic cell structure is achieved; combined with the comparative learning and masking mechanism of the teacher-student model, the model's sensitivity to local lesions and the generalization ability of cross-device data are enhanced, ultimately improving the encoder's representation accuracy of pathological features.

[0004] According to a first aspect, the present application provides a method for constructing a medical image base model encoder based on self-supervised learning. In some embodiments, the method includes:

[0005] generating an image block based on a sample medical image; the sample medical image is an original medical image that has undergone a series of processing; the series of processing includes preprocessing;

[0006] The image blocks are input into the student network and teacher network based on the Vision Transformer architecture respectively;

[0007] Calculate image-level loss based on the global features output by the student network and the teacher network;

[0008] Calculate the image block level loss based on the local features output by the student network and the teacher network;

[0009] Perform pixel-level feature mapping based on the local features output by the student network and the teacher network, and calculate the pixel-level loss based on the mapping results;

[0010] Jointly train the student network and the teacher network using image-level loss, image block-level loss, and pixel-level loss.

[0011] The medical image base model encoder is obtained based on the trained student network.

[0012] The method for constructing a medical image basic model encoder based on self-supervised learning provided in this application can complete the construction of the medical image basic model encoder without labeled data by adopting a self-supervised learning method. It also realizes cross-scale feature capture from macroscopic tissue distribution to microscopic cell structure through multi-level feature joint optimization at the image level, image block level and pixel level, and combines the comparative learning and masking mechanism of the teacher-student model to enhance the model's sensitivity to local lesions and the generalization ability of cross-device data, ultimately improving the encoder's representation accuracy of pathological features.

[0013] According to a second aspect, the present application provides a method for constructing a medical image classification model based on weakly supervised learning. In some embodiments, the method includes:

[0014] Segment the sample medical film into multiple image blocks, input the multiple image blocks into the feature extraction model, and obtain the feature vector of each image block;

[0015] The feature vectors of all image blocks are input into a multi-instance learning model based on the attention mechanism, which includes an attention module and an instance clustering module. The attention module is used to assign a multi-class attention score to each image block according to the feature vector of each image block. The instance clustering module is used to determine a plurality of high-scoring image blocks and a plurality of low-scoring image blocks corresponding to each class according to the multi-class attention score of each image block, and set a pseudo label for each high-scoring image block and low-scoring image block.

[0016] The classification loss is determined based on the multi-class attention scores of each image block, the clustering loss is calculated based on the feature maps and pseudo labels of each high-scoring image block and low-scoring image block, the model loss is calculated based on the classification loss and clustering loss, and the multi-instance learning model is reversely updated based on the model loss;

[0017] A medical image classification model is obtained based on the trained multi-instance learning model.

[0018] The present application provides a method for constructing a medical image classification model based on weakly supervised learning. This method employs weakly supervised learning and uses a large number of medical images combined with a small amount of individual-level annotation information to complete the construction of the medical image classification model. The feature extraction model can be a medical image basic model encoder trained using the method provided in the first aspect. In this way, a combination of self-supervised learning methods and weakly supervised learning methods can be used to construct a medical image classification model in a step-by-step manner. This can reduce the need for a large amount of finely labeled data, thereby reducing the cost of constructing the medical image set used for model training and improving the efficiency of constructing the medical image classification model.

[0019] According to a third aspect, the present application provides a method for constructing a medical image classification model. In some embodiments, the method includes:

[0020] Obtain a sample medical image set;

[0021] Generating a comprehensive attention score for each sample medical image in the sample medical image set to obtain an attention score set; the operation of generating the comprehensive attention score for the sample medical image includes: dividing the sample medical image into multiple image blocks, extracting features from each image block using multiple artificial intelligence base model encoders to obtain multiple groups of image block features corresponding to different artificial intelligence base model encoders, generating an attention score for the image block features in each group of image block features to obtain multiple groups of attention scores corresponding to different artificial intelligence base model encoders, and fusing the multiple groups of attention scores to obtain a comprehensive attention score;

[0022] The specified machine learning model is trained using the attention score set as training data to obtain a medical image classification model.

[0023] The medical image classification model construction method provided in this application adopts a staged processing approach to train the model. First, a comprehensive attention score is generated for the medical image, and then the comprehensive attention score of the medical image is used to train the model. Among them, using the attention score of the medical image to train the model enables the model to directly use the pre-calculated comprehensive attention score without the need to dynamically learn important areas during the training process, thereby accelerating convergence and reducing computational overhead, thereby improving the efficiency of model training. In the stage of generating comprehensive attention scores for medical images, by integrating multiple different artificial intelligence basic model encoders, cross-model feature complementarity is achieved. The differences in the ability of different architectures to capture image textures, edges, and spatial relationships form feature diversity, effectively avoiding the representation bias of a single model and helping to provide the classification accuracy of the final model. In addition, unlike the attention dynamically generated within the model, this method provides stable guidance through external pre-calculated scores, which may reduce randomness in training.

[0024] In some embodiments, the multiple artificial intelligence base model encoders include a medical image base model encoder constructed based on the self-supervised learning-based medical image base model encoder construction method provided in any embodiment of the first aspect above.

[0025] In some embodiments, a medical image classification model constructed using the medical image classification model construction method based on weakly supervised learning provided by any embodiment of the second aspect above generates attention scores for the image block features in each group of image block features.

[0026] According to a fourth aspect, the present application provides a medical image classification method based on deep learning. In some embodiments, the method includes:

[0027] Acquire a target medical image to be classified;

[0028] Processing the target medical image using a medical image classification model constructed by the medical image classification model construction method based on weakly supervised learning according to any embodiment of the second aspect, or the medical image classification model construction method according to any embodiment of the third aspect;

[0029] A classification result of the target medical image is determined based on the output data of the medical image classification model.

[0030] According to a fifth aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method of any embodiment provided in any one of the first to fourth aspects is implemented.

[0031] According to the sixth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method in any embodiment provided in any one of the first to fourth aspects is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a method for constructing a medical image basic model encoder based on self-supervised learning according to one or more embodiments of the present application;

[0033] Figure 2 A schematic diagram of the training process of a medical image base model encoder provided by the present application according to one or more embodiments;

[0034] Figure 3 A flowchart of a method for constructing a medical image classification model based on weakly supervised learning according to one or more embodiments of the present application is provided;

[0035] Figure 4A schematic diagram of a process for constructing a medical image classification model according to one or more embodiments of the present application;

[0036] Figure 5 A schematic diagram of a process for generating a comprehensive attention score for a sample medical image according to one or more embodiments of the present application;

[0037] Figure 6 This is a diagram of the internal structure of a computer device provided in accordance with one or more embodiments of the present application. DETAILED DESCRIPTION

[0038] To make the purpose, technical solutions, and advantages of this application more clear, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. It should be understood that the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0039] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0040] In the description of this application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances. In addition, in the description of this application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship.

[0041] According to the first aspect, the present application provides a method for constructing a medical image basic model encoder based on self-supervised learning. By adopting a self-supervised learning method, the method can complete the construction of the medical image basic model encoder without labeled data. It also realizes cross-scale feature capture from macroscopic tissue distribution to microscopic cell structure through multi-level feature joint optimization at the image level, image block level, and pixel level. In addition, it combines the comparative learning and masking mechanism of the teacher-student model to enhance the model's sensitivity to local lesions and the generalization ability of cross-device data, ultimately improving the encoder's characterization accuracy of pathological features. The method is described in detail below through some embodiments.

[0042] In some embodiments, the method includes Figure 1 The steps shown below are combined Figure 2 The training process of the medical image base model encoder is shown in Figure 1 The steps are introduced below.

[0043] S110: Generate an image block based on a sample medical image.

[0044] The sample medical image is an original medical image that has undergone a series of processing, and the image block is an image block divided from the sample medical image.

[0045] In some embodiments, the DINOv2 algorithm, based on the Vision Transformer (ViT) architecture, is used in conjunction with a specially designed medical image dataset to construct a medical image base model encoder. ViT segments the input image into a series of fixed-size patches and uses a multi-head self-attention mechanism to process the global dependencies between these patches, thereby achieving a deep understanding of the image content.

[0046] DINOv2 is a model for pre-training image encoders on large image datasets. Its model structure is based on ViT and obtains semantic visual features through self-supervised learning. It aims to learn robust visual feature representations from unlabeled image data. These features can be directly applied to a wide range of downstream visual tasks without fine-tuning or additional supervisory signals. Dinov2 introduces knowledge distillation, which is achieved by maintaining a teacher network and a student network. The student network is responsible for learning feature representations from the data, while the teacher network is used to guide the learning process of the student network. During the knowledge distillation process, the parameters of the teacher network are updated in each training iteration, gradually approaching the parameters of the student network while maintaining a certain stability. This process is achieved through exponential moving average.

[0047] The aforementioned medical image datasets include multiple raw medical images. These datasets can be single-center or multi-center data collected and organized from clinical medical institutions, or they can be open-source datasets obtained from domestic and international public databases. These datasets are used to train the encoder for the medical image base model. Because this method utilizes self-supervised learning, it does not require the collection of clinical information related to the raw medical images, nor does it require labeling of the raw medical images.

[0048] Raw medical images refer to collected, unprocessed medical images. Medical images are visualizations of the structure and function of internal organs and tissues in the human body, obtained clinically through various imaging technologies. They are used for disease screening, medical diagnosis, disease monitoring, treatment planning, and biomedical research. Medical images include, but are not limited to, digital pathology slides, computed tomography (CT), X-ray imaging, magnetic resonance imaging (MRI), ultrasound imaging (US), positron emission tomography (PET), single electron emission computed tomography (SPECT), thermal imaging, optical coherence tomography (OCT), dermoscopy, endoscopic images, angiography, and fundus images.

[0049] In order to ensure the performance and generalization ability of the basic model encoder, as well as its effectiveness and reliability in actual clinical applications, when constructing a medical image dataset, it is necessary to collect a sufficient number of clinically representative medical images, including but not limited to tumor tissue, adjacent cancerous tissue, benign tumors, non-tumor disease lesions and normal tissue, so that the dataset can cover the same or multiple types of medical images in various organs and parts of the human body.

[0050] Sample medical images are obtained by performing a series of processing on the original medical image. This series of processing includes preprocessing, image segmentation, and image block division. Preprocessing includes quality control processing. Quality control processing includes, but is not limited to, quality inspection, color correction, and other processing.

[0051] Quality checks include checking that the image is clear and not blurry or out of focus; confirming that the image is not overexposed or underexposed; verifying that the image has no obvious artifacts or damaged areas; ensuring that the image's colors are not distorted and that there is no physical or chemical damage that would hinder diagnosis. Once the original medical image is determined to be unsuitable for use, it needs to be removed from the dataset or marked as invalid.

[0052] In order to ensure color consistency between medical images from different sources, it is necessary to perform color correction on the original medical images. In some embodiments, the pre-processing operations on the original medical images include:

[0053] Taking the original medical image as the source image, determining the maximum illumination intensity estimation value of the source image and the target image respectively according to the first pixel point in the source image and the target image;

[0054] The RGB intensity matrices of the source and target images are decomposed into corresponding color appearance matrices and staining intensity matrices through the objective function; the objective function is:

[0055] ;

[0056] Where I represents the RGB intensity matrix of the image, represents the maximum illumination intensity, D represents the color appearance matrix, and H represents the staining intensity matrix; Represents the minimization of the reconstruction error term (used to minimize the difference between the original image and the reconstructed The error between represents the sparse constraint term, represents the smoothness constraint, represents the color consistency loss term;

[0057] Adjusting the stain density distribution range of the stain intensity matrix of the source image to match the distribution characteristics of the stain intensity matrix of the target image;

[0058] A color-corrected source image is generated according to the maximum illumination intensity estimate of the source image and the adjusted stain intensity matrix, and the color appearance matrix of the target image.

[0059] In order to ensure color consistency between medical images from different sources, this embodiment uses an improved Vahadane method to perform color correction on the original medical images.

[0060] The source image and the target image (which is an image used for reference) are respectively optimized by the above objective function to decompose the RGB intensity matrix of the source image into a color appearance matrix and a staining intensity matrix, and to decompose the RGB intensity matrix of the target image into a color appearance matrix and a staining intensity matrix.

[0061] In the above objective function is a sparse regularization hyperparameter to ensure the sparsity of the staining intensity matrix H; yes Regularization hyperparameter, used to ensure the smoothness of the staining intensity matrix H; is the color consistency loss parameter, which ensures the color consistency in the stained intensity matrix.

[0062] Loss of color consistency To reconstruct the image (i.e., by The difference between the calculated image and the original image (such as the source image and the target image) in the CIELAB color space is calculated using the following formula:

[0063] ;

[0064] is the i-th pixel of the original image I, is the reconstructed image The i-th pixel, Represents a function that converts RGB values ​​to CIELAB color space, where n is the total number of pixels. Represents the Euclidean norm of a vector.

[0065] The estimated maximum illumination intensity of the source and target images can be calculated by the following operations. First, a sufficient number of pixels are randomly sampled from the image (source or target image), ensuring that these pixels cover different areas of the entire image. For each sampled pixel, check whether its RGB three-channel values ​​are all greater than a fixed threshold (this threshold can be set according to the specific situation of the image, usually an empirical value or the best value determined through experiments). The appropriate percentile of the RGB values ​​of these pixels (which can be a preset percentile threshold) is taken as the maximum illumination intensity. estimated value.

[0066] Before color correction of the source image, a fixed-size sliding window can be used for sub-sampling, and sub-blocks with RGB channel values ​​greater than a fixed threshold (this threshold can also be set according to the specific situation of the image, usually an empirical value or the best value determined through experiments) and a proportion less than a preset ratio are selected for dyeing to reduce black or gray artifacts in the background.

[0067] In addition, the staining intensity matrix H can be scaled using appropriate percentile values ​​(which can be preset percentile values) to address the problem of insufficient or excessive staining intensity.

[0068] For the pre-processed original medical image, image segmentation is used to distinguish the foreground (such as tissue area) and background (such as non-tissue area) in the image. For example, the Otsu threshold method (also known as the Otsu algorithm) can be used to implement the image segmentation operation. The Otsu threshold method is an adaptive thresholding technique that automatically determines the optimal threshold by maximizing the inter-class variance between the foreground and background. Assume that the grayscale range of the image is [0, L−1], and the probability of the pixel appearing at each grayscale level i is ,in , Denotes the number of pixels with gray level i, and N is the total number of pixels in the image. For each possible threshold T, the image is divided into two parts: one part contains pixels with gray level less than or equal to T, and the other part contains pixels with gray level greater than T.

[0069] The core of the algorithm is to find a threshold T so that the inter-class variance between the two parts Maximize the between-class variance Defined as:

[0070] ;

[0071] and They represent the weights of pixels less than or equal to T and the weights of pixels greater than T, respectively, which can be expressed as:

[0072] ;

[0073] at the same time, and Represent the average grayscale value of these two parts respectively:

[0074] ;

[0075] By traversing all gray levels T, calculate each step , and select The largest T is taken as the optimal threshold. The threshold T obtained in this way is the value that can best distinguish the foreground and background.

[0076] In some embodiments, the operation of extracting image blocks from an original medical image includes: obtaining a preset benchmark magnification and its corresponding benchmark image block size; and determining the image block size of the original medical image based on the magnification of the original medical image, the benchmark magnification and its corresponding benchmark image block size.

[0077] After preprocessing and image segmentation of the original medical image, image patches are further extracted from the image. Traditional methods generally use a fixed image patch size to extract image patches. However, medical images are usually high-resolution, and the resolution of medical images from different sources may vary. Using traditional methods to extract image patches may make it difficult to conduct effective comparative analysis between images of different resolution levels. To this end, the present application also provides a new image patch extraction method that can dynamically adjust the image patch size based on the actual resolution of the image. In this way, when processing medical images of different magnifications, the actual physical area covered by each extracted image patch can be kept the same.

[0078] You can first determine a baseline magnification and its corresponding image block size When processing images with different magnifications, in order to keep the actual physical area covered by each image block the same, the image block size used when extracting image blocks from the image can be adaptively adjusted. For example, if the magnification of the currently processed image is X, the new image block size yes , the value of n can be calculated by the following formula, and then the new image block size can be obtained:

[0079] .

[0080] For example, the benchmark magnification is 20X, and the corresponding benchmark image block size is Pixels, when processing images with a magnification of 40X, you can use The image patches are extracted from this image to unify the patch size and maintain consistent data density.

[0081] The above-mentioned adaptive image block size adjustment strategy not only helps to maintain the feasibility of the computational task, but also ensures effective comparative analysis between images at different resolution levels.

[0082] Furthermore, considering that a large image block size may have a negative impact on computational efficiency, and a small image block size may not capture image details well, in order to balance the two, a minimum image block size can be set. and the maximum image block size , the final image patch size Should satisfy .

[0083] This embodiment uses a combination of image-level, patch-level, and pixel-level objectives to train the encoder for the medical image base model. This approach comprehensively learns global structural information and local detail features. Accordingly, during training, for each sample image, multiple layers of input images are constructed based on the sample image. The student network accepts all preprocessed image patches, including large-scale crops from a global perspective and small-scale local crops. The teacher network, on the other hand, only uses large-scale crops from a global perspective.

[0084] During the training process, Figure 2 As shown in Figure 1, we consider the objective functions at the image level, image patch level, and pixel level, and configure independent MLP (Multilayer Perceptron) projection heads for each of the three levels. In addition, we introduce KoLeo regularization to encourage uniform distribution of features within a batch. The formula is as follows:

[0085] ;

[0086] ;

[0087] in Represents the minimum distance to other vectors in the batch, and the distance is calculated using the L2 norm. To perform effective batch normalization, the Sinkhorn-Knopp iterative algorithm is used instead of the traditional softmax-centering step. Three Sinkhorn-Knopp iterations are performed on the student network output before calculating the loss, and softmax normalization is applied on this basis.

[0088] S120: Input the image block into the student network and teacher network based on the Vision Transformer architecture.

[0089] After inputting the image blocks into the student network and the teacher network respectively, the image-level loss, image-block-level loss, and pixel-level loss (these three types of losses are parallel) are calculated based on the data output by the student network and the teacher network (local features and global features). The calculation details of each type of loss are explained below.

[0090] S130: Calculate image-level loss based on the global features output by the student network and the teacher network.

[0091] For image-level targets, the features of ViT’s cls token are used to calculate the cross entropy loss by comparing the cls tokens (representing the global representation of the input image) of the student network and the teacher network obtained from different parts of the same image. , as the image-level loss. When calculating the image-level loss, the influence of the positive sample weight factor and the hard negative sample is further considered. The image-level loss can be calculated based on the following formula:

[0092] ;

[0093] represents the image-level loss, is the weight factor for each positive sample, is the predicted probability distribution of the student network for the image-level image output, and is the teacher network for the The predicted probability distribution of image-level image output, It is used to control hard negative samples Hyperparameters that influence the image-level loss, The feature vector representing the output of the student network and the hard negative sample feature vector The similarity between them (which can be measured by cosine similarity), It is the temperature parameter for hard negative samples.

[0094] Temperature parameters It can be automatically adjusted according to the performance indicators during training to better balance the similarity scores between positive and negative samples. At the beginning of training, set an initial value , monitor the performance indicators during training, including accuracy, F1 score, etc., pay special attention to the difference in similarity scores between positive and negative samples, and dynamically adjust the temperature parameter 𝜏 according to the changes in performance indicators. If it is found that the model's ability to distinguish difficult negative samples is insufficient, you can appropriately reduce 𝜏: , where 𝛼 is an adjustment factor to make the probability distribution smoother, thereby improving the model's ability to distinguish hard negative samples; if the model's ability to distinguish positive samples is insufficient, 𝜏 can be appropriately increased: , where 𝛽 is another adjustment factor to make the probability distribution sharper, thereby improving the model's ability to distinguish positive samples.

[0095] Using ViT's [CLS] token to capture the overall pathological pattern and enhancing semantic consistency through teacher-student model comparison helps improve the accuracy of full-film classification.

[0096] S140: Calculate the image block level loss based on the local features output by the student network and the teacher network.

[0097] For patch-level targets, only the image input to the student network is masked (which can be a random mask), while the image input to the teacher network is unmasked. The patch-level images are fed into the iBOT head of each network for processing, and then softmax and centering steps are applied to calculate the patch-level cross entropy loss as the patch-level loss.

[0098] Among them, the image block level loss can be calculated based on the following formula:

[0099] ;

[0100] represents the image block level loss, Is the student network for the The predicted probability distribution of image patches, is the predicted probability distribution of the teacher network for the th image patch.

[0101] In the above operations, the student network learns and restores the masked image blocks, and the teacher network provides stable signals, which helps to strengthen tissue microstructure modeling and improve the accuracy of local lesion detection.

[0102] S150: Perform pixel-level feature mapping based on the local features output by the student network and the teacher network, and calculate the pixel-level loss based on the mapping results.

[0103] For pixel-level targets, only the image input to the student network is masked, such as randomly selecting some pixels and replacing them with specific values ​​(such as 0 or the average value). The image input to the teacher network is not masked. The local features output by the student network and the teacher network are mapped to pixels to obtain pixel-level features. The pixel-level features are fed into the segmentation head. Through the softmax layer and the centering step, the predicted probability distribution of each pixel can be calculated. The cross-entropy loss is then calculated as the pixel-level loss based on the predicted probability distribution of each pixel. The pixel-level loss can be calculated based on the following formula:

[0104] ;

[0105] represents pixel-level loss, Represents the predicted probability corresponding to the true category at position (h, w), where H and W represent the height and width of the sample image, respectively. represents the true category label at position (h,w).

[0106] S160: Jointly train the student network and the teacher network using image-level loss, image block-level loss, and pixel-level loss.

[0107] By combining image-level loss, image block-level loss, and pixel-level loss, we get the total loss function:

[0108] ;

[0109] The total model loss is calculated based on image-level loss, image block-level loss, and pixel-level loss Afterwards, the student network and the teacher network are collaboratively updated, such as applying the gradient descent method to update the parameters of the student network, and updating the weights of the teacher network through the exponential moving average method.

[0110] S170: Obtain a medical image basic model encoder based on the trained student network.

[0111] The process of steps S110 to S160 continues iterating until a pre-defined termination condition is met, such as model convergence or a predetermined number of training rounds. During model optimization, training checkpoints are saved regularly. Ultimately, a fully trained student network is saved as a pre-trained model with deep understanding capabilities. In some embodiments, the feature extraction portion of the fully trained student network can also be extracted and used as a feature extractor.

[0112] In some embodiments, after obtaining the medical image basic model encoder based on the trained student network, the method further includes: inputting the medical image from which features are to be extracted into the medical image basic model encoder to obtain a feature vector of the medical image.

[0113] The Medical Image Base Model Encoder can be applied to a variety of downstream tasks. Accordingly, an evaluation method can be selected based on the task to which the Medical Image Base Model Encoder is applied. The following illustrates the evaluation method for the Medical Image Base Model Encoder using several exemplary tasks.

[0114] (1) For the whole medical image lesion detection and classification tasks, the medical image classification model based on weakly supervised learning is mainly used to evaluate the medical image basic model encoder. The evaluation indicators mainly include balanced accuracy, weighted F1 score and area under the receiver operating characteristic curve (AUROC).

[0115] Balanced accuracy is calculated by taking the unweighted average of the recall rates for each class, taking into account the imbalance of classes in the evaluation set. The formula is as follows:

[0116] ;

[0117] Weighted F1 score, which is obtained by calculating the F1 score (harmonic mean of precision and recall) of each class and weighting them according to their respective support set sizes. The formula is as follows:

[0118] ;

[0119] Then take the weighted average: ,in is the weight of the 𝑖th category.

[0120] AUROC is the area under the curve drawn between the true positive rate (TPR) and the false positive rate (FPR) when the classification threshold is varied. The formulas for TPR and FPR are as follows:

[0121] ;

[0122] ;

[0123] Where TP and TN are the correct classification results, TP is true positive, which means the number of positive samples predicted as positive by the model; TN is true negative, which means the number of negative samples predicted as negative by the model; FP is false positive, which means the number of negative samples wrongly predicted as positive by the model; FN is false negative, which means the number of positive samples wrongly predicted as negative by the model.

[0124] (2) For the region of interest (ROI) classification task, linear and K nearest neighbor probes are used to evaluate the medical image base model encoder, and the evaluation metrics include balanced accuracy and weighted F1 score.

[0125] (3) For the cell type segmentation task, Mask2Former is used to evaluate the medical image base model encoder. The evaluation indicators mainly include Dice coefficient and average precision.

[0126] The Dice coefficient measures the similarity between two sets, and its value range is [0, 1], where 1 indicates a perfect match and 0 indicates no overlap. The formula is:

[0127] ;

[0128] Where X is the predicted segmentation mask and Y is the true segmentation mask, representing the intersection between the predicted and true segmentation.

[0129] The average precision (AP) measures the comprehensive performance of the model at different confidence thresholds. The formula is:

[0130] ;

[0131] Where R(r) is the recall rate and P(r) is the precision rate, which changes with the change of the threshold r. Indicates the number of samples that actually belong to the target class.

[0132] (4) For image retrieval tasks, consider using the KNN (K-Nearest Neighbor) algorithm to evaluate the medical image base model encoder. The evaluation indicators include Acc@K, and MVAcc@5.

[0133] Acc@K means that the correctly labeled image is among the first K retrieved images. The accuracy (Acc) formula is as follows:

[0134] ;

[0135] MVAcc@5 represents the majority vote of the first five retrieved images. The formula for the mean average accuracy (MVAcc) is as follows: ,in, represents the importance weight of the i-th category label, represents the average precision of the i-th class label.

[0136] (5) For few-shot whole-image lesion detection, classification, and ROI classification, SimpleShot is mainly used to evaluate the encoder of the medical image base model. The evaluation metrics include balanced accuracy and weighted F1 score.

[0137] Current deep learning-based methods often require large amounts of finely labeled data for training to achieve high recognition accuracy. However, in the field of medical image processing, limited expert resources and a complex and time-consuming labeling process often make it difficult to obtain the large amounts of labeled data required for model training.

[0138] In response to the above problems, the present application provides a method for constructing a medical image classification model based on weakly supervised learning according to the second aspect. This method adopts weakly supervised learning and can complete the construction of the medical image classification model using massive medical images and a small amount of individual-level annotation information.

[0139] In some embodiments, the method includes Figure 3 The steps shown are steps S210-S240.

[0140] This method uses a medical image set and a small amount of annotated information to iteratively train a multi-instance learning model based on an attention mechanism in a weakly supervised manner. The training process is shown in steps S210-S230. When a preset end condition is met, training stops. A trained medical image classification model is obtained in step S240.

[0141] The medical image set used for model training can be collected according to the specific classification objectives. Taking the classification of tumor diseases as an example, it is necessary to collect a medical image dataset with sufficient sample size and clinical representativeness, including but not limited to tumor tissue, adjacent cancer tissue, benign tumors, non-tumor disease lesion tissue and normal tissue.

[0142] Medical images include, but are not limited to, digital pathology slide images, computed tomography (CT), X-ray imaging, magnetic resonance imaging (MRI), ultrasound imaging (US), positron emission tomography (PET), single electron emission computed tomography (SPECT), thermal imaging, optical coherence tomography (OCT), dermatoscope images, endoscopic images, angiography, fundus images, etc.

[0143] The dataset will cover the same or multiple types of medical images of various organs and parts of the human body.

[0144] Tumor diseases in medical images include but are not limited to lung cancer, breast cancer, gastric cancer, esophageal cancer, colorectal cancer, pancreatic cancer, liver cancer, gallbladder cancer, endometrial cancer, cervical cancer, ovarian cancer, thyroid cancer, glioma, renal cell carcinoma, laryngeal cancer, bladder cancer, melanoma, lymphoma, prostate cancer, adrenal cancer, bladder cancer, etc., as well as non-tumor diseases. The main subtypes include:

[0145] Lung cancer subtypes and control categories include but are not limited to: adenocarcinoma, squamous cell carcinoma, neuroendocrine tumors (typical carcinoid, atypical carcinoid, small cell carcinoma, large cell neuroendocrine carcinoma), adenosquamous carcinoma, sarcomatoid carcinoma, salivary gland carcinoma, large cell carcinoma, undifferentiated tumors with SMARCA4 deficiency in the chest; benign epithelial tumors, adenomas, non-neoplastic diseases, and adjacent normal control tissues;

[0146] Breast cancer subtypes and control categories include but are not limited to: invasive breast cancer, non-invasive breast tumors (except ductal carcinoma in situ), ductal carcinoma in situ, malignant lymphoma of the breast, mesenchymal sarcoma, neuroendocrine tumors, other types of breast cancer, adenosis and benign sclerosing lesions, adenoma, intraductal papilloma; non-tumor diseases such as breast hyperplasia, cysts, mammary duct ectasia (plasma cell mastitis), breast tuberculosis, and adjacent cancer control tissues;

[0147] Gastric cancer subtypes and control categories include but are not limited to: adenocarcinoma, undifferentiated carcinoma, mucinous adenocarcinoma, mucinous cell carcinoma, squamous cell carcinoma, adenosquamous carcinoma, gastric lymphoma, gastrointestinal stromal tumor, gastric neuroendocrine tumor, gastric adenoma, gastric leiomyoma, gastric lipoma, gastric schwannoma, gastric precancerous lesions, non-tumor controls (benign gastric ulcer, etc.), non-tumor diseases, and adjacent adjacent tissues;

[0148] Esophageal cancer subtypes and control categories include but are not limited to: squamous cell carcinoma, adenocarcinoma, adenosquamous carcinoma, small cell carcinoma, sarcomatoid carcinoma, neuroendocrine carcinoma, neuroendocrine tumor, lymphoepithelial carcinoma, mucoepidermoid carcinoma, non-neoplastic diseases, and adjacent normal control tissues;

[0149] Colorectal cancer subtypes and control categories include but are not limited to: adenocarcinoma (not special type), serrated adenocarcinoma, mucinous adenocarcinoma, mucinous cell carcinoma, adenomatous adenocarcinoma, adenosquamous carcinoma, micropapillary adenocarcinoma, poorly adherent carcinoma, signet ring cell carcinoma, medullary carcinoma, undifferentiated carcinoma (not special type), carcinoma with sarcomatoid elements, other colorectal cancers, non-neoplastic diseases, and adjacent tumor control tissues;

[0150] Pancreatic cancer subtypes and control categories include but are not limited to: ductal adenocarcinoma, neuroendocrine tumors, islet cell tumors, cystic tumors, solid and pseudopapillary epithelial tumors, adenosquamous carcinoma, undifferentiated carcinoma, chronic pancreatitis, acute pancreatitis, islet cell tumors, cystic tumors, solid pseudopapillary tumors, pancreatic intraepithelial neoplasia, non-neoplastic diseases, and adjacent normal control tissues;

[0151] Liver cancer subtypes and control categories include but are not limited to: hepatocellular carcinoma, cholangiocarcinoma, mixed hepatocellular carcinoma-cholangiocarcinoma, intrahepatic neuroendocrine tumors, sarcomatoid carcinoma, non-tumor diseases (chronic liver disease, etc.) and adjacent normal control tissues;

[0152] Gallbladder cancer subtypes and control categories include but are not limited to: adenocarcinoma, squamous cell carcinoma, small cell carcinoma, neuroendocrine tumor, sarcomatoid carcinoma, micropapillary carcinoma, clear cell carcinoma, mucinous adenocarcinoma, non-neoplastic diseases (cholecystitis, etc.) and adjacent normal tissues;

[0153] Endometrial cancer subtypes and control categories include but are not limited to: endometrioid carcinoma, serous carcinoma, clear cell carcinoma, mucinous adenocarcinoma, adenosquamous carcinoma, sarcomatoid carcinoma, neuroendocrine tumors, non-tumor diseases (benign endometrial lesions, etc.) and adjacent normal control tissues;

[0154] Cervical cancer subtypes and control categories include but are not limited to: adenocarcinoma, squamous cell carcinoma, adenosquamous carcinoma, small cell carcinoma, clear cell carcinoma, sarcomatoid carcinoma, neuroendocrine tumors, non-neoplastic diseases, and adjacent normal control tissues;

[0155] Ovarian cancer subtypes and control categories include but are not limited to: serous carcinoma, mucinous carcinoma, endometrioid carcinoma, clear cell carcinoma, neuroendocrine tumor, immature teratoma, endodermal sinus tumor, dysgerminoma, granulosa cell tumor, Sertoli cell-Leydig cell tumor, non-neoplastic diseases (ovarian cysts, etc.) and adjacent normal control tissues;

[0156] Thyroid cancer subtypes and control categories include but are not limited to: papillary carcinoma, follicular carcinoma, medullary carcinoma, undifferentiated carcinoma, follicular adenoma, follicular tumor with Hürthle cell features, small cell carcinoma, non-neoplastic diseases (thyroid nodules, etc.) and adjacent normal tissues;

[0157] Glioma subtypes and control categories include but are not limited to: astrocytic tumors, oligodendrogliomas, medulloblastomas, ependymomas, pinealomas, colloid cysts, non-tumor diseases, and adjacent normal control tissues;

[0158] Renal cell carcinoma subtypes and control categories include but are not limited to: clear cell carcinoma, papillary renal cell carcinoma, chromophobe cell carcinoma, collecting duct carcinoma, Bellini duct carcinoma, multicentric small cell carcinoma, lipid-rich renal cell carcinoma, non-neoplastic diseases, and adjacent normal control tissues;

[0159] Laryngeal cancer subtypes and control categories include but are not limited to: squamous cell carcinoma, adenocarcinoma, adenosquamous carcinoma, small cell carcinoma, sarcomatoid carcinoma, basaloid squamous cell carcinoma, non-tumor controls: laryngitis, laryngeal papillary carcinoma, non-tumor diseases and adjacent normal tissues;

[0160] Bladder cancer subtypes and control categories include but are not limited to: invasive urothelial carcinoma, invasive urothelial carcinoma, squamous cell carcinoma, adenocarcinoma, adenosquamous carcinoma, small cell carcinoma, sarcomatoid carcinoma, neuroendocrine tumors, non-neoplastic diseases (cystitis, etc.) and adjacent normal control tissues;

[0161] Melanoma subtypes and control categories include but are not limited to: superficial spreading melanoma, acral lentiginous melanoma, lentigo maligna, nodular melanoma, acral melanoma, mucosal melanoma, ocular melanoma, small cell melanoma, epithelioid cell melanoma, acanthocyte melanoma, non-neoplastic diseases (common nevus, etc.) and adjacent non-cancer control tissues;

[0162] Lymphoma subtypes and control categories include but are not limited to: Hodgkin lymphoma: nodular lymphocyte-predominant type, classical Hodgkin lymphoma; non-Hodgkin lymphoma: diffuse large B-cell lymphoma, follicular lymphoma, mantle cell lymphoma, marginal zone lymphoma, small lymphocytic lymphoma, Burkitt lymphoma; T-cell and natural killer cell non-Hodgkin lymphoma: peripheral T-cell lymphoma, cutaneous T-cell lymphoma, NK / T-cell lymphoma, anaplastic large cell lymphoma, adult T-cell leukemia / lymphoma; non-tumor diseases (lymphadenitis, etc.) and adjacent normal tissues;

[0163] Prostate cancer subtypes and control categories include, but are not limited to, adenocarcinoma, small cell carcinoma, large cell neuroendocrine carcinoma, clear cell carcinoma, basal cell carcinoma, transitional cell carcinoma, sarcomatoid carcinoma, adenosquamous carcinoma, prostate sarcoma, non-neoplastic disease, and adjacent adjacent tissue.

[0164] Adrenal cancer subtypes and control categories include but are not limited to: adrenal sebaceous adenoma, sex cord-stromal tumor, mesenchymal and stromal tumor, adenomatoid tumor, pheochromocytoma, paraganglioma, adrenal neuroblastoma, etc., non-tumor diseases and adjacent adjacent tissues;

[0165] Patient tissue specimens include surgical resection, biopsy and other sources.

[0166] There are many types of non-tumor diseases, covering almost all medical specialties, including but not limited to: cardiovascular and cerebrovascular diseases, respiratory diseases, digestive system diseases, nervous system diseases, immune system diseases, infectious diseases, musculoskeletal system diseases, genitourinary system diseases, metabolic diseases, skin diseases, ophthalmological diseases, and otolaryngological diseases.

[0167] In addition to collecting medical images, it is also necessary to collect individual-level clinical information related to the medical images in order to create label information for the medical images. This clinical information can come from the hospital medical record information management system, including but not limited to: disease diagnosis, examination results and reports, tumor type, tumor histological subtype, histological grade, clinical stage, drug effectiveness, prognosis, somatic mutation status, the proportion of tissue pathological change areas, as well as ID information such as case ID, pathology ID, and hospitalization number. It can also come from laboratory tests, such as biomarker status from immunohistochemistry tests and clinical medication indications from various molecular biology tests.

[0168] Collected medical images undergo a series of processes, including preprocessing, image segmentation, and image block division, before they can be used for model training. After these processes, the sample medical images used for model training are obtained. The various operations included in this series of processes (such as preprocessing, image segmentation, and image block division) can be found in the introduction to the encoder construction method for the basic medical image model based on self-supervised learning above and will not be detailed here.

[0169] The following is a detailed description of each step of the method.

[0170] S210: Segment the sample medical film into multiple image blocks, input the multiple image blocks into a feature extraction model, and obtain a feature vector for each image block.

[0171] After a series of processing on the massive amount of collected medical images, a set of sample medical images can be obtained. A pre-trained feature extraction model can then be used to pre-extract features from each sample medical image, obtaining the overall features corresponding to each sample medical image. The feature vector matrix corresponding to the sample medical image can then be used to train the classification model. The sample medical image can first be segmented into multiple overlapping or non-overlapping image blocks. Each image block is then fed into the pre-trained feature extraction model to obtain the feature vector for each image block.

[0172] In some embodiments, the pre-trained feature extraction model can be an encoder based on a traditional convolutional neural network. The following uses ResNet50 and Inception V3 as examples for illustration.

[0173] (1) ResNet50 is a deep residual network consisting of a series of residual blocks, each of which includes multiple convolutional layers. By introducing the residual module, it effectively solves the gradient vanishing and gradient exploding problems in the deep neural network training process. The architecture of ResNet50 has the advantages of depth, high accuracy, and high computational efficiency, so it has been widely used in computer vision tasks. The Resnet50 network contains 49 convolutional layers and one fully connected layer. There are two types of residual blocks in the ResNet network. One is the Identity Block, which has the same number of input and output channels. Multiple blocks can be connected in series and can be directly added without changing the dimension. The other is the Conv Block, which cannot be connected in series. Its purpose is to change the dimension of the feature vector. The number of input and output channels is different, so it cannot be connected in series.

[0174] This example uses a pre-trained ResNet50 model and then adjusts it appropriately, removing the last fully connected layer and retaining all convolutional layers to build an efficient feature extractor. Each medical image is first segmented into multiple smaller image blocks (the size of the image blocks is not specifically limited in this example). Each image block is then independently input into the ResNet50 model for processing. In this model, each image block passes through a series of carefully designed convolutional layers, residual blocks, and other nonlinear transformations to ultimately generate an information-rich, high-dimensional feature vector. If there are n image blocks, n high-dimensional feature vectors will be generated. These feature vectors will be integrated into an HDF5 file to represent the overall characteristics of the medical image.

[0175] (2) Inception V3 is a deep convolutional neural network architecture proposed by Google, which aims to improve the accuracy and efficiency of image classification through modular design and computational optimization. It introduces a decomposition convolution strategy, splitting large-size convolution kernel operations into multiple small-size convolution steps, reducing computational complexity and enhancing nonlinear expression capabilities; adopts auxiliary classifiers to provide additional supervision signals, accelerate training convergence and alleviate the gradient vanishing problem; applies label smoothing technology to prevent overfitting and ensure more robust model predictions; and fully uses batch normalization to speed up training and allow higher learning rates without causing numerical instability. In addition, Inception V3 explores the application of asymmetric convolution. Each Inception block contains multiple parallel operation paths, including but not limited to: 1×1 convolution for dimensionality reduction, 3×3 or 5×5 standard convolution, 3×3 maximum pooling followed by 1×1 convolution for dimensionality recovery, and asymmetric convolution. The results of all these paths are spliced ​​together to form the output feature map, and finally the feature map is converted into a fixed-length feature vector through a global average pooling layer.

[0176] Each medical image is first segmented into multiple smaller patches. Each patch is then independently fed into the Inception V3 model for processing. This model extracts image features through a series of carefully designed convolutional layers and modular Inception blocks. Each block integrates multiple convolution paths of varying sizes and asymmetric convolutions, capturing image information at multiple scales. The outputs of these paths are then concatenated across the channel dimension to achieve multi-scale feature fusion. After processing by the Inception blocks, the feature maps are converted into fixed-length feature vectors through a global average pooling layer. For the entire image, the feature vectors of all patches are further processed or aggregated into a single HDF5 file, representing the overall features of the medical image.

[0177] In some embodiments, the feature extraction model is specifically a medical image base model encoder constructed based on the medical image base model encoder construction method based on self-supervised learning provided in any of the above embodiments.

[0178] The method for constructing a medical image classification model provided in this embodiment uses a combination of self-supervised learning methods and weakly supervised learning methods to construct a medical image classification model in a step-by-step manner. This can reduce the need for a large amount of finely labeled data, thereby reducing the cost of constructing a medical image set for model training and improving the efficiency of constructing the medical image classification model. In the pre-training stage of the feature extraction model based on self-supervised learning, pre-training only needs to be performed based on a large number of medical images (the training process can be found in the introduction to the method for constructing a basic medical image model encoder based on self-supervised learning above, which will not be repeated here). Individual-level label information is not required. In the training stage of the classification model based on weakly supervised learning, a combination of massive medical images and some individual-level label information (i.e., individual-level labels) is required to train an efficient medical image classification model.

[0179] Specifically, by using the self-supervised learning method, a medical image basic model encoder can be constructed for feature extraction without relying on any labeled data. It can extract hundreds of millions of feature information from medical images and has the ability to recognize categories with few or even zero samples. By using the weakly supervised learning method, it is possible to achieve classification and recognition of medical images and accurate identification and positioning of lesion areas therein without relying on a large amount of finely labeled data. This not only improves the efficiency of building medical image visual classification AI models, but also improves the accuracy of medical image diagnosis and the generalization performance of rare categories. Moreover, the medical image classification model trained by this embodiment can adapt to a variety of different modalities and types of medical images such as CT, MRI, X-ray, and pathological slice images, and maintain good performance on different types of medical images. This means that the model construction method provided by this embodiment can be widely used in the construction scenarios of various clinical medical image artificial intelligence classification models, thereby improving its potential value in the application of artificial intelligence in clinical practice.

[0180] The above methods can be used to pre-extract the overall features of each sample medical film, which will serve as the model training dataset. The training dataset can be divided into 80% as a training set, 10% as a validation set, and 10% as a test set. This division method maximizes the use of data for model training while ensuring the model's generalization ability.

[0181] S220: The feature vectors of all image blocks are input into a multi-instance learning model based on the attention mechanism, which includes an attention module and an instance clustering module; the attention module is used to assign a multi-class attention score to each image block according to the feature vector of each image block; the instance clustering module is used to determine a plurality of high-score image blocks and a plurality of low-score image blocks corresponding to each class according to the multi-class attention score of each image block, and set a pseudo label for each high-score image block and low-score image block.

[0182] Under the standard multi-instance (MIL) formulation and weakly supervised learning paradigm, a multi-instance learning model based on the attention mechanism is constructed. The model uses an attention-based pooling function to aggregate the features of each image patch obtained in the previous step into a full-slice-level representation for classification. At a high level, during training and inference, the model examines and ranks all image patches in the pathology full-slice tissue region, assigning an attention score to each image patch, that is, the attention score of the k-th image patch of the i-th class, denoted as, the formula is as follows:

[0183] ;

[0184] in , For the first two layers of the attention network; , Represents N parallel attention branches of the attention network; It is the first fully connected layer in the model; Further each fixed patch level Represents a compressed 2m-dimensional vector.

[0185] At the same time, the output of the attention network is used to generate pseudo labels for each full piece in each training iteration to supervise the clustering. For each of the N classes, a fully connected layer is placed after the first layer W1; if the weight of the clustering layer corresponding to the i-th class is expressed as , then the cluster assignment score for the k-th patch prediction is Indicates that: Instead of gathering all patches in the entire film, the objective is optimized only on the subsets of the most and least attended areas. For the output scored patches, all are sorted from low to high according to the scores, and gate_attention is used to convert the M×high-dimensional (4m) feature vectors into M×1 attention scores and M×2m ​​feature maps (M is the number of all patches in a full film). The attention scores are sorted, and the feature maps corresponding to the largest top k scores (the first k scores after sorting from large to small) and the smallest top k scores (the first k scores after sorting from small to large) are taken out. The labels of the feature maps corresponding to the largest top k scores are set to 1, and the labels of the feature maps corresponding to the smallest top k scores are set to 0 as instance labels.

[0186] During training, the attention module and the instance clustering module work together to improve the performance and interpretability of the model. The attention module provides an intuitive understanding of key areas in the tissue, while the instance clustering module enhances the model's discriminative ability by refining the feature space.

[0187] S230: Determine the classification loss based on the multi-class attention score of each image block, calculate the clustering loss based on the feature maps and pseudo labels of each high-score image block and low-score image block, calculate the model loss based on the classification loss and the clustering loss, and reversely update the multi-instance learning model based on the model loss.

[0188] This interpretation of attention scores is reflected in the attention-based pooling full-slice aggregation rule, which is implemented by computing the full-slice-level representation as the average of all image patches, with the contribution of each patch normalized and weighted by its respective attention score; in this multi-class attention pool design, the attention network predicts N different sets of attention scores corresponding to N classes in a multi-class classification problem; the attention scores are multiplied by the corresponding feature maps and all feature maps are summed to obtain the feature vector , expressed as , input the feature vector into the fully connected layer, get the bag classification result, and calculate the loss of bag classification, that is, , including but not limited to cross entropy loss (CE Loss), focal loss (Focal Loss), etc.

[0189] Cross entropy loss (CE Loss) mainly measures the difference between two probability distributions. In binary or multi-classification problems, cross entropy loss is often used to measure the gap between the probability distribution predicted by the model and the probability distribution of the actual label. For binary classification problems, assuming there is a dataset ,in , and the output of the model is a real number between 0 and 1 , represents the probability that the sample belongs to the positive class. For multi-classification problems, represents the true category label of the sample, Representing the probability vector predicted by the model, the cross entropy loss can be defined as:

[0190] ;

[0191] It is a one-hot vector, that is, only one element is 1 (corresponding to the true category) and the rest are 0; is the model’s predicted probability for each class; in this case, the cross-entropy loss is determined solely by the log-probability of the true class.

[0192] Focal Loss is based on cross-entropy loss and is primarily used to address the problem of class imbalance. It reduces the weight of simple negative samples, allowing the model to focus more on samples that are difficult to classify. The formula for the focal loss function is as follows:

[0193] ;

[0194] represents the true category label of the sample, Representing the probability vector predicted by the model, it is a focusing parameter that controls the degree of weight reduction for easy examples. Focal loss can automatically reduce the contribution of easy examples during training while increasing attention to difficult examples, thereby helping to improve the model's performance on imbalanced datasets.

[0195] At the same time, we use the output of the attention network to generate pseudo labels for each full image in each training iteration to supervise the clustering. For each of the N classes, we place a fully connected layer after the first layer W1; if we denote the weight of the clustering layer corresponding to the i-th class as , then the cluster assignment score for the k-th patch prediction is Indicates that: ; Instead of gathering all the patches in the whole film, we optimize the target only on the subsets of the most and least focused areas; for the output scored patches, we sort them all by score from low to high, and use gate_attention to convert the M×high-dimensional (4m) feature vectors into M×1 attention scores and M×2m ​​feature maps (M is the number of all patches in a whole film); sort the attention scores, take out the feature maps corresponding to the largest and smallest topk scores, set the label of the largest topk to 1, and the label of the smallest topk to 0 as the instance label, input N binary classification fully connected layers to the 2×topk feature maps, and obtain N binary classification outputs (N is the predicted category), and calculate the loss of the N binary classification outputs and the instance label, that is, , including but not limited to Hinge Loss, Squared Hinge Loss.

[0196] Hinge Loss can maximize the classification interval and can be defined as .

[0197] Squared Hinge Loss is a smoothed Hinge Loss variant suitable for multi-classification problems. It aims to maximize the classification interval while making the loss function more friendly to the optimization process of the gradient descent method through smoothing. SmoothTop1SVM Loss can be defined as:

[0198] ;

[0199] Where y is the true label of the sample, is the prediction score of the model.

[0200] S240: Obtain a medical image classification model based on the trained multi-instance learning model.

[0201] After the multi-instance learning model is fully trained through the above steps, it can be used as a medical image classification model and used to process medical image classification tasks.

[0202] In some embodiments, after obtaining a medical image classification model based on the trained multi-instance learning model, the method further includes: dividing the medical image to be classified into multiple image blocks; inputting the multiple image blocks into the medical image classification model to obtain output data of the medical image classification model; the output data includes the classification results of the medical image and the attention score of each image block; normalizing the attention score of each image block to obtain a normalized score, generating a heat map based on the normalized score, and superimposing the heat map on the medical image.

[0203] Medical image classification models not only perform classification but also calculate an attention score for each image patch. These scores reflect the model's belief that the region is important for the final classification. The model output includes an attention score for each image patch. Each attention score is normalized to fall within a specific range, such as between 0 and 1, to obtain the normalized score (i.e., the normalized attention score).

[0204] Finally, a heatmap is generated based on the normalized scores and visualized. A diverging color spectrum can be used to convert the normalized scores into corresponding RGB color values. In this spectrum, warm tones (such as red) represent areas of high interest, while cool tones (such as blue) represent areas of low interest. This results in a heatmap. To ensure that the heatmap highlights key areas without compromising the details of the original image, an appropriate transparency level is set (typically, a transparency of approximately 0.5 is appropriate, but this can be adjusted based on actual needs). This transparency level is then used to encode the heatmap. Finally, the color-encoded heatmap is overlaid on the original medical image to form the final visualized heatmap. This overlay is typically semi-transparent, allowing users to simultaneously see the original image information and the regions of interest (ROIs) highlighted in the heatmap.

[0205] In some embodiments, pathologists' medical image annotation and heat map technology are used to construct lesion region annotation data for model training samples to improve the model's accuracy in predicting the proportion of lesion regions (e.g., tumor lesions). The proportion of lesion regions refers to the proportion of the lesion region in the entire slice.

[0206] Training samples are medical images containing tumor lesions. Lesion annotation data for some training samples is generated using two methods. The first method involves expert pathologists professionally annotating lesions (e.g., tumor lesions) in medical images. These annotations include the lesion's location within the image and its proportion within the overall sample. The second method uses heatmap technology to automatically calculate the specific area of ​​the lesion within the image. First, the medical image is processed using image segmentation technology to clearly distinguish between lesion and non-lesion areas. Then, the proportion of the lesion area to the total image area is calculated to determine the exact area of ​​the tumor. By combining these two methods, corresponding lesion annotation data is assigned to each medical image. Some images are manually annotated by pathologists, resulting in more detailed and accurate annotation data. Other images are automatically annotated using heatmap technology. While these annotations are less accurate, they are more efficient. Combining these two methods effectively balances annotation efficiency and data accuracy.

[0207] Each training sample in the final training data consists of the overall features of a medical image (including the feature vectors corresponding to the n image blocks divided into the medical image) and a label indicating the proportion of the tumor lesion area in the medical image. Using this training data for model training can improve the model's accuracy in predicting the proportion of lesion areas (such as tumor lesions).

[0208] Regarding the specific limitations of the method for constructing a medical image classification model based on weakly supervised learning provided in the second aspect, please refer to the limitations of the method for constructing a medical image basic model encoder based on self-supervised learning provided in the first aspect above, which will not be repeated here.

[0209] According to the third aspect, the present application provides a method for constructing a medical image classification model. Figure 4 As shown, the method includes:

[0210] S310: Obtain a sample medical image set.

[0211] The sample medical image set includes multiple sample medical images. The sample medical images are medical images that have undergone a series of processes, including preprocessing, image segmentation, and image block division (the specific operations can be found above and will not be repeated here).

[0212] S320: Generate a comprehensive attention score for each sample medical image in the sample medical image set to obtain an attention score set.

[0213] Operations that generate comprehensive attention scores for sample medical images, such as Figure 5 Shown, including:

[0214] S321: Divide the sample medical image into a plurality of non-overlapping image blocks;

[0215] S322: Extract features from each image block using multiple artificial intelligence base model encoders to obtain multiple sets of image block features corresponding to different artificial intelligence base model encoders;

[0216] S323: using the medical image classification model to generate an attention score for the image block features in each group of image block features, to obtain multiple groups of attention scores corresponding to different artificial intelligence base model encoders;

[0217] S324: Fuse multiple sets of attention scores to obtain a comprehensive attention score.

[0218] Each AI base model encoder is an encoder in a pretrained AI base model used to extract features from medical images. These AI base models may include, but are not limited to, UNI (a universal self-supervised base model for computational pathology), Prov-gigapath (a whole-slide pathology base model), CHIEF (Clinical Histopathology Imaging Evaluation Foundation, a universal machine learning framework for weakly supervised histopathology image analysis), and Virchow2 (a pathology base model). They may also include a medical image base model encoder constructed using any of the embodiments of the first aspect described above. The pretraining process for AI base models other than the medical image base model encoder is not within the scope of improvement of this embodiment and is therefore not described in detail. Those skilled in the art should be aware of the specific pretraining operations.

[0219] A sample medical image is first segmented into multiple image blocks, such as n non-overlapping, equal-sized blocks. Each block is then fed into a separate AI base model encoder. Each AI base model encoder focuses on capturing specific subtle structures and patterns in the image, generating a set of highly representative feature vectors for each block. Next, the feature vectors generated by each AI base model encoder are fed into a trained medical image visual classification model (which can be trained using the method provided in any embodiment of the second aspect). The model's internal attention mechanism dynamically assigns weights based on the importance of each block, thereby calculating an "attention score." This process generates an independent attention score for each feature vector extracted from each block output by the AI ​​base model encoder. This score reflects the importance of the associated block from the encoder's perspective. Assuming there are five AI base model encoders, the above process can generate five sets of attention scores for each n block of the medical image. These five sets of attention scores correspond one-to-one to the five AI base model encoders, and each set of attention scores includes n independent attention scores. These five sets of attention scores are then fused to produce a composite attention score. Methods for fusing attention scores include but are not limited to: weighted averaging, maximum selection, median selection, sorting aggregation, Adaboost algorithm, hard voting, and Bayesian fusion.

[0220] Weighted average method: For each image block i, calculate the weighted average of the attention scores of m AI base model encoders ,in, is the weight of the jth AI base model encoder, satisfying , the weights can be determined based on the cross-validation results or classification results.

[0221] Maximum selection method: For each image block i, the one with the highest attention score from all AI base model encoders is selected as the final attention score.

[0222] Median selection method: For each image block i, the median of the attention scores from all AI base model encoders is calculated as the final attention score.

[0223] Sorting aggregation: For each AI base model encoder j, sort the attention scores of the n image blocks from high to low, and give the corresponding Borda score according to the sorting position of each image block under each AI base model encoder. The sorting position of image block i under AI base model encoder j is , then its Borda score can be defined as For each image patch i, calculate the sum of its Borda scores under all AI base model encoders, sort all image patches according to the total Borda score, and map it back to the final attention score in the range of 0-100.

[0224] Adaboost algorithm: Use the Adaboost algorithm to combine the results of m artificial intelligence base model encoders: ,in is the weight of the jth AI base model encoder, which can be dynamically adjusted based on its performance on the training data. Adaboost builds a strong classifier by iteratively training weak classifiers and assigning weights to each classifier.

[0225] Hard voting method: For each image block, its attention score is binarized, that is, an appropriate threshold is set, and the number of 5 encoders that judge it as a "high attention area" is counted. The key image block is decided by majority voting.

[0226] Bayesian fusion: A Bayesian framework is used to fuse the outputs of encoders from different AI-based models. The prior probability is updated using the Bayesian theorem to obtain the posterior probability as the final attention score, i.e. , where D represents the attention score of each AI base model encoder, is the posterior probability, is the likelihood function, is the prior probability.

[0227] Through the above operations, a corresponding comprehensive attention score can be generated for each sample medical image, thereby obtaining an attention score set.

[0228] S330: Train the specified machine learning model using the attention score set as training data to obtain a medical image classification model.

[0229] This step will use the attention score set as training data to train the specified machine learning model.

[0230] For example, the attention score set can be divided into a training set, a test set, and a validation set, and the ratio of these data sets can be 7:2:1. Then, support vector machines (SVM), progressive gradient regression trees, elastic net regression, K-nearest neighbors, random forests, and deep neural networks are used as candidate models, and each candidate model is trained using the attention score set so that these models can achieve classification predictions for medical images. During the training process, cross-validation technology is used to evaluate the generalization ability of the model and adjust the hyperparameters of each candidate model to optimize model performance. Specific optimization methods include but are not limited to grid search and random search. After training is completed, the prediction results of each candidate model are evaluated, and the candidate model whose prediction results meet the preset conditions (such as the highest accuracy) is selected as the medical image classification model.

[0231] This embodiment adopts a phased processing approach to train the model, first generating a comprehensive attention score for the medical image, and then using the comprehensive attention score of the medical image to train the model. Among them, using the attention score of the medical image to train the model enables the model to directly use the pre-calculated comprehensive attention score without the need to dynamically learn important areas during the training process, thereby accelerating convergence and reducing computational overhead, thereby improving model training efficiency; and in the stage of generating comprehensive attention scores for medical images, by integrating multiple different artificial intelligence basic model encoders, cross-model feature complementarity is achieved, and the differences in the ability of different architectures to capture image texture, edges, and spatial relationships form feature diversity, effectively avoiding the representation bias of a single model, and helping to provide the classification accuracy of the final model; in addition, unlike the attention dynamically generated within the model, this method provides stable guidance through external pre-calculated scores, which may reduce randomness in training.

[0232] For the specific limitations of the medical image classification model construction method provided in the third aspect, please refer to the above limitations of the medical image basic model encoder construction method based on self-supervised learning provided in the first aspect and the medical image classification model construction method based on weakly supervised learning provided in the second aspect, which will not be repeated here.

[0233] According to a fourth aspect, the application provides a medical image classification method based on deep learning. In some embodiments, the method includes:

[0234] (1) Obtain the target medical image to be classified;

[0235] (2) processing the target medical image using a medical image classification model constructed based on the method provided in any embodiment of the second aspect or the third aspect;

[0236] (3) Determine the classification result of the target medical image based on the output data of the medical image classification model.

[0237] For the specific limitations of the medical image classification method based on deep learning provided in the fourth aspect, please refer to the limitations of the medical image classification model construction method based on weakly supervised learning provided in the second aspect and the medical image classification model construction method provided in the third aspect, which will not be repeated here.

[0238] The present application also provides a computer device. In some embodiments, the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method of any embodiment provided in any one of the first to fourth aspects can be implemented.

[0239] It should be noted that, in the embodiments of the present application, when it comes to data related to user information or user data, it is necessary to obtain and process it after the user has authorized it. When the embodiments of the present application are applied to specific products or technologies, it is necessary to obtain user permission or consent, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions. In addition, with respect to the various steps included in the method provided in any one of the above embodiments, unless there is a clear explanation in this article, the execution of these steps does not have strict order restrictions, and these steps can be performed in other orders. Moreover, at least a portion of these steps can include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0240] In some embodiments, the internal structure diagram of the computer device can be as follows: Figure 6As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as training data (such as medical image sets, etc.) for model construction, various parameters of the initially constructed model, and the like. For specific stored data, please refer to the definitions in the above-mentioned method embodiments. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements the method in any embodiment provided in any aspect from the first aspect to the fourth aspect.

[0241] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0242] The present application also provides a computer-readable storage medium. In some embodiments, a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the method of any embodiment provided in any one of the first to fourth aspects is implemented.

[0243] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0244] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink), DRAM (SLDRAM), RAMbus, direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0245] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0246] The above embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for constructing a medical image basic model encoder based on self-supervised learning, characterized in that: The method comprises: generating an image block based on a sample medical image; the sample medical image is an original medical image that has undergone a series of processing; the series of processing includes preprocessing; Input the image patch into the student network and teacher network based on the Vision Transformer architecture; Calculating image-level loss based on global features output by the student network and the teacher network; Calculating image block-level loss based on local features output by the student network and the teacher network; Perform pixel-level feature mapping based on the local features output by the student network and the teacher network, and calculate pixel-level loss based on the mapping results; Collaboratively training the student network and the teacher network by combining the image-level loss, the image block-level loss, and the pixel-level loss; Obtaining a medical image basic model encoder based on the trained student network; The collaborative training of the student network and the teacher network by combining the image-level loss, the image block-level loss, and the pixel-level loss includes: Calculating the total model loss according to the image-level loss, the image block-level loss, and the pixel-level loss, applying the gradient descent method to update the parameters of the student network, and updating the weights of the teacher network by the exponential moving average method; The operation of preprocessing the original medical image includes: Taking the original medical image as a source image, determining maximum illumination intensity estimation values ​​of the source image and the target image respectively according to first pixel points in the source image and the target image; The RGB intensity matrices of the source image and the target image are decomposed into corresponding color appearance matrices and staining intensity matrices by an objective function; the objective function is: ； Among them, I represents the RGB intensity matrix of the image, represents the maximum illumination intensity, D represents the color appearance matrix, and H represents the staining intensity matrix; represents the minimization of the reconstruction error term, represents the sparse constraint term, represents the smoothness constraint, represents the color consistency loss term; is the sparsity regularization hyperparameter, yes Regularization hyperparameters, is the color consistency loss parameter; Adjusting the color density distribution range of the color intensity matrix of the source image to match the distribution characteristics of the color intensity matrix of the target image; A color-corrected source image is generated according to the maximum illumination intensity estimate value of the source image and the adjusted stain intensity matrix, and the color appearance matrix of the target image.

2. The method according to claim 1, wherein Calculating image-level loss based on the global features output by the student network and the teacher network, including: The image-level loss is calculated based on the following formula: ； represents the image-level loss, is the weight factor for each positive sample, Is the student network for the The predicted probability distribution of image-level global feature outputs, Is the teacher network for the first The predicted probability distribution of image-level image output, It is used to control hard negative samples Hyperparameters that influence the image-level loss, The feature vector representing the output of the student network and the hard negative sample feature vector The similarity between It is the temperature parameter for hard negative samples.

3. The method according to claim 1, wherein Calculating image block-level loss based on local features output by the student network and the teacher network, including: The image block level loss is calculated based on the following formula: ； represents the image block level loss, Is the student network for the The predicted probability distribution of image patches, Is the teacher network for the first The predicted probability distribution of each image patch.

4. The method according to claim 1, wherein Performing pixel-level feature mapping according to the local features output by the student network and the teacher network, and calculating pixel-level loss based on the mapping results, including: The pixel-level loss is calculated based on the following formula: ； represents pixel-level loss, Represents the predicted probability corresponding to the true category at position (h, w), where H and W represent the height and width of the sample image, respectively. represents the true category label at position (h,w).

5. The method according to claim 1, wherein The series of processing further includes image block extraction; the operation of extracting image blocks from the original medical image includes: Obtaining a preset reference magnification and its corresponding reference image block size; The image block size of the original medical image is determined according to the magnification of the original medical image, the reference magnification and the corresponding reference image block size.

6. A method for constructing a medical image classification model based on weakly supervised learning, characterized in that: The method comprises: Segmenting the sample medical film into multiple image blocks, inputting the multiple image blocks into a feature extraction model to obtain a feature vector for each image block; Inputting the feature vectors of all image blocks into a multi-instance learning model based on an attention mechanism, the multi-instance learning model includes an attention module and an instance clustering module; the attention module is used to assign a multi-class attention score to each image block according to the feature vector of each image block; the instance clustering module is used to determine a plurality of high-scoring image blocks and a plurality of low-scoring image blocks corresponding to each class according to the multi-class attention score of each image block, and set a pseudo label for each high-scoring image block and low-scoring image block; Determine a classification loss based on the multi-class attention score of each image block, calculate a clustering loss based on the feature maps and pseudo labels of each high-scoring image block and the low-scoring image block, calculate a model loss based on the classification loss and the clustering loss, and reversely update the multi-instance learning model based on the model loss; A medical image classification model is obtained based on the trained multi-instance learning model; and the feature extraction model is constructed based on the method described in any one of claims 1 to 5.

7. A method for constructing a medical image classification model, characterized in that: The method comprises: Obtain a sample medical image set; Generating a comprehensive attention score for each sample medical image in the sample medical image set to obtain an attention score set; the operation of generating the comprehensive attention score for the sample medical image includes: dividing the sample medical image into multiple image blocks, extracting features from each image block using multiple artificial intelligence base model encoders to obtain multiple groups of image block features corresponding to different artificial intelligence base model encoders, generating an attention score for the image block features in each group of image block features to obtain multiple groups of attention scores corresponding to different artificial intelligence base model encoders, and fusing the multiple groups of attention scores to obtain a comprehensive attention score; The specified machine learning model is trained using the attention score set as training data to obtain a medical image classification model; the multiple artificial intelligence basic model encoders include a medical image basic model encoder constructed based on the method described in any one of claims 1-5.

8. The method according to claim 7, wherein Use a medical image classification model constructed based on the method of claim 6 to generate attention scores for the image block features in each group of image block features.

9. A medical image classification method based on deep learning, characterized in that: Acquire a target medical image to be classified; Processing the target medical image using a medical image classification model constructed based on the method according to any one of claims 6 to 8; A classification result of the target medical image is determined based on the output data of the medical image classification model.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images