A medical image classification method, system and storage medium based on contrastive learning

Abstract

The present invention discloses a medical image classification method, system and storage medium based on contrast learning, which relates to the technical field of long-tail medical image classification. The long-tail medical image dataset is divided into a training set and a test set. According to a preset scheme, weak data augmentation and strong data augmentation are respectively performed on the training set images in a batch processing manner. The obtained weakly data-augmented images and strongly data-augmented images are used to perform contrast learning tasks through a deep convolutional neural network, learn network parameters, and obtain a deep convolutional neural network with optimized parameters; and classify the long-tail medical images in the test set. The present invention generates learnable class prototypes for data augmentation through a prototype-enhanced contrast learning strategy, and obtains a balanced implicit augmented contrast learning loss, having advantages such as high accuracy, high efficiency, low cost and wide applicability.

Description

Technical Field

[0001] The present invention relates to the technical field of long-tailed medical image classification, and more specifically, to a medical image classification method, system and storage medium based on contrast learning. Background Art

[0002] At present, chromosomal abnormalities refer to abnormalities in the structure or number of chromosomes. Common ones include chromosomal lesions such as monosomy and trisomy, which may lead to genetic diseases or developmental disorders. Traditional chromosomal detection methods rely on manual microscope analysis or partially automated image processing techniques. However, these methods are time-consuming, limited in accuracy, rely on the experience of professionals, and are difficult to screen on a large scale. Therefore, how to improve the automation level of chromosomal image classification, reduce labor costs, and improve screening efficiency is a technical problem that urgently needs to be solved at present. At this stage, most of the research on genetic abnormality detection is based on gene abnormality detection, including base sequences, gene loci, etc., and there is little related detection based on chromosomal images. Secondly, regarding the research on chromosomal abnormalities, deep learning technology is only simply used for auxiliary detection such as denoising or reducing the false positive rate of abnormal sites. From the perspective of research methods, in this research field, the deep learning models currently used are mostly simple basic models such as CNN and RNN for feature extraction, and the effects are relatively average.

[0003] The patent application document with the publication number CN109300111A discloses a chromosomal classification method based on deep learning. By calculating chromosomal morphological features, a ResNet residual neural network is trained for chromosomal classification. However, due to the relatively simple modeling of the selected morphological features, the classification accuracy of the model is not high, and effective classification of abnormal chromosomes cannot be achieved.

[0004] The patent application document with the publication number CN109285174A discloses chromosomal segmentation and karyotype analysis based on crowdsourcing and deep learning. By preprocessing the segments identified by crowdsourcing to improve the classification achieved using a deep convolutional network (DCNN), but since the chromosomes are classified after straightening the curved chromosomes, the original chromosomal morphological structure is damaged, resulting in loss of feature information.

[0005] Therefore, how to improve the accuracy of long-tailed medical image classification is an urgent problem for those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a medical image classification method, system and storage medium based on contrast learning. Through a prototype-enhanced contrast learning strategy, learnable class prototypes are generated for data augmentation, and a balanced implicit augmented contrast learning loss is obtained, which has the advantages of high precision, high efficiency, low cost and wide applicability.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A medical image classification method based on contrastive learning, comprising:

[0009] Step 1: Divide the collected long-tail medical image dataset into a training set and a test set. According to a preset scheme, perform weak data augmentation and strong data augmentation on the training set images in a batch processing manner to obtain weakly data-augmented images and strongly data-augmented images;

[0010] Step 2: Perform a contrastive learning task on the weakly data-augmented images and strongly data-augmented images obtained in Step 1 through an image classification model based on a deep convolutional neural network, learn the network parameters, and obtain an image classification model based on a deep convolutional neural network with optimized parameters;

[0011] Step 3: Use the image classification model based on a deep convolutional neural network with optimized parameters obtained in Step 2 to classify the long-tail medical images in the test set of Step 1.

[0012] Optionally, the weak data augmentation includes: cropping and resizing, horizontal flipping, color jittering; the strong data augmentation includes: cropping and resizing, horizontal flipping, vertical flipping, color jittering, and grayscaling.

[0013] Optionally, the image classification model based on a deep convolutional neural network consists of a backbone network, a mapping network, and a prototype enhancement model. The backbone network includes a feature extraction layer and a classification layer. Among them, the feature extraction layer uses a residual neural network, and the classification layer consists of a fully connected layer and a Softmax layer; the mapping network is a two-layer perceptron.

[0014] Optionally, the training process of the image classification model based on a deep convolutional neural network is as follows: First, input the weakly data-augmented images and strongly data-augmented images obtained in Step 1 into the feature extraction layer of the backbone network for feature extraction and then input them into the mapping network to obtain image embeddings. At the same time, input the features extracted by the feature extraction layer into the classification layer to obtain a basic cross-entropy loss; Second, input the image embeddings obtained after passing through the mapping network into the prototype enhancement model, use the prototype enhancement contrastive learning strategy to generate learnable class prototypes for data augmentation, and finally, obtain a balanced implicit enhancement contrastive learning loss.

[0015] Optionally, Step 2 includes the following steps:

[0016] Step 2.1: Input the weakly data-augmented images and strongly data-augmented images obtained in Step 1 into the feature extraction layer of the image classification model based on a deep convolutional neural network respectively to obtain features with a dimension of (2 B， 2048) dimensions, including the iThe weakly data-augmented image features of the i -th sample and the strongly data-augmented image features of the B -th sample, where

[0017] is the number of batch training samples; B , C C is the number of classes in the training set in Step 1, and the cross-entropy loss is calculated through the cross-entropy loss function;

[0018] Step 2.2: Input the weakly data-augmented image features obtained in Step 2.1 into the classification layer of the image classification model based on the deep convolutional neural network to obtain the prediction (

[0019] ), and input the weakly data-augmented image features and strongly data-augmented image features obtained in Step 2.1 into the mapping network to obtain the (2B, 128)-dimensional image embeddings for contrastive learning training, specifically divided into weakly augmented image embeddings and strongly augmented image embeddings, and input the obtained image embeddings into the prototype enhancement model; m ;

[0020] Step 2.3: Construct a batch of learnable class prototype repositories for each class, and the number of learnable class prototypes for each class is

[0021] Step 2.4: Calculate the number of learnable class prototypes that can be used for each class in the batch according to the number of training samples of different classes in the training set in Step 1;

[0022] Step 2.5: Model the semantic enhancement direction of the image embeddings obtained in Step 2.3 of the same class as a multi-dimensional Gaussian distribution;

[0023] Step 2.6: Calculate the similarity between the image embeddings and the learnable class prototypes of other classes for the learnable class prototypes in the learnable class prototype repository obtained in Step 2.4 through the cosine distance, select the learnable class prototype of other classes that is most similar to the image embedding, and then perform linear interpolation between the image embedding and the similar learnable class prototype to obtain the optimized learnable class prototype;

[0024] Step 2.7: Randomly select the corresponding number of learnable class prototypes for each class in the learnable class prototype repository obtained in Step 2.4 according to the number of learnable class prototypes that can be used for each class in the batch obtained in Step 2.5; Apply a class-aware Gaussian perturbation constructed based on the semantic enhancement direction modeled by the multi-dimensional Gaussian distribution in Step 2.6 to the learnable class prototypes to obtain the enhanced class prototypes;

[0025] Step 2.10: According to the set Q obtained in Step 2.9, use the exponential weighted moving average method in batches during training to estimate the covariance matrix of the Gaussian distribution;

[0026] Step 2.11: Randomly select the learnable class prototypes corresponding to the number of classes in the learnable class prototype repository obtained in Step 2.4, the enhanced class prototypes obtained in Step 2.8, and the image embeddings obtained in Step 2.3, and splice them according to the 0th dimension to obtain the set S for balanced contrast learning;

[0027] Step 2.12: Calculate the balanced implicit enhanced contrast learning loss for the set S obtained in Step 2.11;

[0028] Step 2.13: Sum the cross-entropy loss obtained in Step 2.2 and the balanced implicit enhanced contrast learning loss obtained in Step 2.12 to obtain the overall loss;

[0029] Step 2.14: By minimizing the overall loss obtained in Step 2.13, adjust the parameters of the deep convolutional neural network, and use the backpropagation algorithm for global parameter adjustment until the loss no longer decreases, then the model converges to obtain the deep convolutional neural network with optimized parameters.

[0030] Optionally, the specific method of Step 3 is as follows:

[0031] Unify the long-tailed medical images in the test set of Step 1 to the same resolution, then input the long-tailed medical images with the same unified resolution into the feature extraction layer of the image classification model based on the deep convolutional neural network obtained in Step 2 to extract image features, and finally input the extracted image features into the classification layer of the image classification model based on the deep convolutional neural network to obtain the classification result.

[0032] A medical image classification system based on contrast learning, comprising:

[0033] A dataset collection and processing module that divides the collected long-tailed medical image dataset into a training set and a test set, and performs weak data augmentation and strong data augmentation on the training set images in batches according to a preset scheme to obtain weakly data-augmented images and strongly data-augmented images;

[0034] An image classification model establishment and training module that performs a contrast learning task on the obtained weakly data-augmented images and strongly data-augmented images through an image classification model based on a deep convolutional neural network, learns the network parameters, and obtains an image classification model based on the deep convolutional neural network with optimized parameters;

[0035] An image classification model testing module that uses the obtained image classification model based on the deep convolutional neural network with optimized parameters to classify the long-tailed medical images in the test set.

[0036] A computer storage medium stores a computer program which, when executed by a processor, can implement the steps of a medical image classification method based on contrast learning.

[0037] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a medical image classification method, system and storage medium based on contrast learning, having the following beneficial effects:

[0038] 1. By modeling the semantic enhancement direction of the embedded images of the same category obtained in the present invention as a multi-dimensional Gaussian distribution, it has universality and generality. And by constructing a batch of learnable class prototypes in step 2.4, obtaining the optimized learnable class prototypes in step 2.7, and obtaining the enhanced class prototypes in step 2.8, compared with the conventional method based on online estimation of the feature space, the present invention can estimate a more accurate class covariance matrix for each class.

[0039] 2. In step 2.12 of the present invention, features for balancing implicit enhancement contrast learning are constructed, and further the optimized overall loss in step 2.13 is obtained, improving the generalization ability of the model, significantly enhancing the classification accuracy, greatly improving the chromosome classification efficiency, and especially showing more superiority when facing complex chromosome abnormal images.

[0040] 3. Through the image processing of weak data augmentation and strong data augmentation described in step 1 of the present invention and the method of modeling the semantic enhancement direction of image embedding as a multi-dimensional Gaussian distribution in step 2.6, not only the original features of the image are retained, but also it is convenient for the construction of the enhanced class prototypes in subsequent step 2.8. The present invention does not lose the original morphological structure feature information of the chromosome, reduces the technical cost, is convenient for manual review, and has a wider applicability.

[0041] In summary, by introducing a contrast learning mechanism and a prototype enhancement strategy, the present invention can significantly improve the performance of the convolutional neural network in the classification of abnormal chromosome images, having advantages such as high precision, high efficiency, low cost and wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0043] Figure 1 It is a schematic diagram of the method principle of the present invention.

[0044] Figure 2 It is a schematic diagram of the prototype enhancement model of the present invention.

[0045] Figure 3(a) is a display diagram of Y chromosome microdeletion abnormality of 25-24-7 (qs), Figure 3(b) is a display diagram of the abnormality of the reduction of the heterochromatic region of the long arm of the Y chromosome of 25-24-2 (qh-), Figure 3(c) is a display diagram of the abnormality of the increase of the heterochromatic region of the long arm of the Y chromosome of 25-24-1 (qh+), and Figure 3(d) is a display diagram of the inversion abnormality of the Y chromosome of 25-24-3 (inv).

[0046] Figure 4 It is an introduction diagram of the ISIC 2018 dataset used in the embodiment of the present invention.

[0047] Figure 5 It is an introduction diagram of the APTOS 2019 dataset used in the embodiment of the present invention. Detailed implementation manners

[0048] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0049] The embodiment of the present invention discloses a medical image classification method based on contrast learning, including:

[0050] Step 1: Divide the long-tailed medical image dataset into a training set and a test set according to a ratio of 7:3, and perform weak data augmentation and strong data augmentation on the training set images in batches according to a preset scheme to obtain weak data-augmented images and strong data-augmented images; wherein, the weak data augmentation includes: cropping and resizing, horizontal flipping, and color jitter; the strong data augmentation includes: cropping and resizing, horizontal flipping, vertical flipping, color jitter, and grayscale conversion;

[0051] The long-tailed medical image dataset is briefly explained as follows. The problem of long-tailed medical image classification is usually caused by the difference in incidence and the difficulty of data collection. Some diseases are common, while others are rare, which makes the collected dataset show an unbalanced phenomenon, with more samples in some categories and fewer samples in some categories; when the samples are arranged in descending order according to the number of samples in each category, a dataset similar to a long-tailed distribution is obtained.

[0052] Step 1.1: Perform weak data augmentation on the training set images. Crop and adjust the image size to 224x224 pixels. Randomly sample 80% of the images and randomly apply a color jitter transformation with a hue change probability of 0.4, a saturation change probability of 0.4, a brightness change probability of 0.4, and a contrast change probability of 0.1. Randomly horizontally flip the images to obtain weakly data-augmented images.

[0053] Step 1.2: Perform strong data augmentation on the training set images. Crop and adjust the image size to 224x224 pixels, and the cropped area size ranges from 80% to 100% of the original image. Randomly sample 50% of the images for random horizontal flipping. Randomly sample 50% of the images for random vertical flipping. Randomly rotate the images, and the rotation angle can be 0°, 90°, 180°, or 270°. Randomly sample 80% of the images and randomly apply a color jitter transformation with a hue change probability of 0.4, a saturation change probability of 0.4, a brightness change probability of 0.4, and a contrast change probability of 0.1. Randomly sample 20% of the images to convert them to grayscale images to obtain strongly data-augmented images.

[0054] Step 2: Use the weakly data-augmented images obtained in Step 1.1 and the strongly data-augmented images obtained in Step 1.2 to perform a contrastive learning task through a deep convolutional neural network, learn the network parameters, and obtain an image classification model based on the deep convolutional neural network with optimized parameters. Specifically, it includes the following steps:

[0055] The deep convolutional neural network consists of a backbone network, a mapping network, and a prototype enhancement model. The backbone network includes a feature extraction layer and a classification layer. Among them, the feature extraction layer uses a residual neural network (ResNet), and the classification layer consists of a fully connected layer and a Softmax layer. The mapping network is a two-layer multi-layer perceptron (MLP). First, input the weakly data-augmented images and strongly data-augmented images obtained in Step 1 into the feature extraction layer of the backbone network for feature extraction, and then input them into the mapping network to obtain image embeddings. At the same time, input the features extracted by the feature extraction layer into the classification layer to obtain the basic cross-entropy loss. Second, input the image embeddings obtained after passing through the mapping network into the prototype enhancement model, and use the prototype enhancement contrastive learning strategy to generate learnable class prototypes for data augmentation. Finally, obtain the balanced implicit enhancement contrastive learning loss.

[0056] Step 2.1: Input the weakly data-augmented images and strongly data-augmented images obtained in Step 1 into the feature extraction layer of the deep convolutional neural network respectively to obtain features with a dimension of (2B, 2048), including the i features of the weakly data-augmented image of the th sample and the features of the strongly data-augmented image of the sample , where B is the number of batch training samples;

[0057] Step 2.2: Input the weakly data-augmented image features obtained in Step 2.1 into the classification layer of the deep convolutional neural network to obtain the prediction (B, C), where C is the number of categories in the training set in Step 1, and calculate the cross-entropy loss through the cross-entropy loss function; the calculation formula is as follows:

[0058] ;

[0059] Among them, is the weakly data-augmented image feature, and are the weights of the th and th nodes of the classification layer (fully connected layer), and are the biases of the th and jth nodes of the classification layer (fully connected layer), N represents the number of training samples, C is the number of categories in the training set;

[0060] Step 2.3: Input the weakly data-augmented image features obtained in Step 2.1 and the strongly data-augmented image features into the mapping network to obtain a (2B, 128)-dimensional image embedding for contrastive learning training, specifically divided into a weakly augmented image embedding and a strongly augmented image embedding , and input the obtained image embedding into the prototype augmentation model, and use the prototype augmentation contrastive learning strategy to generate learnable class prototypes for data augmentation;

[0061] As Figure 2 shown, Step 2.4: Construct a batch of learnable class prototype repositories for each class , and the number of learnable class prototypes for each class is 10;

[0062] Step 2.5: According to the number of training samples of different classes in the training set in Step 1, calculate the number of learnable class prototypes that can be used for each class in the batch as:

[0063] ;

[0064] Among them represents the number of training samples of the cth class, is the number of samples of the class with the most samples in the training set;

[0065] Step 2.6: Model the semantic enhancement direction of the image embeddings obtained in Step 2.3 of the same class as a multi-dimensional Gaussian distribution , represents the covariance matrix of class ;

[0066] Step 2.7: Aggregate the learnable class prototype repository obtained in Step 2.4 with the image embedding to obtain a set , where the image embedding v is obtained by aggregating the weakly augmented image embeddings of a mini-batch and the strongly augmented image embeddings . For each element in and its corresponding label , the set can be divided into two subsets and . For each image embedding , find the negative class embeddings that are semantically similar to it to form . The semantic similarity is calculated by the following formula:

[0067] ;

[0068] Select the learnable class prototype of other classes that is most similar to this image embedding, and then perform linear interpolation between this image embedding and the similar learnable class prototype to obtain the enhanced image embedding as:

[0069] ;

[0070] where is the random interpolation coefficient of each sample embedding, t has an upper bound value of 0.0001, is the set of enhanced image embeddings of the image embedding .

[0071] Step 2.8: According to the number of learnable class prototypes available for each class in the batch obtained in Step 2.5, select the corresponding number of learnable class prototypes in the learnable class prototype repository obtained in Step 2.4 through class-aware sampling Apply a class-aware Gaussian perturbation constructed based on the semantic enhancement direction modeled by the multi-dimensional Gaussian distribution in Step 2.6 to the learnable class prototype to obtain the enhanced class prototype ; where and are the vectors composed of the square roots of the diagonal elements of the covariance matrix of class . The resulting set of enhanced prototypes is 。

[0072] Step 2.9: According to the image embedding obtained in Step 2.3 , the enhanced image embedding obtained in Step 2.7 , the learnable class prototypes obtained in Step 2.8 and the enhanced class prototypes are concatenated along the 0th dimension to obtain a set ;

[0073] Step 2.10: Based on the set Q obtained in Step 2.9, during training, use the exponential weighted moving average method (EMA method) in batch form to estimate the covariance matrix of the Gaussian distribution:

[0074] ;

[0075] where and , represents the total number of sample embeddings of the th class in the current Q and the learnable proxy, is the total number of sample embeddings of the th class and the learnable proxy;

[0076] Step 2.11: According to the image embedding obtained in Step 2.3 , the learnable class prototypes obtained in Step 2.8 and the enhanced class prototypes are concatenated along the 0th dimension to obtain a set ; The set is obtained for balanced contrastive learning;

[0077] Step 2.12: For the set S obtained in Step 2.11, calculate the balanced implicit enhanced contrastive learning loss according to the following formula:

[0078] ;

[0079] where is the adjustment factor, where , represents the set of sample embeddings and learnable prototypes of class S in the set c , B is the number of samples in a batch, is the number of samples of class c in a batch, is the number of samples of the cth class in the training set;

[0080] Step 2.13: Sum up the cross-entropy loss obtained in Step 2.2 and the balanced implicit augmented contrastive learning loss obtained in Step 2.12 to obtain the overall loss :

[0081] ;

[0082] wherein, and are used to adjust the weights of different losses, ;

[0083] Step 2.14: By minimizing the overall loss obtained in Step 2.13 , adjust the parameters of the deep convolutional neural network, and use the backpropagation algorithm for global parameter adjustment until the loss no longer decreases, then the model converges, obtaining the deep convolutional neural network with optimized parameters, and further obtaining the image classification model based on the deep convolutional neural network with optimized parameters.

[0084] The present invention enables the model to learn stable features through the prototype augmentation and contrastive learning methods.

[0085] Step 3: Use the image classification model based on the deep convolutional neural network with optimized parameters obtained in Step 2 to classify the long-tailed medical images in the test set of Step 1.

[0086] Unify the long-tailed medical images in the test set of Step 1 to the same resolution, then input the long-tailed medical images with the unified same resolution into the deep convolutional neural network with optimized parameters obtained in Step 2 to extract image features, and finally input the extracted image features into the classification layer to obtain the classification result.

[0087] As Figure 1 shown, in the above steps, for the feature extraction layer of the deep convolutional neural network backbone network, the network structure adopts the residual neural network (ResNet-50); the weak data-augmented image features of the i th sample and the strong data-augmented image features of the i th sample ; the mapping network from the image features to the image embedding is the MLP, i.e., the two-layer perceptron; the weakly augmented image embedding and the strongly augmented image embedding ; the prototype augmentation model.

[0088] The long-tailed medical image classification method based on prototype-enhanced contrast learning proposed by the present invention combines technologies such as image enhancement, convolutional neural network, prototype generation, and contrast learning, significantly improving the classification efficiency and accuracy of abnormal long-tailed medical images, providing a new solution for intelligent and automated abnormal disease screening, and having important clinical application value.

[0089] The present invention can be widely applied to fields such as automated chromosomal abnormality screening, medical image analysis, genomics research, etc., and has significant advantages especially in improving the detection efficiency and accuracy of chromosomal abnormalities.

[0090] Experimental analysis

[0091] 1. Dataset introduction:

[0092] The present invention conducted experiments on two publicly available datasets: the ISIC 2018 dataset provided by the International Skin Imaging Collaboration (ISIC) and the APTOS 2019 dataset provided by the APTOS 2019 Blind Detection Competition.

[0093] The ISIC 2018 dataset was released by the International Skin Imaging Collaboration (ISIC) and is mainly used for image classification and segmentation tasks of skin lesions. This dataset contains a large number of high-resolution skin lesion images with professional annotations, including classification labels of lesions and segmentation masks of lesions. These images cover various types of skin lesions, such as benign and malignant melanoma, basal cell carcinoma, keratotic lesions, etc. The ISIC 2018 dataset provides rich training resources for the automated diagnosis of skin lesions and has become an important benchmark for skin lesion detection tasks due to the image quality and diversity. The number of samples in each category in the dataset is as Figure 4 shown.

[0094] The APTOS 2019 dataset was released by the Asia Pacific Tele-Ophthalmology Society (APTOS) and is mainly used for the grading task of diabetic retinopathy (DR). The dataset contains retinal color fundus images with different lesion grading labels, divided into five levels: normal, mild, moderate, severe, and proliferative diabetic retinopathy. The images in the APTOS 2019 dataset were collected from actual screening scenarios, with uneven image quality and containing real factors such as halos and noise, increasing the challenge of the grading task. This dataset is widely used to evaluate and optimize diabetic retinopathy detection algorithms and plays an important role especially in the training and validation of deep learning models. The number of samples in each category in the dataset is asFigure 5 as shown

[0095] APTOS 2019 contains 3,662 labeled images, classifying the severity of diabetic retinopathy into 5 categories, from no DR to proliferative DR, and ISIC2018 contains 10,015 pigmented skin lesion images in 7 categories. The two datasets considered show an important problem related to the class imbalance issue, that is, the samples belonging to the most dominant class account for half of the entire dataset. For a fair comparison, based on previous studies, the present invention randomly divides each dataset into 70% training data and 30% test data.

[0096] Meanwhile, to verify the method proposed by the present invention, the First Affiliated Hospital of Xi'an Jiaotong University in Xi'an, China provided 1,148 microscopic metaphase images. All chromosomes in the metaphase images were manually annotated by cytologists. The masks of each metaphase image were manually marked by the researchers in the laboratory of the present invention and then verified by cytologists. Meanwhile, 17,808 single chromosomes were collected from 1,148 metaphase chromosome images. Partial data display diagrams of the ACR-v1 long-tail dataset used in the embodiments of the present invention are as shown below. Among them, Fig. 3(a) is a display diagram of 25-24-7 (qs), i.e., Y chromosome microdeletion anomaly, Fig. 3(b) is a display diagram of 25-24-2 (qh-), i.e., reduction anomaly of heterochromatin region on the long arm of the Y chromosome, Fig. 3(c) is a display diagram of 25-24-1 (qh+), i.e., increase anomaly of heterochromatin region on the long arm of the Y chromosome, and Fig. 3(d) is a display diagram of 25-24-3 (inv), i.e., inversion anomaly of the Y chromosome. Based on the obtained chromatid images and label information, the present invention constructs the ACR-v1 long-tail dataset. Table 1 shows the detailed information of the dataset.

[0097] Table 1 ACR-v1 Long-tail Dataset

[0098] Category 25-24-7 (qs) 25-24-2 (qh-) 25-24-1 (qh+) 25-24-3 (inv) Train 347 105 67 13 Test 150 45 30 6

[0099] 2. Experiment Introduction

[0100] To evaluate the performance of the IPECL of the present invention, the method of the present invention was compared with more than 10 advanced methods. Among them, Focal, LDAM-DRW, Logit Adjusted, and MWNL are reweighted loss methods. BBN is a method based on a rebalancing training strategy, while Hybrid-SC, SCL, BCL, TSC, and the method of the present invention, IPECL, are CL-based methods. In addition, MWNL and SCL have been verified to perform well in the medical long-tail image classification task. To ensure fairness, the present invention retrains all methods and re-runs their published codes on the datasets divided by the present invention under the same experimental settings.

[0101] The present invention also confirms that all models converge and select the best checkpoints. The results are shown in Tables 2, 3, and 4. It can be seen that IPECL has significant advantages and reaches the highest level in most metrics on the two datasets. It is worth noting that the IPECL of the present invention has a great improvement compared with other imbalance methods. For example, on ISIC 2018, the F1 is increased by 1.8% compared with the TSC method, and on APTOS - 2019, the Acc is increased by 1.1% compared with the ProCo method.

[0102] On the ACR - v1 dataset proposed by the present invention, the IPECL method proposed by the present invention achieves the best results in terms of overall accuracy and accuracy of each category compared with the existing methods. The overall accuracy is increased by 1.3%. The category accuracies for categories 0, 1, and 2 are stably close to 100%. For category 3 where the existing methods all perform poorly, there is also a great improvement. It is worth mentioning that, different from most previous methods that reduce the performance of the head classes, the IPECL of the present invention further improves the performance of the head classes while improving the performance of the tail classes.

[0103] Table 2 Experimental results on the ACR - v1 long - tailed dataset

[0104]

[0105] Table 3 Experimental results on the ISIC 2018 dataset

[0106]

[0107] Table 4 Experimental results on the APTOS 2019 dataset

[0108]

[0109] The comparison results between the IPECL proposed by the present invention and other existing methods on ISIC 2018 and APTOS 2019 are shown in Tables 3 and 4. It can be seen from the tables that IPECL is always superior to other methods. In addition, IPECL achieves better performance on the ISIC2018 dataset with a larger imbalance factor, and the F1 is increased by 5.3% compared with the Focal method. This result is mainly attributed to the fact that when the imbalance problem is more serious, the traditional supervised contrastive loss will lead to more serious representation learning bias. Focal uses a set of parameter centers in supervised contrastive learning, and these centers are given greater weights. However, the prototypes used in IPECL supplement the samples of each class to ensure that all classes appear in each mini - batch. Compared with Focal, IPECL achieves a better overall accuracy of 84.8%, and there are significant accuracy improvements in the head and minority classes. This shows that IPECL has significant improvements from the minority to the majority in most categories.

[0110] A medical image classification system based on contrast learning, comprising:

[0111] A dataset collection and processing module that divides the collected long-tail medical image dataset into a training set and a test set, and performs weak data augmentation and strong data augmentation on the training set images in batches according to a preset scheme to obtain weakly data-augmented images and strongly data-augmented images;

[0112] An image classification model establishment and training module that performs a contrast learning task on the obtained weakly data-augmented images and strongly data-augmented images through an image classification model based on a deep convolutional neural network, learns network parameters, and obtains an image classification model based on a deep convolutional neural network with optimized parameters;

[0113] An image classification model testing module that uses the obtained image classification model based on a deep convolutional neural network with optimized parameters to classify the long-tail medical images in the test set.

[0114] A computer storage medium stores a computer program, and when the computer program is executed by a processor, it can implement the steps of a medical image classification method based on contrast learning.

[0115] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0116] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A medical image classification method based on contrastive learning, characterized in that Including: Step 1: Divide the collected long-tail medical image dataset into a training set and a test set. According to a preset scheme, perform weak data augmentation and strong data augmentation on the training set images in a batch processing manner to obtain weakly data-augmented images and strongly data-augmented images. Step 2: Perform a contrastive learning task on the weakly data-augmented images and strongly data-augmented images obtained in Step 1 through an image classification model based on a deep convolutional neural network, learn the network parameters, and obtain an image classification model based on a deep convolutional neural network with optimized parameters. Step 3: Use the image classification model based on a deep convolutional neural network with optimized parameters obtained in Step 2 to classify the long-tail medical images in the test set of Step 1. The said Step 2 includes the following steps: Step 2.1: Input the weakly data-augmented images and strongly data-augmented images obtained in Step 1 into the feature extraction layer of the image classification model based on a deep convolutional neural network, to obtain features with a dimension of (2 B， 2048), including the weakly data-augmented image features of the i th sample and the strongly data-augmented image features of the i th sample, B being the number of batch training samples; Step 2.2: Input the weakly data-augmented image features obtained in Step 2.1 into the classification layer of the image classification model based on a deep convolutional neural network to obtain predictions ( B , C ), where C is the number of categories in the training set of Step 1, and calculate the cross-entropy loss through the cross-entropy loss function; Step 2.3: Input the weakly data-augmented image features and strongly data-augmented image features obtained in Step 2.1 into a mapping network to obtain (2B, 128)-dimensional image embeddings for contrastive learning training, specifically divided into weakly augmented image embeddings and strongly augmented image embeddings, and input the obtained image embeddings into a prototype enhancement model. Step 2.4: Construct a batch of learnable class prototype repositories for each class, where the number of learnable class prototypes for each class is m ; Step 2.5: Calculate the number of learnable class prototypes that can be used for each class in the batch according to the number of training samples of different classes in the training set in Step 1. Step 2.6: Model the semantic enhancement direction of the image embeddings obtained in Step 2.3 of the same class as a multi-dimensional Gaussian distribution. Step 2.7: Calculate the similarity between the image embedding and the learnable class prototypes of other classes through the cosine distance for the learnable class prototypes in the learnable class prototype repository obtained in Step 2.4, select the learnable class prototype of other classes that is most similar to this image embedding, and then perform linear interpolation between this image embedding and the most similar learnable class prototype to obtain an optimized learnable class prototype. Step 2.8: According to the number of learnable class prototypes that can be used for each class in the batch obtained in Step 2.5, randomly select the corresponding number of learnable class prototypes in the learnable class prototype repository obtained in Step 2.4; apply a class-aware Gaussian perturbation constructed based on the semantic enhancement direction modeled by the multi-dimensional Gaussian distribution in Step 2.6 to the learnable class prototypes to obtain enhanced class prototypes. Step 2.9: Concatenate the optimized learnable class prototypes obtained in Step 2.7, the enhanced class prototypes obtained in Step 2.8, and the image embeddings obtained in Step 2.3 according to the 0th dimension to obtain a set Q. Step 2.10: According to the set Q obtained in Step 2.9, estimate the covariance matrix of the Gaussian distribution by using the exponential weighted moving average method in a batch processing manner during training. Step 2.11: Concatenate the learnable class prototypes randomly selected from the learnable class prototype repository obtained in Step 2.4, the enhanced class prototypes obtained in Step 2.8, and the image embeddings obtained in Step 2.3 according to the 0th dimension to obtain a set S for balanced contrastive learning. Step 2.12: Calculate the balanced implicit enhancement contrastive learning loss for the set S obtained in Step 2.

11. Step 2.13: Sum the cross-entropy loss obtained in Step 2.2 and the balanced implicit enhancement contrastive learning loss obtained in Step 2.12 to obtain the overall loss. Step 2.14: Adjust the parameters of the deep convolutional neural network by minimizing the overall loss obtained in Step 2.

13. Use the backpropagation algorithm for global parameter adjustment until the loss no longer decreases, at which point the model converges, and a deep convolutional neural network with optimized parameters is obtained.

2. The medical image classification method based on contrastive learning according to claim 1, wherein The weak data augmentation includes: cropping and resizing, horizontal flipping, and color jittering; the strong data augmentation includes: cropping and resizing, horizontal flipping, vertical flipping, color jittering, and grayscaling.

3. A medical image classification method based on contrastive learning according to claim 1, characterized in that, The image classification model based on the deep convolutional neural network consists of a backbone network, a mapping network, and a prototype enhancement model. The backbone network includes a feature extraction layer and a classification layer. Among them, the feature extraction layer uses a residual neural network, and the classification layer consists of a fully connected layer and a Softmax layer; the mapping network is a two-layer perceptron.

4. A medical image classification method based on contrastive learning according to claim 3, characterized in that, The training process of the image classification model based on the deep convolutional neural network is as follows: First, input the weakly data-augmented images and strongly data-augmented images obtained in Step 1 into the feature extraction layer of the backbone network for feature extraction, and then input them into the mapping network to obtain image embeddings. At the same time, input the features extracted by the feature extraction layer into the classification layer to obtain the basic cross-entropy loss. Second, input the image embeddings obtained after passing through the mapping network into the prototype enhancement model, and use the prototype enhancement contrast learning strategy to generate learnable class prototypes for data augmentation. Finally, obtain the balanced implicit enhancement contrast learning loss.

5. A medical image classification method based on contrastive learning according to claim 1, wherein The specific method of Step 3 is as follows: Unify the long-tail medical images in the test set of Step 1 to the same resolution, and then input the long-tail medical images with the same unified resolution into the feature extraction layer of the image classification model based on the deep convolutional neural network with optimized parameters obtained in Step 2 to extract image features. Finally, input the extracted image features into the classification layer of the image classification model based on the deep convolutional neural network to obtain the classification result.

6. A medical image classification system based on contrastive learning, characterized in that, Applying the medical image classification method based on contrast learning according to any one of claims 1-5, includes: A dataset acquisition and processing module that divides the collected long-tail medical image dataset into a training set and a test set, and performs weak data augmentation and strong data augmentation on the training set images in batches according to a preset scheme to obtain weakly data-augmented images and strongly data-augmented images; An image classification model establishment and training module that performs a contrast learning task on the obtained weakly data-augmented images and strongly data-augmented images through an image classification model based on a deep convolutional neural network, learns the network parameters, and obtains an image classification model based on a deep convolutional neural network with optimized parameters; An image classification model testing module that uses the obtained image classification model based on the deep convolutional neural network with optimized parameters to classify the long-tail medical images in the test set.

7. A computer storage medium, characterized in that, A computer storage medium stores a computer program, and when the computer program is executed by a processor, it can implement the steps of the medical image classification method based on contrast learning according to any one of claims 1-5.

Images