Medical image classification method and system based on comparative learning, and storage medium

Abstract

The invention discloses a medical image classification method and system based on comparative learning, and a storage medium, and relates to the technical field of long-tail medical image classification. A long-tail medical image data set is divided into a training set and a test set, and weak data enhancement and strong data enhancement are respectively performed on images of the training set in a batch processing form according to a preset scheme; performing a comparative learning task on the obtained weak data enhanced image and the strong data enhanced image through a deep convolutional neural network, learning network parameters, and obtaining the deep convolutional neural network after parameter optimization; the long-tail medical images in the test set are classified; through the prototype enhancement contrast learning strategy, the learnable prototype is generated for data enhancement, the balance implicit enhancement contrast learning loss is obtained, and the method has the advantages of high precision, high efficiency, low cost, wide applicability and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of long-tail medical image classification, and more specifically to a medical image classification method, system and storage medium based on contrastive learning. Background Art

[0002] At present, chromosomal abnormality refers to abnormalities in the structure or number of chromosomes, including common chromosomal lesions such as monosomy and trisomy, which may lead to genetic diseases or developmental disorders. Traditional chromosome detection methods rely on manual microscope analysis or partially automated image processing technology, but these methods are time-consuming, limited in accuracy, rely on the experience of professionals, and are difficult to screen on a large scale. Therefore, how to improve the automation level of chromosome image classification, reduce labor costs, and improve screening efficiency is a technical problem that needs to be solved urgently. At present, most of the research on genetic abnormality detection is based on gene abnormality detection, including base sequences, gene loci, etc., and there are few related detections based on chromosome images; secondly, for the research on chromosomal abnormalities, deep learning technology is simply used as an auxiliary detection for denoising or reducing the false positive rate of abnormal sites. From the perspective of research methods, in this research field, the deep learning models currently used are mostly simple basic models such as CNN and RNN for feature extraction, and the effect is relatively general.

[0003] The patent application document with publication number CN109300111A discloses a chromosome classification method based on deep learning. By calculating the chromosome morphological features, the ResNet residual neural network is trained to perform chromosome classification. However, since the selected morphological features are relatively simple to model, the model classification accuracy is not high and effective classification of abnormal chromosomes cannot be achieved.

[0004] The patent application document with publication number CN109285174A discloses chromosome segmentation and karyotype analysis based on crowdsourcing and deep learning. The fragments identified by crowdsourcing are preprocessed to improve the classification achieved using a deep convolutional network (DCNN). However, since the chromosomes are classified after straightening the curved chromosomes, the original chromosome morphological structure is destroyed, resulting in loss of feature information.

[0005] Therefore, how to improve the accuracy of long-tail medical image classification is an urgent problem that technicians in this field need to solve. Summary of the invention

[0006] In view of this, the present invention provides a medical image classification method, system and storage medium based on contrastive learning. Through the prototype enhancement contrastive learning strategy, a learnable prototype is generated for data enhancement, and a balanced implicit enhancement contrastive learning loss is obtained, which has the advantages of high precision, high efficiency, low cost and wide applicability.

[0007] In order to achieve the above object, the present invention adopts the following technical solution: A medical image classification method based on contrastive learning, comprising: Step 1: Divide the collected long-tail medical image dataset into a training set and a test set, and perform weak data enhancement and strong data enhancement on the training set images in a batch processing form according to a preset scheme to obtain weak data enhanced images and strong data enhanced images; Step 2: Perform a comparative learning task on the weak data enhanced image and the strong data enhanced image obtained in step 1 through an image classification model based on a deep convolutional neural network, learn network parameters, and obtain an image classification model based on a deep convolutional neural network with optimized parameters; Step 3: Use the deep convolutional neural network-based image classification model with optimized parameters obtained in step 2 to classify the long-tail medical images in the test set in step 1.

[0008] Optionally, the weak data enhancement includes: cropping and resizing, horizontal flipping, and color dithering; the strong data enhancement includes: cropping and resizing, horizontal flipping, vertical flipping, color dithering, and grayscale conversion.

[0009] Optionally, the image classification model based on deep convolutional neural network is composed of a backbone network, a mapping network and a prototype enhancement model, the backbone network includes a feature extraction layer and a classification layer, wherein the feature extraction layer adopts a residual neural network, and the classification layer consists of a fully connected layer and a Softmax layer; the mapping network is a two-layer perceptron.

[0010] Optionally, the training process of the image classification model based on deep convolutional neural network is: first, the weakly data enhanced image and the strongly data enhanced image obtained in step 1 are input into the feature extraction layer of the backbone network for feature extraction and then input into the mapping network to obtain image embedding. At the same time, the features extracted by the feature extraction layer are input into the classification layer to obtain basic cross entropy loss; secondly, the image embedding obtained after the mapping network is input into the prototype enhancement model, and the prototype enhancement contrast learning strategy is used to generate a learnable class prototype for data enhancement, and finally, a balanced implicit enhancement contrast learning loss is obtained.

[0011] Optionally, step 2 includes the following steps: Step 2.1: Input the weak data augmented image and the strong data augmented image obtained in step 1 into the feature extraction layer of the image classification model based on the deep convolutional neural network, and obtain a feature extraction layer with a dimension of (2 B， 2048) dimension features, including the i The weak data enhancement image features of samples and the i Strong data-enhanced image features of samples, B is the number of batch training samples; Step 2.2: Input the weak data-enhanced image features obtained in step 2.1 into the classification layer of the image classification model based on the deep convolutional neural network to obtain the prediction ( B , C ), C is the number of categories in the training set in step 1, and the cross entropy loss is calculated using the cross entropy loss function; Step 2.3: Input the weak data enhancement image features and strong data enhancement image features obtained in step 2.1 into the mapping network to obtain (2B, 128)-dimensional image embeddings for contrastive learning training, which are specifically divided into weak enhancement image embeddings and strong enhancement image embeddings. The obtained image embeddings are input into the prototype enhancement model; Step 2.4: Construct a batch of learnable class prototype repositories for each class. The number of learnable class prototypes for each class is m ; Step 2.5: Calculate the number of learnable class prototypes that can be used in the batch for each class based on the number of training samples of different categories in the training set in step 1; Step 2.6: Model the semantic enhancement direction of the image embedding obtained in step 2.3 of the same category as a multi-dimensional Gaussian distribution; Step 2.7: Calculate the similarity between the image embedding and other types of learnable class prototypes in the learnable class prototype repository obtained in step 2.4 by cosine distance, select other types of learnable class prototypes that are most similar to the image embedding, and then perform linear interpolation between the image embedding and the similar learnable class prototypes to obtain the optimized learnable class prototype; Step 2.8: According to the number of learnable class prototypes available for each class in the batch obtained in step 2.5, randomly select the learnable class prototypes of the corresponding number of classes in the learnable class prototype repository obtained in step 2.4; apply the class-aware Gaussian perturbation constructed based on the semantic enhancement direction of multi-dimensional Gaussian distribution modeling in step 2.6 to the learnable class prototypes to obtain the enhanced class prototypes; Step 2.9: Concatenate the optimized learnable prototype obtained in step 2.7, the enhanced prototype obtained in step 2.8, and the image embedding obtained in step 2.3 according to the 0th dimension to obtain a set Q; Step 2.10: Based on the set Q obtained in step 2.9, the covariance matrix of the Gaussian distribution is estimated using the exponentially weighted moving average method in the form of batch processing during training; Step 2.11: randomly select the corresponding number of learnable prototypes from the learnable prototype repository obtained in step 2.4, the enhanced prototype obtained in step 2.8, and the image embedding obtained in step 2.3, and concatenate them according to the 0th dimension to obtain a set S for balanced contrastive learning; Step 2.12: Calculate the balanced implicit enhanced contrastive learning loss for the set S obtained in step 2.11; Step 2.13: Sum the cross entropy loss obtained in step 2.2 and the balanced implicit enhanced contrastive learning loss obtained in step 2.12 to get the overall loss; Step 2.14: By minimizing the overall loss obtained in step 2.13, adjust the parameters of the deep convolutional neural network and use the back propagation algorithm to adjust the global parameters until the loss no longer decreases, the model converges, and the deep convolutional neural network with optimized parameters is obtained.

[0012] Optionally, the specific method of step 3 is: The long-tail medical images in the test set in step 1 are unified to the same resolution, and then the long-tail medical images unified to the same resolution are input into the feature extraction layer of the image classification model based on the deep convolutional neural network after parameter optimization obtained in step 2 to extract image features. Finally, the extracted image features are input into the classification layer of the image classification model based on the deep convolutional neural network to obtain the classification results.

[0013] A medical image classification system based on contrastive learning, comprising: The data set acquisition and processing module divides the acquired long-tail medical image data set into a training set and a test set, and performs weak data enhancement and strong data enhancement on the training set images in a batch processing form according to a preset scheme to obtain weak data enhanced images and strong data enhanced images; The image classification model establishment and training module compares the obtained weak data enhancement images and strong data enhancement images through the image classification model based on deep convolutional neural network to learn the network parameters and obtain the image classification model based on deep convolutional neural network after parameter optimization. The image classification model testing module uses the image classification model based on deep convolutional neural network with optimized parameters to classify the long-tail medical images in the test set.

[0014] A computer storage medium stores a computer program, which can implement the steps of a medical image classification method based on contrast learning when executed by a processor.

[0015] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a medical image classification method, system and storage medium based on contrastive learning, which have the following beneficial effects: 1. The present invention has universality and generality by modeling the semantic enhancement direction of the image embedding of the same category as a multi-dimensional Gaussian distribution. A batch of learnable class prototypes are constructed through step 2.4, the optimized learnable class prototypes obtained in step 2.7 and the enhanced class prototypes obtained in step 2.8. Compared with the conventional method based on online estimation of feature space, the present invention can estimate a more accurate class covariance matrix for each class.

[0016] 2. Step 2.12 of the present invention constructs features for balancing implicit enhanced contrast learning, and further obtains the optimized overall loss in step 2.13, which improves the generalization ability of the model, significantly improves the classification accuracy, and greatly improves the chromosome classification efficiency, especially when facing complex chromosome abnormality images.

[0017] 3. The present invention not only retains the original features of the image, but also facilitates the construction of the enhanced class prototype in the subsequent step 2.8 through the image processing of weak data enhancement and strong data enhancement described in step 1 and the method of multi-dimensional Gaussian distribution modeling of the semantic enhancement direction of image embedding in step 2.6. The present invention does not lose the original morphological structure feature information of the chromosome, reduces the technical cost, and is convenient for manual review, and has wider applicability.

[0018] In summary, the present invention can significantly improve the performance of convolutional neural networks in abnormal chromosome image classification by introducing contrastive learning mechanism and prototype enhancement strategy, and has the advantages of high precision, high efficiency, low cost and wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0020] Figure 1 It is a schematic diagram of the method principle of the present invention.

[0021] Figure 2 It is a schematic diagram of the prototype enhancement model of the present invention.

[0022] Figure 3 (a) is a display diagram of 25-24-7 (qs), i.e., abnormal microdeletion of the Y chromosome; Figure 3 (b) is a display diagram of 25-24-2 (qh-), i.e., abnormal reduction of heterochromatin region in the long arm of the Y chromosome; Figure 3 (c) is a display diagram of 25-24-1 (qh+), i.e., abnormal increase of heterochromatin region in the long arm of the Y chromosome; and Figure 3 (d) is a display diagram of 25-24-3 (inv), i.e., abnormal inversion of the Y chromosome.

[0023] Figure 4 This is an introduction diagram of the ISIC 2018 dataset used in the embodiments of the present invention.

[0024] Figure 5 This is an introduction diagram of the APTOS 2019 dataset used in the embodiments of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The embodiment of the present invention discloses a medical image classification method based on contrastive learning, comprising: Step 1, divide the long-tail medical image dataset into a training set and a test set in a ratio of 7:3, perform weak data enhancement and strong data enhancement on the training set images in a batch processing form according to a preset scheme, and obtain weak data enhanced images and strong data enhanced images; wherein, weak data enhancement includes: cropping and resizing, horizontal flipping, and color jittering; strong data enhancement includes: cropping and resizing, horizontal flipping, vertical flipping, color jittering, and grayscale; The long-tail medical image dataset is briefly explained as follows. The long-tail medical image classification problem is usually caused by differences in incidence and difficulties in data collection. Some diseases are common, while others are rare, which makes the collected dataset unbalanced, with more samples in some categories and fewer samples in some categories. When the samples are arranged from high to low according to the number of classes, a dataset similar to the long-tail distribution is obtained.

[0027] Step 1.1: Perform weak data enhancement on the training set images, crop and resize the images to 224x224 pixels; randomly sample 80% of the images and randomly apply a color jitter transformation, with a hue change probability of 0.4, a saturation change probability of 0.4, a brightness change probability of 0.4, and a contrast change probability of 0.1; randomly flip the images horizontally to obtain weak data enhanced images; Step 1.2: Perform strong data enhancement on the training set images, crop and resize the images to 224x224 pixels, and the size of the cropped area is between 80% and 100% of the original image; randomly sample 50% of the images and perform random horizontal flipping; randomly sample 50% of the images and perform random vertical flipping; randomly rotate the images, and the rotation angle can be 0°, 90°, 180° or 270°; randomly sample 80% of the images and randomly apply a color jitter transformation, with a hue change probability of 0.4, a saturation change probability of 0.4, a brightness change probability of 0.4, and a contrast change probability of 0.1; randomly sample 20% of the images and convert them to grayscale images to obtain strongly data enhanced images.

[0028] Step 2: Perform a comparative learning task on the weak data enhanced image obtained in step 1.1 and the strong data enhanced image obtained in step 1.2 through a deep convolutional neural network, learn network parameters, and obtain an image classification model based on a deep convolutional neural network after parameter optimization; specifically, the following steps are included: The deep convolutional neural network consists of a backbone network, a mapping network and a prototype enhancement model. The backbone network includes a feature extraction layer and a classification layer. The feature extraction layer adopts a residual neural network (ResNet), and the classification layer consists of a fully connected layer and a Softmax layer. The mapping network is a two-layer perceptron (MLP). First, the weakly data enhanced image and the strongly data enhanced image obtained in step 1 are input into the feature extraction layer of the backbone network for feature extraction and then input into the mapping network to obtain image embedding. At the same time, the features extracted by the feature extraction layer are input into the classification layer to obtain the basic cross entropy loss. Secondly, the image embedding obtained after the mapping network is input into the prototype enhancement model, and the prototype enhancement contrast learning strategy is used to generate a learnable class prototype for data enhancement. Finally, a balanced implicit enhancement contrast learning loss is obtained.

[0029] Step 2.1: Input the weak data augmented image and the strong data augmented image obtained in step 1 into the feature extraction layer of the deep convolutional neural network respectively to obtain features with a dimension of (2B, 2048), including the first i Weak data enhancement image features of samples Strong data augmented image features of samples , B is the number of batch training samples; Step 2.2: Enhance the image features with the weak data obtained in step 2.1 Input to the classification layer of the deep convolutional neural network to get the prediction (B, C), where C is the number of categories in the training set in step 1. The cross entropy loss is calculated using the cross entropy loss function; the calculation formula is as follows: ; in, is a weak data enhanced image feature, and It is the classification layer (fully connected layer) and The weight of a node, and It is the classification layer (fully connected layer) The deviation between the jth and jth nodes, N represents the number of training samples, C is the number of categories in the training set; Step 2.3: Enhance the image features with the weak data obtained in step 2.1 and strong data-enhanced image features Input mapping network, get (2B, 128) dimensional image embedding for contrastive learning training, specifically divided into weak enhancement image embedding and strongly enhanced image embedding , the obtained image embedding is input into the prototype enhancement model, and the prototype enhancement contrast learning strategy is used to generate learnable class prototypes for data enhancement; like Figure 2 As shown, step 2.4: Construct a batch of learnable class prototype repositories for each class , the number of learnable class prototypes for each class is 10; Step 2.5: According to the number of training samples of different categories in the training set in step 1, calculate the number of learnable class prototypes that can be used in the batch for each class: ; in represents the number of training samples of the cth class, is the number of samples in the category with the most samples in the training set; Step 2.6: Model the semantic enhancement direction of the image embedding obtained in step 2.3 of the same category as a multidimensional Gaussian distribution , Indicates The covariance matrix of the class; Step 2.7: Repository of learnable prototypes obtained in step 2.4 Embedded with images Aggregate to get a collection , where the image embedding v is the weakly enhanced image embedding by aggregating mini-batches and strongly enhanced image embedding Obtained, b is the number of samples in the batch. Each element in and its corresponding label ,gather can be divided into two subsets and For each image embedding , find the semantically similar Negative class embedding composition , the semantic similarity is calculated by the following formula: ; Select the other class learnable prototype that is most similar to the image embedding, and then linearly interpolate the image embedding with the similar learnable class prototype to obtain the enhanced image embedding: ; in, is the random interpolation coefficient embedded in each sample, t is the upper bound value of 0.0001, is an image embedding A collection of augmented image embeddings.

[0030] Step 2.8: Based on the number of learnable class prototypes available for each class in the batch obtained in step 2.5, the learnable class prototype repository obtained in step 2.4 In the above, class-aware sampling is used to select the learnable class prototypes of the corresponding number of classes. For learnable prototypes Apply the multidimensional Gaussian distribution based on step 2.6 The class-aware Gaussian perturbation constructed in the semantic enhancement direction of the modeling is used to obtain the enhanced class prototype ;in as well as Is a class The vector of square roots of the diagonal elements of the covariance matrix of . The resulting set of enhanced prototypes is .

[0031] Step 2.9: Embed the image according to step 2.3 , step 2.7 gets the enhanced image embedding , the learnable prototype obtained in step 2.8 And enhance the prototype According to the 0th dimension, we can obtain a set ; Step 2.10: Based on the set Q obtained in step 2.9, the covariance matrix of the Gaussian distribution is estimated using the exponentially weighted moving average method (EMA method) in the form of batch processing during training: ; in, and , Indicates the current Q The sample embeddings of the class and the total number of learnable agents, It is Sample embeddings of the class and the total number of learnable agents; Step 2.11: Image embedding obtained from step 2.3 , the learnable prototype obtained in step 2.8 And enhance the prototype According to the 0th dimension, we can obtain a set ; Get the set for balanced contrastive learning; Step 2.12: For the set obtained in step 2.11 S The balanced implicit enhancement contrastive learning loss is calculated according to the following formula: ; in, is the adjustment factor, where , Representing a collection S Medium Category c The sample embedding and the set of learnable prototypes, B is the number of samples in a batch, For a batch category c The number of samples, is the number of samples of the cth category in the training set; Step 2.13: Sum the cross entropy loss obtained in step 2.2 and the balanced implicit enhanced contrastive learning loss obtained in step 2.12 to get the overall loss : ; in, and Used to adjust the weights of different losses, ; Step 2.14: Minimize the overall loss obtained in step 2.13 , adjust the parameters of the deep convolutional neural network, and use the back propagation algorithm to adjust the global parameters until the loss no longer decreases, the model converges, and the deep convolutional neural network with optimized parameters is obtained, and then the image classification model based on the deep convolutional neural network with optimized parameters is obtained.

[0032] The present invention enables the model to learn stable features through prototype enhancement and contrast learning methods.

[0033] Step 3: Use the deep convolutional neural network-based image classification model with optimized parameters obtained in step 2 to classify the long-tail medical images in the test set in step 1.

[0034] The long-tail medical images in the test set in step 1 are unified to the same resolution, and then the long-tail medical images unified to the same resolution are input into the deep convolutional neural network with optimized parameters obtained in step 2 to extract image features. Finally, the extracted image features are input into the classification layer to obtain the classification results.

[0035] like Figure 1 As shown in the above steps, the feature extraction layer of the deep convolutional neural network backbone network uses a residual neural network (ResNet-50) as the network structure; i Weak data enhancement image features of samples and i Strong data augmentation image features for samples ; The mapping network from image features to image embedding is MLP, i.e. two-layer perceptron; weakly enhanced image embedding and strongly enhanced image embedding ; Prototype enhancement model.

[0036] The long-tail medical image classification method based on prototype enhanced contrastive learning proposed in the present invention combines image enhancement, convolutional neural network, prototype generation and contrastive learning technologies, significantly improves the classification efficiency and accuracy of abnormal long-tail medical images, and provides a new solution for intelligent and automated screening of abnormal diseases, which has important clinical application value.

[0037] The present invention can be widely used in the fields of automated chromosome abnormality screening, medical image analysis, genomics research, etc., and has significant advantages in improving the efficiency and accuracy of chromosome abnormality detection.

[0038] Experimental analysis 1. Dataset introduction: The present invention conducts experiments on two public datasets: the ISIC 2018 dataset provided by the International Skin Imaging Collaboration (ISIC) and the APTOS 2019 dataset provided by the APTOS 2019 blind detection competition.

[0039] The ISIC 2018 dataset is released by the International Skin Imaging Collaboration (ISIC) and is mainly used for image classification and segmentation of skin lesions. The dataset contains a large number of high-resolution skin lesion images with professional annotations, including classification labels of lesions and segmentation masks of lesions. These images cover a variety of skin lesion types, such as benign and malignant melanomas, basal cell carcinomas, keratotic lesions, etc. The ISIC 2018 dataset provides a rich training resource for the automated diagnosis of skin lesions. Due to its image quality and diversity, it has become an important benchmark for skin lesion detection tasks. The number of samples in each category in the dataset is as follows: Figure 4 shown.

[0040] The APTOS 2019 dataset was released by the Asia Pacific Tele-Ophthalmology Society (APTOS) and is mainly used for the grading task of diabetic retinopathy (DR). The dataset contains retinal color film images annotated with different lesion grading labels, which are divided into five levels: normal, mild, moderate, severe, and proliferative diabetic retinopathy. The images in the APTOS 2019 dataset were collected from actual screening scenes. The image quality varies and contains realistic factors such as halos and noise, which increases the challenge of the grading task. This dataset is widely used to evaluate and optimize diabetic retinopathy detection algorithms, especially in the training and verification of deep learning models. The number of samples in each category in the dataset is as follows: Figure 5 shown.

[0041] APTOS 2019 contains 3662 labeled images that classify the severity of diabetic retinopathy into 5 categories, from no DR to proliferative DR, and ISIC2018 contains 10015 images of pigmented skin lesions in 7 categories. The two datasets considered show an important problem related to the class imbalance problem, that is, the samples belonging to the most dominant class account for half of the entire dataset. In order to make a fair comparison, the present invention randomly divides each dataset into 70% training data and 30% test data based on previous studies.

[0042] At the same time, in order to verify the method proposed in the present invention, 1148 microscopic metaphase images were provided by the First Affiliated Hospital of Xi'an Jiaotong University in Xi'an, China. All chromosomes in the metaphase images were manually annotated by cytologists. The mask of each metaphase image was manually marked by the laboratory researchers of the present invention and then verified by the cytologist. At the same time, 17,808 single chromosomes were collected from 1148 metaphase chromosome images, and the partial data display diagram of the ACR-v1 long-tail data set used in the embodiment of the present invention is shown below, wherein Figure 3 (a) is 25-24-7 (qs), i.e., the abnormal display diagram of Y chromosome microdeletion, Figure 3 (b) is 25-24-2 (qh-), i.e., the abnormal display diagram of the reduction of the heterochromatin region of the long arm of the Y chromosome, Figure 3 (c) is 25-24-1 (qh+), i.e., the abnormal display diagram of the increase of the heterochromatin region of the long arm of the Y chromosome, and Figure 3 (d) is 25-24-3 (inv), i.e., the abnormal display diagram of the inversion of the Y chromosome. Based on the acquired chromatid images and label information, the present invention constructs the ACR-v1 long-tail data set. Table 1 shows the details of the dataset.

[0043] Table 1 ACR-v1 long-tail dataset category 25-24-7 (qs) 25-24-2 (qh-) 25-24-1(qh+) 25-24-3 (inv) Train 347 105 67 13 Test 150 45 30 6 2. Experimental Introduction In order to evaluate the performance of IPECL of the present invention, the method of the present invention is compared with more than 10 advanced methods. Among them, Focal, LDAM-DRW, Logit Adjusted and MWNL are reweighted loss methods. BBN is a method based on the rebalancing training strategy, while Hybrid-SC, SCL, BCL, TSC and the method of the present invention IPECL are CL-based methods. In addition, MWNL and SCL have been verified to perform well in medical long-tail image classification tasks. In order to ensure fairness, the present invention retrains all methods and reruns their published codes on the dataset divided by the present invention under the same experimental settings.

[0044] The present invention also confirms that all models converge and select the best checkpoint. The results are shown in Tables 2, 3, and 4. It can be seen that IPECL has a significant advantage and reaches the highest level in most indicators on both datasets. It is worth noting that the IPECL of the present invention has a significant improvement over other imbalanced methods. For example, F1 on ISIC 2018 is 1.8% higher than the TSC method, and Acc on APTOS-2019 is 1.1% higher than the ProCo method.

[0045] On the ACR-v1 dataset proposed in the present invention, the IPECL method proposed in the present invention achieves the best results in terms of overall accuracy and accuracy of each category compared with existing methods, with an overall accuracy improvement of 1.3%, and the category accuracy of categories 0, 1, and 2 is stable and close to 100%. It is also greatly improved in category 3 where existing methods perform poorly. It is worth mentioning that, unlike most previous methods that reduce the performance of the head class, the IPECL method of the present invention further improves the performance of the head class while improving the performance of the tail class.

[0046] Table 2 Experimental results of ACR-v1 long-tail dataset

[0047] Table 3 Experimental results of ISIC 2018 dataset

[0048] Table 4 Experimental results on the APTOS 2019 dataset

[0049] The comparison results of the IPECL proposed in the present invention and other existing methods on ISIC 2018 and APTOS 2019 are shown in Tables 3 and 4. As can be seen from the table, IPECL always outperforms other methods. In addition, IPECL achieved better performance on the ISIC2018 dataset with a large imbalance factor, and improved by 5.3% on F1 compared with the Focal method. This result is mainly attributed to the fact that when the imbalance problem is more serious, the traditional supervised contrast loss will lead to more serious representation learning bias. Focal uses a set of parameter centers in supervised contrastive learning, which are given greater weights. However, the prototypes used in IPECL supplement the samples of each class to ensure that all classes appear in each mini-batch. Compared with Focal, IPECL achieved a good overall accuracy of 84.8%, with significant accuracy improvements on the head and minority classes. This shows that IPECL has significant improvements in most categories from minority to majority.

[0050] A medical image classification system based on contrastive learning, comprising: The data set acquisition and processing module divides the acquired long-tail medical image data set into a training set and a test set, and performs weak data enhancement and strong data enhancement on the training set images in a batch processing form according to a preset scheme to obtain weak data enhanced images and strong data enhanced images; The image classification model establishment and training module compares the obtained weak data enhancement images and strong data enhancement images through the image classification model based on deep convolutional neural network to learn the network parameters and obtain the image classification model based on deep convolutional neural network after parameter optimization. The image classification model testing module uses the image classification model based on deep convolutional neural network with optimized parameters to classify the long-tail medical images in the test set.

[0051] A computer storage medium stores a computer program, which can implement the steps of a medical image classification method based on contrast learning when executed by a processor.

[0052] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0053] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A medical image classification method based on contrastive learning, characterized in that: include: Step 1: Divide the collected long-tail medical image dataset into a training set and a test set, and perform weak data enhancement and strong data enhancement on the training set images in a batch processing form according to a preset scheme to obtain weak data enhanced images and strong data enhanced images; Step 2: Perform a comparative learning task on the weak data enhanced image and the strong data enhanced image obtained in step 1 through an image classification model based on a deep convolutional neural network, learn network parameters, and obtain an image classification model based on a deep convolutional neural network with optimized parameters; Step 3: Use the deep convolutional neural network-based image classification model with optimized parameters obtained in step 2 to classify the long-tail medical images in the test set in step 1.

2. The medical image classification method based on contrastive learning according to claim 1, characterized in that: The weak data enhancement includes: cropping and resizing, horizontal flipping, and color dithering; the strong data enhancement includes: cropping and resizing, horizontal flipping, vertical flipping, color dithering, and grayscale.

3. The medical image classification method based on contrastive learning according to claim 1, characterized in that: The image classification model based on deep convolutional neural network consists of a backbone network, a mapping network and a prototype enhancement model. The backbone network includes a feature extraction layer and a classification layer, wherein the feature extraction layer adopts a residual neural network, and the classification layer consists of a fully connected layer and a Softmax layer; the mapping network is a two-layer perceptron.

4. The medical image classification method based on contrastive learning according to claim 3, characterized in that: The training process of the image classification model based on deep convolutional neural network is as follows: first, the weak data enhanced image and the strong data enhanced image obtained in step 1 are input into the feature extraction layer of the backbone network for feature extraction and then input into the mapping network to obtain image embedding. At the same time, the features extracted by the feature extraction layer are input into the classification layer to obtain the basic cross entropy loss; secondly, the image embedding obtained after the mapping network is input into the prototype enhancement model, and the prototype enhancement contrast learning strategy is used to generate a learnable class prototype for data enhancement, and finally, a balanced implicit enhancement contrast learning loss is obtained.

5. The medical image classification method based on contrastive learning according to claim 1, characterized in that: The step 2 comprises the following steps: Step 2.1: Input the weak data augmented image and the strong data augmented image obtained in step 1 into the feature extraction layer of the image classification model based on the deep convolutional neural network, and obtain a feature extraction layer with a dimension of (2 B， 2048) dimension features, including the i The weak data enhancement image features of samples and the i Strong data-enhanced image features of samples, B is the number of batch training samples; Step 2.2: Input the weak data-enhanced image features obtained in step 2.1 into the classification layer of the image classification model based on the deep convolutional neural network to obtain the prediction ( B , C ), C is the number of categories in the training set in step 1, and the cross entropy loss is calculated using the cross entropy loss function; Step 2.3: Input the weak data enhancement image features and strong data enhancement image features obtained in step 2.1 into the mapping network to obtain (2B, 128)-dimensional image embeddings for contrastive learning training, which are specifically divided into weak enhancement image embeddings and strong enhancement image embeddings. The obtained image embeddings are input into the prototype enhancement model; Step 2.4: Construct a batch of learnable class prototype repositories for each class. The number of learnable class prototypes for each class is m ; Step 2.5: Calculate the number of learnable class prototypes that can be used in the batch for each class based on the number of training samples of different categories in the training set in step 1; Step 2.6: Model the semantic enhancement direction of the image embedding obtained in step 2.3 of the same category as a multi-dimensional Gaussian distribution; Step 2.7: Calculate the similarity between the image embedding and other types of learnable class prototypes in the learnable class prototype repository obtained in step 2.4 by cosine distance, select other types of learnable class prototypes that are most similar to the image embedding, and then perform linear interpolation between the image embedding and the similar learnable class prototypes to obtain the optimized learnable class prototype; Step 2.8: According to the number of learnable class prototypes available for each class in the batch obtained in step 2.5, randomly select the learnable class prototypes of the corresponding number of classes in the learnable class prototype repository obtained in step 2.4; apply the class-aware Gaussian perturbation constructed based on the semantic enhancement direction of multi-dimensional Gaussian distribution modeling in step 2.6 to the learnable class prototypes to obtain the enhanced class prototypes; Step 2.9: Concatenate the optimized learnable prototype obtained in step 2.7, the enhanced prototype obtained in step 2.8, and the image embedding obtained in step 2.3 according to the 0th dimension to obtain a set Q; Step 2.10: Based on the set Q obtained in step 2.9, the covariance matrix of the Gaussian distribution is estimated using the exponentially weighted moving average method in the form of batch processing during training; Step 2.11: randomly select the corresponding number of learnable prototypes from the learnable prototype repository obtained in step 2.4, the enhanced prototype obtained in step 2.8, and the image embedding obtained in step 2.3, and concatenate them according to the 0th dimension to obtain a set S for balanced contrastive learning; Step 2.12: Calculate the balanced implicit enhanced contrastive learning loss for the set S obtained in step 2.11; Step 2.13: Sum the cross entropy loss obtained in step 2.2 and the balanced implicit enhanced contrastive learning loss obtained in step 2.12 to get the overall loss; Step 2.14: By minimizing the overall loss obtained in step 2.13, adjust the parameters of the deep convolutional neural network and use the back propagation algorithm to adjust the global parameters until the loss no longer decreases, the model converges, and the deep convolutional neural network with optimized parameters is obtained.

6. The medical image classification method based on contrastive learning according to claim 1, characterized in that: The specific method of step 3 is: The long-tail medical images in the test set in step 1 are unified to the same resolution, and then the long-tail medical images unified to the same resolution are input into the feature extraction layer of the image classification model based on the deep convolutional neural network after parameter optimization obtained in step 2 to extract image features. Finally, the extracted image features are input into the classification layer of the image classification model based on the deep convolutional neural network to obtain the classification results.

7. A medical image classification system based on contrastive learning, characterized in that: A medical image classification method based on contrastive learning according to any one of claims 1 to 6 is applied, comprising: The data set acquisition and processing module divides the acquired long-tail medical image data set into a training set and a test set, and performs weak data enhancement and strong data enhancement on the training set images in a batch processing form according to a preset scheme to obtain weak data enhanced images and strong data enhanced images; The image classification model establishment and training module compares the obtained weak data enhancement images and strong data enhancement images through the image classification model based on deep convolutional neural network to learn the network parameters and obtain the image classification model based on deep convolutional neural network after parameter optimization. The image classification model testing module uses the image classification model based on deep convolutional neural network with optimized parameters to classify the long-tail medical images in the test set.

8. A computer storage medium, characterized in that The computer storage medium stores a computer program, and when the computer program is executed by a processor, it can implement the steps of a medical image classification method based on contrastive learning as described in any one of claims 1 to 6.

Images