Image classification method and system based on resampling and contrast learning

Abstract

The invention relates to the technical field of image classification, and provides an image classification method and system based on resampling and comparative learning, the method is realized based on an image classification model, and the model comprises a backbone network, a feature extractor and an output unit; a backbone network is arranged in the backbone network, and the backbone network is used for receiving an image data set and converting each image in the image data set into an image feature; the feature extractor is used for converting image features output by the backbone network into normalized features; the output unit is used for clustering the normalized features by using a clustering algorithm to obtain a classification result of the image data set; by adopting the model and the method, the classification accuracy of the image data set can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of image classification, and in particular to an image classification method and system based on resampling and contrastive learning. Background Art

[0002] The generalized category discovery problem is a subproblem of semi-supervised classification (using unlabeled data to improve classification performance). This research relaxes the constraints of semi-supervised learning, assuming that the unlabeled dataset contains samples of categories different from those in the labeled dataset. During the test phase, it is necessary to accurately classify known categories while effectively assigning samples of unknown categories to new clusters.

[0003] Existing technologies can learn classification through traditional semi-supervised classification algorithms. The first existing technology adopts a fixed threshold to screen high-confidence pseudo-labels for supervised learning. However, this method filters too many samples in the early stage, resulting in low utilization of unlabeled data and slow model convergence. In addition, the imbalanced distribution problem and the situation where unknown categories appear in unlabeled data and test sets are not considered. The simple use of semi-supervised learning methods will make it difficult to discover new class information. The second existing technology proposes to deal with the imbalanced distribution problem by using a dynamic combination of linear and semantic pseudo-labels based on the current estimated category distribution of unlabeled data, showing strong generalization ability for minority classes. However, this assumption is often violated in real-world applications. For example, labeled and unlabeled data are collected from different tasks. Unlabeled data may have a large category distribution gap with labeled data, and using incorrect class distribution assumptions may seriously deteriorate performance.

[0004] The third existing technology uses a unified objective function for known classes and new classes. It adopts a self-labeling strategy based on optimal transfer to assign cluster labels to new class samples, and then trains the network with the generated pseudo labels. At the same time, it proposes that coupled training of known classes and new classes helps to improve the classification performance of both. Due to its simplicity, high performance and strong scalability, this method has become a benchmark framework for many subsequent methods. The new class discovery problem explores methods to transfer knowledge from labeled data to unlabeled data, which is also what the generalized category discovery problem needs to solve. However, in the setting of the new class discovery problem, the categories of unlabeled data do not overlap with the labeled data. For data collected in the real world, it is difficult to guarantee that there are no known categories in the unlabeled data. This assumption limits the use of new class discovery methods.

[0005] Furthermore, due to the long-tail nature of visual classes, we are more likely to encounter known classes in the wild. For example, categories like cats and dogs, with their readily available labels, dominate the top classes in the distribution, while samples from categories like lions and pandas are more difficult to obtain, resulting in an unbalanced distribution of classes in the dataset. Unbalanced datasets can lead to severe performance degradation and model bias, a problem that is even more challenging in generalized category discovery because prior dataset distributions are unavailable in open-world scenarios.

[0006] To overcome the above problems, the fourth existing technology introduces a setting that is closer to the real-world scenario for the first time, solving the generalized category discovery problem under the conditions of class imbalance and unknown distribution. This problem is based on the generalized category discovery problem, and considers the assumption that there is an imbalanced distribution of labeled data and unlabeled data, and the distribution of unlabeled data is unknown. This method performs pseudo-label debiasing based on the estimated prior distribution and sets a fixed sampling rate to increase comparative learning samples to enhance feature learning and alleviate imbalance. However, this method involves the selection of sampling rate hyperparameters, which requires cumbersome parameter adjustments. At the same time, for data with new classes and imbalance, this sampling method cannot fully obtain high-quality samples. Summary of the Invention

[0007] In order to overcome the imbalance problem in training data and insufficient sample sampling in the above-mentioned prior art, the present invention provides an image classification method and system based on resampling and contrastive learning, which can improve the classification accuracy of image datasets.

[0008] In order to solve the above technical problems, the technical solutions of the present invention are as follows: An image classification model, comprising: a backbone network, a feature extractor, and an output unit; The backbone network is used to receive an image dataset and convert each image in the image dataset into an image feature; The feature extractor is used to convert the image features output by the backbone network into normalized features; The output unit is used to cluster the normalized features using a clustering algorithm to obtain a classification result of the image data set; The image classification model is a trained model. When the image classification model is trained, the image classification model further includes: a classifier, a classification loss, and a contrastive learning loss; The classifier is used to convert the image features output by the backbone network into pseudo labels corresponding to the image features; the classification loss is used to enable the classifier to output the optimal pseudo labels; the contrastive learning loss is designed based on normalized features and resampling methods under the guidance of the similarity between the pseudo labels output by the classifier and the pseudo labels, and is used to enable the feature extractor to output the optimal normalized features.

[0009] The present invention also proposes an image classification method based on resampling and contrastive learning, which is implemented based on the above-mentioned image classification model and includes the following steps: Obtain the image dataset to be classified; Inputting the image dataset to be classified into the image classification model; The image classification model outputs an image classification result corresponding to each image in the image dataset to be classified.

[0010] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: During the training process, this application introduces the design of contrastive learning loss guided by the similarity of pseudo-labels output by the classifier, and uses the classification loss to train the classifier so that the classifier outputs the optimal pseudo-label; uses contrastive learning to train the feature extractor so that the feature extractor outputs the optimal, that is, the normalized features that can best distinguish the image category, and the contrastive learning loss is designed based on the resampling method, which can solve the imbalance problem in the training data and the problem of insufficient sample sampling, thereby improving the classification accuracy of the image data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a schematic diagram of the structure of the image classification model proposed in Example 1; Figure 2 This is a schematic diagram of the first flow chart of the image classification method based on resampling and contrastive learning proposed in Example 1; Figure 3 This is a schematic diagram of a second flow chart of the image classification method based on resampling and contrastive learning proposed in Example 1; Figure 4 This is a schematic diagram of the third flow chart of the image classification method based on resampling and contrastive learning proposed in Example 2. DETAILED DESCRIPTION

[0012] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting the present application; In order to better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product size; It is understandable to those skilled in the art that some well-known structures and descriptions thereof may be omitted in the drawings.

[0013] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0014] Example 1 This embodiment proposes an image classification model. Figure 1 This is a schematic diagram of the structure of the image classification model proposed in this embodiment.

[0015] like Figure 1 As shown, the image classification model of this embodiment includes: a backbone network, a feature extractor and an output unit; The backbone network is used to receive an image dataset and convert each image in the image dataset into an image feature; The feature extractor is used to convert the image features output by the backbone network into normalized features; The output unit is used to cluster the normalized features using a clustering algorithm to obtain a classification result of the image data set.

[0016] In the specific implementation process, the present application introduces the similarity of the pseudo-labels output by the classifier to guide the design of contrastive learning loss during the training process, and uses the classification loss to train the classifier so that the classifier outputs the optimal pseudo-label; uses contrastive learning to train the feature extractor so that the feature extractor outputs the optimal, that is, the normalized features that can best distinguish the image category, and the contrastive learning loss is designed based on the resampling method, which can solve the imbalance problem in the training data and the problem of insufficient sample sampling, thereby improving the classification accuracy of the image data set.

[0017] In an optional embodiment, the backbone network is a transformer network, and the feature extractor includes at least a projection head.

[0018] As an exemplary illustration, the clustering algorithm is a k-means clustering algorithm.

[0019] This embodiment also proposes an image classification method based on resampling and contrastive learning. Figure 2 and Figure 3 Schematic diagrams of the first and second flow charts of the image classification method based on resampling and contrastive learning proposed in this embodiment, as shown in Figure 2 and Figure 3 As shown, the following steps are included: S1: Obtain the image dataset to be classified; S2: Inputting the image dataset to be classified into the image classification model; S3: The image classification model outputs an image classification result corresponding to each image in the image dataset to be classified.

[0020] In an optional embodiment, the image classification model is a trained image classification model. When the image classification model is trained, the image classification model further includes: a classifier, the classifier being configured to convert image features output by the backbone network into pseudo labels corresponding to the image features; The steps to train an image classification model include: Get the image dataset for training; Input the training image dataset into the image classification model, perform a preset number of iterative updates on the image classification model, and obtain a trained image classification model; In each iterative update, the classifier of the image classification model outputs pseudo labels, and the feature extractor of the image classification model outputs normalized features; Calculate the global threshold and the adaptive local threshold corresponding to each label; record the normalized features corresponding to each iterative update in the most recent Pseudo-labels, global thresholds, and adaptive local thresholds in iterations; Establish a balanced feature memory library, and selectively add normalized features to the balanced feature memory library during each iterative update process; Calculate the classification loss for the training image dataset; Compute contrastive learning loss on training image datasets based on pseudo-labels, global thresholding, adaptive local thresholding, and balanced feature memory. In a preset number of iterative updates, the classifier and feature extractor are updated alternately; when the classifier is iteratively updated, the self-distillation learning method is used to update the classifier with the goal of minimizing the classification loss, and the classification loss is returned during each update; when the feature extractor is iteratively updated, the feature extractor is updated with the goal of minimizing the contrastive learning loss, and the contrastive learning loss is returned during each update.

[0021] As an exemplary illustration, the classifier includes at least a classification head.

[0022] As an example, by establishing a label memory library, the normalized features corresponding to each iteration are updated in the most recent The pseudo labels, global thresholds, and adaptive local thresholds in the iteration are recorded in the label memory.

[0023] As an exemplary illustration, before training the image classification model, the parameters of the backbone network of the image classification model are initialized to the parameters of the backbone network of the DINO pre-trained model, and all blocks before the last block of the backbone network of the image classification model are frozen.

[0024] In an optional embodiment, the classification loss includes supervised cross entropy loss and pseudo-label guided cross entropy loss; The image dataset used for training includes a number of labeled images and unlabeled images; The supervised cross entropy loss and the pseudo-label guided cross entropy loss are both multi-classification cross entropy losses; The supervised cross entropy loss uses the label of the labeled image as the true label in the multi-classification cross entropy loss formula; The pseudo-label guided cross entropy loss uses the pseudo-label corresponding to the unlabeled image as the true label in the multi-classification cross entropy loss formula.

[0025] In an optional embodiment, the expression of the contrast loss includes:

[0026]

[0027]

[0028]

[0029]

[0030]

[0031] Where, represents the contrast loss, represents the nearest neighbor contrastive learning loss, represents the supervised contrastive learning loss, represents soft contrastive learning loss; 、 and All are weight values; When The value is 1, and when indicator function with a time value of 0; Represents the set of normalized features contained in a batch, express The total number of features in , 、 and Represents the first images , No. images Hedi images The corresponding normalized features, Included Features The nearest neighbor normalization features and characteristics of Normalized features of the enhanced view; Indicates belonging to a relationship; represents a scalar parameter; Represents the division symbol, represents the exponential function; Represents the set of normalized features corresponding to the labeled images in the training image dataset, Represents the union symbol; Represents the set of normalized features corresponding to the unlabeled images in the training image dataset; Represents a batch of images A collection of normalized features with the same label, express The total number of features in , in Represents a set of normalized features corresponding to the images in the training image dataset; express Medium Satisfaction The set of normalized features of express The corresponding pseudo labels The corresponding adaptive local threshold, and Calculated based on the resampling method; Represents the pseudo label probability distribution vector and similarity between and All indicate labels.

[0032] In an optional embodiment, the adaptive threshold The expressions include:

[0033]

[0034]

[0035]

[0036]

[0037]

[0038] In the formula, if the pseudo label is the known label category in the image dataset used for training, , Indicates that the known class is The global threshold of the step, otherwise, , Indicates that the new class The global threshold of the step; Jordi i The pseudo labels corresponding to the normalized features are the known label categories in the image dataset used for training. ,otherwise, ; express The corresponding normalized features are The pseudo-label probability distribution vector of the step; express The corresponding normalized features are The pseudo-label probability distribution vector of the step; Represents the set of label categories obtained by the image classification model for the training image dataset. express The set of known label categories in , express The category set of the new class in ; Represents pseudo labels The corresponding label prediction consistency, Represents the parameters controlling the momentum update; count[ ] represents a function that obtains the frequency of occurrence of the label that appears most times in the array; Representing an image The corresponding pseudo labels, When The corresponding pseudo label is c when the value is 1, and when The indicator function whose value is 0 when the corresponding pseudo label is not c; Represents pseudo labels The total number of corresponding normalized features.

[0039] In an optional embodiment, the balanced feature memory is used to store the normalized features according to the pseudo labels corresponding to the normalized features, and to provide the front-end with the highest similarity value with the normalized features of the image to be supplemented with the positive sample in the balanced feature memory. Normalized features, which are the neighboring normalized features corresponding to the normalized features of the image to be supplemented with positive samples; Among them, the normalized features that have not been added to the balanced feature database are added to the balanced feature memory according to the preset enqueuing probability. When the balanced feature memory reaches the maximum capacity, the normalized features in the balanced feature memory are discarded from the feature memory according to the preset dequeuing probability. The expressions of the enqueue probability and the dequeue probability respectively include:

[0040]

[0041] Where, Indicates that the feature memory library is not added and the pseudo label belongs to The probability of joining the team corresponding to the normalized features of the pseudo labels; Indicates that the pseudo label in the feature memory belongs to The probability of exiting the queue corresponding to the normalized features of the pseudo labels; Indicates that the feature memory belongs to The total number of normalized features of pseudo labels; Represents a hyperparameter greater than 0, and also represents a coefficient that controls the degree of memory balance of the feature memory library. The larger the value, the more balanced the memory of the feature memory.

[0042] In an optional embodiment, when calculating the contrastive learning loss of a training image dataset, the normalized features are clustered using a clustering algorithm to obtain a plurality of clusters; the total number of normalized features corresponding to each cluster is normalized to obtain a normalized frequency number corresponding to each cluster; and based on the normalized frequency number, the pseudo-label probability distribution vector calculated by the classifier is distributed adjusted to obtain an adjusted pseudo-label probability distribution vector; When calculating the contrastive learning loss of the training image dataset, the contrastive learning loss of the training image dataset is calculated based on the adjusted pseudo-label probability distribution vector; The expression of the adjusted pseudo-label probability distribution vector includes:

[0043] Where, Indicates the The adjusted pseudo-label probability distribution vector corresponding to the normalized features, represents the hyperparameter, Indicates that the The normalized frequency number of clusters corresponding to the normalized features.

[0044] This embodiment further proposes an image classification system based on resampling and contrastive learning, which is used to implement the aforementioned image classification method based on resampling and contrastive learning, including: an image classification unit, on which an image classification model is provided, for receiving an image dataset and outputting a classification result corresponding to each image in the image dataset using the image classification model; The model training unit is provided with a label memory and a balanced feature memory, which is used to train the image classification model when the image classification model of the image classification unit needs to be trained, wherein the label memory is used to store the normalized features corresponding to each iteration update in the training process. The pseudo labels, global thresholds and adaptive local thresholds in the iterations are determined; the balanced feature memory bank is used to selectively store normalized features during each iterative update process.

[0045] Example 2 This embodiment provides a specific implementation example based on the image classification model proposed in Example 1 and the image classification method based on resampling and contrastive learning.

[0046] Figure 4 This is a schematic diagram of the third process of the image classification method based on resampling and contrastive learning proposed in this embodiment; Figure 4 As shown, the main purpose of the present invention is to overcome the imbalance problem in training data and the problem of insufficient sample sampling, and to provide a generalized category discovery method based on resampling and contrastive learning. This method introduces a transform pre-training model and proposes to construct a balanced set for nearest neighbor contrastive learning. By utilizing domain information, more supervision information is provided for the minority classes that lack supervision to learn more robust feature representations. In addition, a flexible sampling method is designed, and the output of multiple historical prediction models is used to set a global threshold based on the overall training level of the model. The global threshold is adjusted in a class-specific manner in conjunction with the adaptive local threshold of each class. In addition, the similarity between the pseudo-labels output by the classification branch (classifier) ​​is introduced to guide the training of the contrastive learning branch (feature extractor). Finally, the model output features are used to perform mean clustering to obtain the final classification result.

[0047] In order to achieve the above object, the present invention adopts the following technical solutions: The proposed method constructs a dual-branch framework, where the classification branch adopts a self-distillation learning method, in which the teacher and student models share the same parameters. The classification loss includes the supervised cross-entropy loss for the labeled data in a batch, and the cross-entropy loss guided by the pseudo-labels generated for all data in a batch. In addition, the class prior distribution is estimated by feature clustering. Specifically, k-means clustering is performed on all samples in the training set to obtain the sample number vector n of each cluster, n∈R 1 ×C , C is the number of categories, and the number of samples n is normalized to frequency The clusters with labeled categories are mapped to labeled classes by the Hungarian allocation algorithm. For the remaining unknown clusters, they are sorted by cluster size and assigned to the new classes in order. Finally, this application uses the estimated distribution of the alignment Regularize the predictions of the pseudo-labeled branch and compute the regularization loss.

[0048] During training, a two-layer optimization method is adopted, that is, the feature network is fixed first, the classifier parameters are updated, and then the classifier parameters are fixed and the parameters of the feature network are updated. The two optimizations are performed alternately.

[0049] In order to make full use of the nearest neighbor information and alleviate the imbalanced distribution, balanced set nearest neighbor contrastive learning is proposed, as follows: For unlabeled data, augmented versions of the original images are typically used as positive examples. For labeled data, supervised learning methods leverage other examples from the same category for training, which means that performance on older categories is often better than on newer ones because there are more relevant examples for learning. In imbalanced datasets, both the top and bottom categories face sample scarcity, which can lead to poor model performance on these categories.

[0050] To address the scarcity of data for new and tail categories in imbalanced data distributions, this application proposes leveraging the neighborhood information of each sample across all categories. By exploring the sample's neighborhood, we can increase the model's learning potential and enrich its semantic knowledge. The key to this approach is leveraging relevant information surrounding each sample, rather than simply enhancing the view of the individual sample itself. This allows the model to better understand the relationships between the data, thereby improving the discriminability of features from the head and tail categories in imbalanced datasets.

[0051] For unlabeled data, an enhanced version of the original image is usually used as a positive sample. For labeled data, the supervised learning used can utilize other samples from the same class, so the performance of the old class is usually better than that of the new class. In imbalanced data, the head and tail classes also encounter the same problem. To address the scarcity of positive samples for the new and tail classes under imbalanced distribution, this application proposes to use the neighborhood information of each sample of all categories. Exploring the neighborhood of the sample helps increase the semantic knowledge of the latent space learned by the model.

[0052] Given an image , this method extracts the backbone by using (backbone network) and projection head (Feature Extractor) Get Normalized features To store a subset of these normalized features, we use a The memory bank is used to store features. In order to calculate the image Neighbors of, this method calculates its normalized features and feature library Finally, this method selects the top sample with the highest cosine similarity value. samples as Positive sample supplement.

[0053] However, for unbalanced datasets, the memory bank may contain too few samples from the tail class, leading to errors when searching for nearest neighbors. To maintain a balance between features across categories in the memory bank, an update strategy using enqueue and dequeue operations is designed. The feature memory bank stores both sample features and their corresponding pseudo-labels. The pseudo-labels provide the number of samples in each category in the memory bank, C = [C1, C2, ..., Ck]. To ensure that the feature set contains as many features as possible for each class, different enqueue and dequeue strategies are required for different categories with varying numbers of samples.

[0054] For each training step, the enqueue operation adds the latest features to the memory with different probabilities. Specifically, if a feature pseudo-label indicates that it belongs to the Class category, it will be based on the library The probability of the number of class features is put into the memory bank:

[0055] in, is a hyperparameter greater than 0, is the number of samples of class i in the memory bank. In this way, features of classes with fewer numbers are more likely to be placed in the memory bank. When the memory bank reaches its maximum capacity, the incoming features and their pseudo-labels will need to replace the existing features in the memory bank. In this case, this method uses a dequeue operation to discard a certain number of features and their pseudo-labels. In order to maintain a class-balanced memory bank, this application removes most features with a higher probability and removes a few features with a lower probability, calculated as follows:

[0056] in Is the coefficient that controls the degree of memory balance. A higher value will result in a more balanced feature library.

[0057] For samples , its neighbors are taken as positive samples together with the enhanced version of , thus allowing a comprehensive characterization of its neighborhood. Now, the neighborhood contrastive loss It can be formally written as:

[0058] Among them, a batch contains |B| samples, for which , which is characterized by ,N( ) including samples and its other enhanced view.

[0059] To transfer knowledge from the pseudo-labeled branch to the contrastive learning branch to further enhance representation learning, a contrastive learning sample re-sampling is proposed. Due to the long-tail distribution of the underlying dataset, additional requirements are put forward for knowledge transfer. Through sampling by consistency filtering, the same effect of alleviating the imbalance problem is achieved. The specific way is as follows: Record the prediction results of the samples in the last m steps, denoted as .

[0060]

[0061] Where argmax( ) represents the prediction result of the sample at time t. In order to alleviate the influence of the long-tail distribution, the prediction probability output by the model is adjusted in distribution, denoted as follows:

[0062] Where is a hyperparameter, and the adjusted prediction is also denoted as .

[0063] Recording the history label is equivalent to voting for multiple historical models, and the number of votes determines the credibility of the sample. According to the prediction of the sample by the model in the last m epochs, the sample is classified as a known class or a new class. For each sample, its attribution will change constantly during the training process. Record the label that appears most frequently in the last m historical predictions of each sample, and the frequency of the appearance of this label is denoted as or :

[0064]

[0065] Where count[ ] is a function that obtains the frequency of the label that appears most frequently in the array, and according to the sample label, the sample i is classified as a known class or a new class .

[0066] Then, the global threshold is designed based on the following two principles. First, the global threshold should be related to the credibility of the model for unlabelled data, reflecting the overall learning state. In addition, the global threshold should be steadily increased during training to ensure that samples with incorrect pseudo-labels are discarded. The global threshold Set to the average label occurrence frequency of samples on the unlabeled data, where t represents the t-th epoch. However, due to the large amount of data, it would be time-consuming to calculate the consistency of all unlabeled data at each time step or even each training epoch. Therefore, the global confidence is estimated as the exponential moving average (EMA) of the confidence at each training time step. This application will Initialized to 0, so that the model has a smaller global threshold in the early stage of training, thereby sampling more samples. Global threshold of known classes Define and adjust to:

[0067] in, Represents the number of known class samples, and λ is the parameter that controls the momentum update. In the same way, the global threshold of the new class can be obtained The local threshold is designed to adjust the global threshold in a class-specific manner. In this application, we calculate the model’s predicted consistency frequency for each class c to estimate the learning state of a specific class:

[0068] in, Represents the predicted label of the sample, and N(c) is the number of samples in class c. Combining the global threshold and the local threshold, the final adaptive threshold is obtained. , which is expressed as follows:

[0069] The additional sample set A( ). For sample i, its predicted probability is , if it satisfies max( )> (argmax( ), then the sample is included in the set A( ). The samples in this set are combined with the labeled sample set for supervised contrastive learning training. The soft contrastive learning method is introduced, and the contrast loss is expressed as follows:

[0070] in, = sim( , ) represents the pseudo-label probability distribution of the classification branch output and Note that the weight coefficient here represents the similarity between samples. The purpose is to transfer the pseudo-label information of the classification branch to the contrastive learning branch. For the pseudo-labels of samples in a batch, generate its pseudo-label similarity matrix to represent the pairwise similarity between samples on the label. For the current sample feature, instead of screening out samples that are consistent with its pseudo-label as positive samples, all samples are regarded as positive samples, and the label pairwise similarity coefficient sim( , ) guides training. A higher coefficient indicates that the two samples are more likely to be a positive pair, and thus a higher weight is assigned. In this way, every sample screened can be fully utilized.

[0071] In addition, supervised contrastive learning is still used for labeled data, and the specific implementation is as follows:

[0072] in, Represents a set of samples in a batch with the same label as sample i.

[0073] The final contrast loss is expressed as follows:

[0074] in, It is a nearest neighbor contrastive learning loss that adds features of its nearest neighbors in addition to different augmented views of the sample As a positive sample. It is supervised contrastive learning trained on labeled data. It is a soft contrastive learning loss performed on labeled samples and resampled samples.

[0075] In summary, for the generalized category discovery task in imbalanced distribution scenarios, and to address the problem of lack of supervision for minority classes, this application proposes two sample supplementation strategies for contrastive learning.

[0076] First, we introduce nearest neighbor contrastive learning, which leverages neighborhood information to provide more supervision for the under-supervised minority class, enabling the model to learn more robust feature representations. Furthermore, we control the entry and exit of features by using the class distribution in the memory to build a balanced feature memory and alleviate the problem of data imbalance.

[0077] Second, a flexible adaptive threshold sampling strategy was designed. A global threshold was set based on the model's multiple historical prediction outputs. Combined with adaptive local thresholds for each class, the global threshold was adjusted in a class-specific manner to select high-quality samples for contrastive learning, further improving the distinguishability of the feature space. Furthermore, the pseudo-label similarity of the classification branch's output was introduced to guide the training of the contrastive learning branch.

[0078] Compared with the prior art, the application has the beneficial effects that: Balanced feature memory bank is constructed for nearest neighbor contrastive learning, and more supervised information is provided for the minority class lacking supervision by utilizing the domain information, so that a more robust feature representation is learned.

[0079] A flexible sampling method is designed to obtain more high-quality samples for model training. The outputs of multiple historical prediction models are used to set a global threshold through overall training of the model, and an adaptive local threshold for each class is used to adjust the global threshold in a class-specific manner. In addition, the similarity between the pseudo-labels output by the classification branch is introduced to guide the training of the contrastive learning branch.

[0080] As an example, in the implementation process, on the CIFAR-10-LT, CIFAR-100-LT, and ImageNet-100-LT data sets, when the imbalance rate is 100, the experimental results are as shown in the following table.

[0081]

[0082] From the above table, it can be seen that on Cifar-10-LT and Cifar-100-LT, the classification performance of known classes, unknown classes and all classes is improved; wherein, the full name of ABC is: Auxiliary balanced classifier, the full name of DARP is: Distribution aligning refinery of pseudo-label, the full name of TRSSL is: Towards realistic semi-supervised learning, the full name of ORCA is: Open-world with unCertainty based Adaptive margin, the full name of SimGCD is: Simple Generalized Category Discovery, the full name of GCD is: Generalized category discovery, the full name of OpenCon is: Open-world contrastive learning, the full name of BaCon-S is: Self-Balanced Co-Advice contrastive framework in SimGCD-style, the full name of BaCon-O is: Self-Balanced Co-Advice contrastive framework in ORCA-style, Ours represents the method proposed in the application, and the numbers in the table represent the percentage of accuracy.

[0083] Example 3 This embodiment provides a specific implementation example based on the image classification model proposed in Example 1 and the image classification method based on resampling and contrastive learning.

[0084] The image classification method based on resampling and contrastive learning may include the following steps when implemented: S1: Obtain an image dataset and divide it into a training dataset and a test dataset; S2: Building a classification model based on transform; the feature extraction model includes a backbone network, a classifier, and a feature extractor; S3: Import the DINO pre-trained model (using the PyTorch library), initialize the network, and freeze all blocks before the last block of the backbone network; S4: The training data set is input into the backbone network, the classifier generates a classification probability vector, and the feature extractor generates a feature vector; S5: Cluster the feature vectors generated in S4 to obtain class distribution.

[0085] S6: Record the labels and feature vectors output by S4. A label memory is established to store the model's predicted labels for each sample over the last n epochs. For each class, the average value of label prediction consistency (i.e., the threshold) is stored. This average value (i.e., the global threshold) is stored for both known and unknown classes, while a class-specific average value (i.e., the local threshold) is also stored for each class. A feature memory is established to selectively store features from samples in the current batch.

[0086] S7: Use the pseudo labels generated by the network and the real labels to train the classifier; The following steps are for contrastive learning loss calculation, S8, S9, and S10 are performed in parallel.

[0087] S8: Calculate the supervised contrastive learning loss for the labeled data in the training set, and calculate the unsupervised contrastive learning loss for the unlabeled data in the training set; S9: Calculate the number of occurrences of the label that appears most frequently in the historical predictions of each sample based on the label memory library, filter out samples with high historical prediction consistency, and calculate the soft contrastive learning loss based on the sample prediction confidence.

[0088] S10: Select the nearest neighbor feature of the sample feature in the feature memory and calculate the nearest neighbor contrast learning loss. In the above steps, S6 is updated alternately with S7, S8, and S9.

[0089] S11: Input the test data set into the model, obtain the feature vector, and perform k-means clustering on the features to obtain the classification results.

Claims

1. An image classification model, characterized in that: include: Backbone network, feature extractor and output unit; The backbone network is used to receive an image dataset and convert each image in the image dataset into an image feature; The feature extractor is used to convert the image features output by the backbone network into normalized features; The output unit is used to cluster the normalized features using a clustering algorithm to obtain a classification result of the image data set.

2. The image classification model according to claim 1, characterized in that The backbone network is a transformer network, and the feature extractor includes at least a projection head.

3. An image classification method based on resampling and contrastive learning, implemented based on the image classification model according to claim 1 or 2, characterized in that: The following steps are involved: Get the image dataset to be classified; Inputting the image dataset to be classified into the image classification model; The image classification model outputs an image classification result corresponding to each image in the image dataset to be classified.

4. The image classification method based on resampling and contrastive learning according to claim 3, characterized in that: The image classification model is a trained image classification model. When the image classification model is trained, the image classification model further includes: a classifier, which is used to convert the image features output by the backbone network into pseudo labels corresponding to the image features; The steps to train an image classification model include: Get the image dataset for training; Input the training image dataset into the image classification model, perform a preset number of iterative updates on the image classification model, and obtain a trained image classification model; In each iterative update, the classifier of the image classification model outputs pseudo labels, and the feature extractor of the image classification model outputs normalized features; Calculate the global threshold and the adaptive local threshold corresponding to each label; record the normalized features corresponding to each iterative update in the most recent Pseudo-labels, global thresholds, and adaptive local thresholds in iterations; Establish a balanced feature memory library, and selectively add normalized features to the balanced feature memory library during each iterative update process; Calculate the classification loss for the training image dataset; Compute contrastive learning loss on training image datasets based on pseudo-labels, global thresholding, adaptive local thresholding, and balanced feature memory. In a preset number of iterative updates, the classifier and feature extractor are updated alternately; when the classifier is iteratively updated, the self-distillation learning method is used to update the classifier with the goal of minimizing the classification loss, and the classification loss is returned during each update; when the feature extractor is iteratively updated, the feature extractor is updated with the goal of minimizing the contrastive learning loss, and the contrastive learning loss is returned during each update.

5. The image classification method based on resampling and contrastive learning according to claim 4, characterized in that: The classification loss includes supervised cross entropy loss and pseudo-label guided cross entropy loss; The image dataset used for training includes a number of labeled images and unlabeled images; The supervised cross entropy loss and the pseudo-label guided cross entropy loss are both multi-classification cross entropy losses; The supervised cross entropy loss uses the label of the labeled image as the true label in the multi-classification cross entropy loss formula; The pseudo-label guided cross entropy loss uses the pseudo-label corresponding to the unlabeled image as the true label in the multi-classification cross entropy loss formula.

6. The image classification method based on resampling and contrastive learning according to claim 4, characterized in that: The expression of the contrast loss includes: Where, represents the contrast loss, represents the nearest neighbor contrastive learning loss, represents the supervised contrastive learning loss, represents soft contrastive learning loss; 、 and All are weight values; When The value is 1, and when indicator function with a time value of 0; Represents the set of normalized features contained in a batch, express The total number of features in , 、 and Represents the first images , No. images Hedi images The corresponding normalized features are, Included Features The nearest neighbor normalization features and characteristics of Normalized features of the enhanced view; Indicates belonging to a relationship; represents a scalar parameter; Represents the division symbol, represents the exponential function; Represents the set of normalized features corresponding to the labeled images in the training image dataset, Represents the union symbol; Represents the set of normalized features corresponding to the unlabeled images in the training image dataset; Represents a batch of images A collection of normalized features with the same label, express The total number of features in , in Represents a set of normalized features corresponding to the images of the training image dataset; express Medium Satisfaction The set of normalized features of express The corresponding pseudo labels The corresponding adaptive local threshold, and Calculated based on the resampling method; Represents the pseudo label probability distribution vector and similarity between and All indicate labels.

7. The image classification method based on resampling and contrastive learning according to claim 4, characterized in that: The adaptive local threshold The expressions include: In the formula, if the pseudo label is the known label category in the image dataset used for training, , Indicates that the known class is The global threshold of the step, otherwise, , Indicates that the new class The global threshold of the step; Jordi i The pseudo labels corresponding to the normalized features are the known label categories in the image dataset used for training. ,otherwise, ; express The corresponding normalized features are The pseudo-label probability distribution vector of the step; express The corresponding normalized features are The pseudo-label probability distribution vector of the step; Represents the set of label categories obtained by the image classification model for the training image dataset. express The set of known label categories in , express The category set of the new class in ; Represents pseudo labels The corresponding label prediction consistency, Represents the parameters controlling the momentum update; count[ ] represents a function that obtains the frequency of occurrence of the label that appears most times in the array; Representing an image The corresponding pseudo labels, When The corresponding pseudo label is c when the value is 1, and when The indicator function whose value is 0 when the corresponding pseudo label is not c; Represents pseudo labels The total number of corresponding normalized features.

8. The image classification method based on resampling and contrastive learning according to claim 4, characterized in that: The balanced feature memory is used to store the normalized features according to the pseudo labels corresponding to the normalized features, and to provide the highest similarity value between the normalized features of the image to be supplemented with the positive sample in the balanced feature memory. Normalized features, which are the neighboring normalized features corresponding to the normalized features of the image to be supplemented with positive samples; Among them, the normalized features that have not been added to the balanced feature database are added to the balanced feature memory according to the preset enqueuing probability. When the balanced feature memory reaches the maximum capacity, the normalized features in the balanced feature memory are discarded from the feature memory according to the preset dequeuing probability. The expressions of the enqueue probability and the dequeue probability respectively include: Where, Indicates that the feature memory library is not added and the pseudo label belongs to The probability of joining the team corresponding to the normalized features of the pseudo labels; Indicates that the pseudo label in the feature memory belongs to The probability of exiting the queue corresponding to the normalized features of the pseudo labels; Indicates that the feature memory belongs to The total number of normalized features of pseudo labels; Represents a hyperparameter greater than 0, and also represents a coefficient that controls the degree of memory balance of the feature memory library. The larger the value, the more balanced the memory of the feature memory.

9. The image classification method based on resampling and contrastive learning according to any one of claims 4 to 8, characterized in that: When calculating the contrastive learning loss of the image data set used for training, clustering the normalized features using a clustering algorithm to obtain a plurality of clusters; normalizing the total number of normalized features corresponding to each cluster to obtain a normalized frequency number corresponding to each cluster; and based on the normalized frequency number, performing distribution adjustment on the pseudo-label probability distribution vector calculated by the classifier to obtain an adjusted pseudo-label probability distribution vector; When calculating the contrastive learning loss of the training image dataset, the contrastive learning loss of the training image dataset is calculated based on the adjusted pseudo-label probability distribution vector; The expression of the adjusted pseudo-label probability distribution vector includes: Where, Indicates the The adjusted pseudo-label probability distribution vector corresponding to the normalized features, represents the hyperparameter, Indicates that the The normalized frequency number of clusters corresponding to the normalized features.

10. An image classification system based on resampling and contrastive learning, for implementing the image classification method based on resampling and contrastive learning according to any one of claims 4 to 9, comprising: an image classification unit, on which an image classification model is provided, for receiving an image dataset and outputting a classification result corresponding to each image in the image dataset using the image classification model; The model training unit is provided with a label memory and a balanced feature memory, which is used to train the image classification model when the image classification model of the image classification unit needs to be trained, wherein the label memory is used to store the normalized features corresponding to each iteration update in the training process. The pseudo labels, global thresholds and adaptive local thresholds in the iterations are determined; the balanced feature memory bank is used to selectively store normalized features during each iterative update process.

Images