Clustering-based unsupervised fine-grained image classification model training method and classification method

Abstract

The application discloses a kind of based on clustering unsupervised fine-grained image classification model training method and classification method, training method includes: obtaining fine-grained image data;Extract the feature of fine-grained image and carry out normalization processing;Using predetermined clustering method, the normalized feature is clustered, and according to clustering result, corresponding image data is assigned pseudo label;Using the image data with pseudo label, fine-grained image classification model is trained, and the parameter in model is updated using batch stochastic gradient descent algorithm, and the feature center of each cluster is updated with momentum;Clustering and training process are repeated, and the unsupervised fine-grained image classification model of training completion is obtained.The present application can solve the learning degradation and non-convergence problem existing when the existing unsupervised learning method is applied to fine-grained image classification task, and fill the blank of unsupervised fine-grained image classification without available method.

Description

TECHNICAL FIELD

[0001] The present application relates to a clustering-based unsupervised fine-grained image classification model training method and classification method, and belongs to the technical field of computer vision. BACKGROUND

[0002] Compared with general image classification, fine-grained image classification is an image classification task that needs to distinguish more fine-grained categories. Since the differences between target categories are smaller, and there are various changes within the categories, the fine-grained image classification task is extremely challenging. Most existing methods for solving the fine-grained image classification task are based on supervised label information to guide the learning of an effective model to achieve good generalization ability on unknown data. However, the small differences between categories in the fine-grained image classification task make data labeling difficult, costly, and it is difficult to determine which of the two similar categories a sample belongs to.

[0003] Considering the advantage of unsupervised learning that does not require data labels, in recent years, many researchers have proposed clustering-based unsupervised learning methods for general image classification tasks and individual-level pedestrian re-identification tasks, which use clustering techniques to assign pseudo-labels to unlabeled data, and then use the pseudo-labels to guide learning. The typical representatives are kmeans clustering-based methods and DBSCAN clustering-based methods. These methods can overcome the labeling problem. When performing image classification tasks, the general image classification task generally has fewer categories and more single-class samples, and the clustering effect is good. Although the individual-level pedestrian re-identification task has many categories and fewer single-class samples, the samples in each category are from the same pedestrian individual in the same space-time, and there are very similar appearances and backgrounds, so the clustering result is also good. However, the fine-grained image classification task not only has many categories and fewer single-class samples, but also each category is a different individual captured at different times and spaces, and the intra-class difference is large. It is found through evaluation that the existing clustering-based methods directly applied to the fine-grained image classification task will have problems such as learning degradation and non-convergence. SUMMARY

[0004] The present application aims to overcome the deficiencies in the prior art, and provides a clustering-based unsupervised fine-grained image classification model training method and classification method, which can solve the learning degradation and non-convergence problems of existing unsupervised learning methods applied to fine-grained image classification tasks, and fill the gap of no available methods for unsupervised fine-grained image classification. To achieve the above-mentioned purpose, the present application is implemented by using the following technical solutions:

[0005] In a first aspect, the present application provides a clustering-based unsupervised fine-grained image classification model training method, comprising:

[0006] obtaining fine-grained image data;

[0007] The fine-grained image classification model is trained by using the image data with pseudo labels, parameters in the model are updated by using a batch stochastic gradient descent algorithm, and the feature centers of each cluster are updated by using momentum; wherein the feature center of each cluster is obtained by calculating a weighted mean of the normalized features corresponding to the image data with the same pseudo label.

[0008] The normalized features are clustered by using a predetermined clustering method, and the corresponding image data is assigned a pseudo label according to the clustering result to construct image data with pseudo labels.

[0009] The fine-grained image classification model is trained by using the image data with pseudo labels, parameters in the model are updated by using a batch stochastic gradient descent algorithm, and the feature centers of each cluster are updated by using momentum; wherein the feature center of each cluster is obtained by calculating a weighted mean of the normalized features corresponding to the image data with the same pseudo label.

[0010] The clustering and training process is repeated, and when the clustering result is unchanged for multiple times or reaches a preset maximum number of executions, the process is exited, and an unsupervised fine-grained image classification model and pseudo labels of fine-grained images are obtained.

[0011] In combination with the first aspect, further, the fine-grained image classification model includes a network layer, a fully connected layer, a global mean pooling layer with parameters, and a processing layer, and the unsupervised fine-grained image is sequentially input into the network layer, the fully connected layer, and the global mean pooling layer with parameters for feature extraction, and the extracted features are input into the processing layer for L2 normalization.

[0012] In combination with the first aspect, further, the predetermined clustering method includes a DBSCAN clustering algorithm or an HDBSCAN clustering algorithm.

[0013] In combination with the first aspect, preferably, in the DBSCAN clustering algorithm, the optimal neighbor distance eps = 0.4, and the minimum number of adjacent samples N = 5.

[0014] In combination with the first aspect, preferably, in the HDBSCAN clustering algorithm, the optimal minimum number of clustering samples N = 5.

[0015] In combination with the first aspect, preferably, the DBSCAN clustering algorithm or the HDBSCAN clustering algorithm only assigns pseudo labels to part of the image data.

[0016] In combination with the first aspect, preferably, the image data without pseudo labels does not participate in the training of the fine-grained image classification model.

[0017] In combination with the first aspect, further, the feature center of each cluster is obtained by the following formula:

[0018]

[0019] In formula (1), is the feature center of the kth cluster, N kis the number of samples in the k-th class, is the normalized feature of the i-th sample in the k-th class, is the corresponding weighted weight, which is obtained by the following formula:

[0020]

[0021] In formula (2), is the set of all sample features of the k-th class, exp() represents the exponential function, and distance metric The following three calculation methods can be used to calculate:

[0022] (1) without considering the distance constraint, taking

[0023] (2) by minimum distance calculation: represents the calculated feature to the minimum distance of the feature of all other samples in the class;

[0024] (3) by average distance calculation: represents the calculated feature to the average distance of the feature of all other samples in the class.

[0025] In combination with the first aspect, further, the fine-grained image classification model uses an InfoNCE loss function to guide learning, and the target loss is:

[0026]

[0027] In formula (3), f q is the normalized feature of the q-th sample, c + represents the feature center of the class to which the q-th sample feature belongs, c k represents the feature center of the k-th class, t represents a temperature hyperparameter, d(,) represents the distance of the normalized feature to the class feature center, and the Euclidean distance metric is adopted; when f q is closer to its class feature center c + and farther from other class feature centers, the loss is smaller.

[0028] In combination with the first aspect, preferably, within a single outer loop, all parameters of the fine-grained image classification model are updated by the InfoNCE loss function, and each class feature center is updated by momentum; between two outer loops, each class feature center is reconstructed, and the update strategy of a single outer loop is executed.

[0029] In combination with the first aspect, further, the batch stochastic gradient descent algorithm is used to update the parameters in the model, the Adam optimizer is selected to update the parameters, and the momentum is used to update the feature centers of each cluster, the class feature center is updated by using the mean of all sampling samples in the class, and the following is shown:

[0030]

[0031] In formula (4), Q k is a sample index set of the kth class, f q represents the sample feature belonging to the set Q k , m represents the update momentum, and c k is the class feature center of the kth class.

[0032] In combination with the first aspect, further, the batch stochastic gradient descent algorithm samples a batch of data in each iteration, and the sampling method includes:

[0033] Randomly sampling P clusters;

[0034] Randomly sampling Q samples in each cluster, for a cluster with a sample number greater than or equal to Q, Q samples are randomly selected by using non-replacement sampling, and for a cluster with a sample number less than Q, Q samples are randomly selected by using replacement sampling;

[0035] Finally, a batch of data composed of PQ samples is obtained.

[0036] Secondly, the present application provides an image classification method, which includes:

[0037] Obtaining fine-grained image data;

[0038] Using the trained unsupervised fine-grained image classification model to perform feature extraction on the obtained image data and obtain a label set of the obtained image data, and constructing a feature index tree;

[0039] Using the trained unsupervised fine-grained image classification model to perform feature extraction on the images in the to-be-predicted data set, searching for the nearest neighbor or k-nearest neighbor data in the feature index tree, adopting the label of the nearest neighbor data or voting the labels of the k-nearest neighbor data to make a decision, and obtaining the label of the to-be-predicted image;

[0040] The unsupervised fine-grained image classification model is obtained by using the method of the first aspect.

[0041] Thirdly, the present application provides an unsupervised fine-grained image classification model training system based on clustering, which includes:

[0042] A first obtaining module is configured to obtain fine-grained image data;

[0043] The first feature extraction module is configured to extract features of the fine-grained images and perform normalization processing by using a fine-grained image classification model pre-trained or trained in a training process, to obtain normalized features.

[0044] The cluster generation pseudo-label module is configured to cluster the normalized features by using a predetermined clustering method, assign pseudo-labels to corresponding image data according to clustering results, and construct image data with pseudo-labels.

[0045] The training module is configured to train the fine-grained image classification model by using the image data with pseudo-labels, update parameters in the model by using a batch stochastic gradient descent algorithm, and update feature centers of each cluster by using momentum; wherein the feature centers of each cluster are obtained by calculating weighted mean values of the normalized features corresponding to the image data with the same pseudo-label.

[0046] The model output module is configured to repeat the clustering and training processes, and quit when the clustering results are unchanged for multiple times or reach a preset maximum number of executions, to obtain a trained unsupervised fine-grained image classification model and pseudo-labels of fine-grained images.

[0047] In a fourth aspect, the present application provides an image classification system, comprising:

[0048] The second acquisition module is configured to acquire fine-grained image data.

[0049] The second feature extraction module is configured to extract features of the acquired image data by using the trained unsupervised fine-grained image classification model, and obtain a label set of the acquired image data, to construct a feature index tree.

[0050] The label output module is configured to extract features of images in a to-be-predicted data set by using the trained unsupervised fine-grained image classification model, search for nearest neighbors or k-nearest neighbors in the feature index tree, adopt labels of the nearest neighbors or vote for labels of the k-nearest neighbors to make a decision, and obtain labels of the to-be-predicted images; wherein the unsupervised fine-grained image classification model is trained by using the method of the first aspect.

[0051] Compared with the prior art, the method and the system provided by the present application have the following advantages:

[0052] The training method provided by the application obtains fine-grained image data; a fine-grained image classification model in a pre-training or training process is used to extract features of the fine-grained image and perform normalization processing to obtain normalized features; a predetermined clustering method is used to cluster the normalized features, pseudo labels are assigned to the corresponding image data according to the clustering results, and image data with pseudo labels is constructed; the application uses a density clustering-based method to assign pseudo labels, and the density-based method is more consistent with the manifold distribution of high-dimensional feature data, and has better clustering results;

[0053] The training method provided by the application trains the fine-grained image classification model using the image data with pseudo labels, updates the parameters in the model using a batch stochastic gradient descent algorithm, and updates the feature centers of each cluster with momentum; the application uses the class feature center as the representative of the class and updates it with momentum, which can avoid noise disturbance and make the algorithm converge better, can solve the learning degradation and non-convergence problem when the existing unsupervised learning method is applied to the fine-grained image classification task, and can improve the accuracy and robustness of the fine-grained image classification result;

[0054] The classification method provided by the application obtains fine-grained image data; the unsupervised fine-grained image classification model is used to extract features of the obtained image data and obtain a label set of the obtained image data, and a feature index tree is constructed; the unsupervised fine-grained image classification model is used to extract features of the image in the to-be-predicted data set, search for the nearest neighbor or k nearest neighbor data in the feature index tree, and adopt the label of the nearest neighbor data or vote for the labels of the k nearest neighbor data to obtain the label of the to-be-predicted image; the method can solve the problem of difficult and high-cost fine-grained image labeling, and fills the gap of no available method for unsupervised fine-grained image classification. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is a flowchart of a clustering-based unsupervised fine-grained image classification model training method provided by an embodiment of the application;

[0056] Figure 2 is a flowchart of an image classification method provided by an embodiment of the application. DETAILED DESCRIPTION

[0057] The application will be further described below with reference to the drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.

[0058] Embodiment one:

[0059] As shown in Figure 1 , the application provides a clustering-based unsupervised fine-grained image classification model training method, which comprises:

[0060] obtaining fine-grained image data;

[0061] extracting features of the fine-grained images and performing normalization processing by using a fine-grained image classification model trained in advance or in a training process, to obtain normalized features;

[0062] performing clustering on the normalized features by using a predetermined clustering method, assigning pseudo labels to corresponding image data according to clustering results, and constructing image data with pseudo labels;

[0063] training the fine-grained image classification model by using the image data with pseudo labels, updating parameters in the model by using a batch stochastic gradient descent algorithm, and updating feature centers of each cluster by using momentum; wherein the feature centers of each cluster are obtained by calculating weighted mean of normalized features corresponding to the same image data with pseudo labels;

[0064] repeating the clustering and training processes, and exiting when the clustering results are unchanged for multiple times or reach a preset maximum number of executions, to obtain an unsupervised fine-grained image classification model trained and pseudo labels of fine-grained images.

[0065] The specific steps are as follows:

[0066] Step 1: obtaining fine-grained image data.

[0067] In the corresponding application scenarios, the fine-grained recognition data set collected in this step does not need to be labeled, so the following two feasible methods can be used for collection:

[0068] Method 1: camera shooting collection. For example, in a traffic environment, a snapshot camera is directly used to collect fine-grained images of vehicles; in an animal breeding farm, a monitoring camera is directly used to shoot animal images; in a train station, an airport, etc., a door access system can be used to shoot pedestrian images; in military reconnaissance, unmanned aerial vehicles, remote sensing satellites, etc. are used to shoot military targets; in the biological field, microcell bodies, etc. are directly shot, and similar methods can be used in many other fields.

[0069] Method 2: collection from Internet search. Common targets such as vehicles and animals can be directly downloaded from the Internet search to download different fine-grained categories of data, and the search engine query interface can be used for batch download, and the search engine crawler interface can be used to download data from a special website. Since the data set does not need to be labeled, any vehicle or animal of a certain category can be collected without the need for manual screening.

[0070] After the collection is completed, a general target detection model is used to detect the large category targets in the images to generate a fine-grained image data set.

[0071] Step 2: Extract the features of the fine-grained image using the pre-trained or fine-grained image classification model during the training process and perform normalization processing to obtain normalized features.

[0072] The pre-trained fine-grained image classification model can directly use a pre-trained model on a large-scale dataset such as ImageNet, such as: ResNet model, VGGNet model, DenseNet model, etc.

[0073] The fine-grained image classification model includes network layers, fully connected layers, parameterized global mean pooling layers, and processing layers. The unsupervised fine-grained image is sequentially input into the network layer, the fully connected layer, and the parameterized global mean pooling layer for feature extraction, and the extracted features are input into the processing layer for L2 normalization. The pre-trained model is used to remove the feature tensor output by the fully connected layer, and then a parameterized global mean pooling layer is used to extract the feature vector. Finally, the features are L2 normalized to more easily calculate the feature distance or similarity.

[0074] Specifically, the parameterized global mean pooling layer has the following form:

[0075]

[0076] In formula (1), when p k = 1, it corresponds to global mean pooling; when p k → ∞, it corresponds to global maximum pooling. This layer has a greater impact on feature extraction and network parameter learning, and is a compromise operation between global mean pooling and global maximum pooling. The parameter p k can be learned according to the objective function to produce the best features.

[0077] Step 3: Cluster the normalized features using a predetermined clustering method, assign pseudo labels to the corresponding image data according to the clustering results, and construct image data with pseudo labels.

[0078] The predetermined clustering method includes using the DBSCAN clustering algorithm or the HDBSCAN clustering algorithm to classify the image data according to the normalized features, and assigning pseudo labels to the corresponding image data according to the clustering results.

[0079] The predetermined clustering method mainly adopts a density-based method, and typical representatives are DBSCAN or HDBSCAN. HDBSCAN can generally obtain relatively good performance compared with DBSCAN, because HDBSCAN can effectively segment adjacent classes, while DBSCAN will cluster adjacent classes together. From the classification target, the cost of splitting a class into multiple classes is smaller than the cost of merging multiple classes into the same class. At the same time, HDBSCAN has fewer hyperparameters. There are two parameters in DBSCAN, the neighbor distance eps and the minimum number of neighbors N, and a preferred setting is eps = 0.4 and N = 5; there is only one parameter in HDBSCAN, the minimum number of cluster samples N, and a preferred setting is N = 5.

[0080] The DBSCAN clustering algorithm or the HDBSCAN clustering algorithm only assigns pseudo labels to part of the image data. This way of clustering only part of the data and assigning pseudo labels can effectively avoid the influence of noise data on subsequent model training, because the data without assigned pseudo labels will not participate in the subsequent training of the model.

[0081] Specifically, the related method of the DBSCAN technology can be seen in Martin Ester, Hans-Peter Kriegel, Sander, et al. A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise [C] / / The Second International Conference on Knowledge Discovery and Data Mining (KDD-96), Portland, Oregon, USA. 1996: 226-231.

[0082] Specifically, the related method of the HDBSCAN technology can be seen in Ricardo J.G.B. Campello, Davoud Moulavi, Sander. Density-Based Clustering Based on Hierarchical Density Estimates [C] / / 17th Pacific-Asia Conference on Knowledge Discovery and Data Mining, (PAKDD-13), Gold Coast, Australia. 2013: 160-172. Accelerated algorithm of the technology can be seen in Leland McInnes, John Healy. Accelerated Hierarchical Density Based Clustering [C] / / IEEE International Conference on Data Mining Workshops, (ICDM), New Orleans, LA, USA. 2017: 33-42.

[0083] Step 4: training the fine-grained image classification model using the pseudo-labeled image data, updating the parameters in the model using a batch stochastic gradient descent algorithm, and updating the feature centers of each cluster with momentum; wherein the feature centers of each cluster are obtained by calculating the weighted mean of the normalized features corresponding to the image data of the same pseudo-label.

[0084] The core of training the fine-grained image classification model using unsupervised learning method is to execute the following steps in a loop:

[0085] (1) based on the features extracted by an initial model or an updated model, using an unsupervised clustering algorithm to cluster the data and generate pseudo-labels;

[0086] (2) using the generated pseudo-labels to guide the model to learn and update the parameters.

[0087] The entire process does not need to rely on manual pre-labeling of data. In the above iterative process, the features extracted by the initial model or the updated model and the clustering algorithm are the core factors that affect the clustering results (i.e. the generated pseudo-labels); on the other hand, the generated pseudo-labels will be used to guide the update of the model parameters, which will further affect the updated model to extract new features. Therefore, in the above process, the clustering algorithm and the updated model should be able to produce a positive promoting effect, so that the clustering generates pseudo-labels that are constantly getting better, and the model extracts features that are constantly clustered.

[0088] The data with assigned pseudo-labels can calculate the feature center of each class according to the division of the class cluster. The feature center of each class is obtained by the following weighted average:

[0089]

[0090] In formula (2), is the feature center of the kth class, N k is the number of samples in the kth class, is the normalized feature of the ith sample of the kth class, is the corresponding weighted weight, obtained by the following formula:

[0091]

[0092] In formula (3), is the set of all sample features of the kth class, exp() represents the exponential function, and distance metric Any of the following three calculation methods can be used for calculation:

[0093] (1) Without considering distance constraints, take

[0094] (2) Calculate by minimum distance: represents the minimum distance of the feature to the feature of all other samples in the class;

[0095] (3) Calculate by average distance: represents the average distance of the feature to the feature of all other samples in the class.

[0096] Among the three optional schemes, the first scheme is simple to calculate, and the third scheme has the best effect. Different schemes can be selected according to needs.

[0097] Specifically, the batch stochastic gradient descent algorithm samples an input batch data in each iteration. The sampling method is as follows: first, randomly sample P clusters, and then sample Q samples in each cluster. For clusters with a sample number greater than or equal to Q, Q samples are randomly selected by non-replacement sampling; for clusters with a sample number less than Q, Q samples are randomly selected by replacement sampling, and finally a batch data with a total number of PQ samples is constructed. The above sampling is only performed on the data with assigned pseudo-labels, and the data without assigned pseudo-labels is not sampled.

[0098] Specifically, the fine-grained image classification model uses the InfoNCE loss function to guide learning, and the target loss is:

[0099]

[0100] In formula (4), f q is the normalized feature of the qth sample, c + represents the feature center of the class to which the qth sample feature belongs, c krepresents the feature center of the kth class, t represents a temperature hyperparameter, d(,) represents the distance of normalized features to the class feature center, and the Euclidean distance metric is adopted; when f q the closer to its class feature center c + the farther from other class feature centers, the smaller the loss.

[0101] Within a single outer loop, all parameters of the fine-grained image classification model are updated by the InfoNCE loss function, and each class feature center is updated by momentum; between two outer loops, each class feature center is reconstructed, and the update strategy of a single outer loop is performed.

[0102] The batch stochastic gradient descent algorithm is used to update the parameters in the model, the Adam optimizer is selected for parameter updating, and the feature centers of each cluster are updated by momentum. The class feature center is updated by momentum using the mean of all sampled samples in the class, and is represented as follows:

[0103]

[0104] In formula (5), Q k is the sample index set of the kth class, f q represents the sample feature belonging to the set Q k , m represents the update momentum, and c k is the class feature center of the kth class.

[0105] Step 5: Repeat the clustering and training process, and exit when the clustering result is unchanged for multiple times or reaches the preset maximum number of executions, to obtain the trained unsupervised fine-grained image classification model and the pseudo label of the fine-grained image.

[0106] The predetermined clustering method and the fine-grained image classification model can learn the best network model through continuous optimization and training by using the training method, and can obtain the label of the obtained image.

[0107] Embodiment two:

[0108] As shown in Figure 2 , the embodiment of the application provides an image classification method, which comprises:

[0109] Obtaining fine-grained image data;

[0110] Using the trained unsupervised fine-grained image classification model to perform feature extraction on the obtained image data and obtain a label set of the obtained image data, and constructing a feature index tree;

[0111] The unsupervised fine-grained image classification model trained is used for feature extraction of images in a to-be-predicted data set, and the nearest neighbor or k-nearest neighbor data in a feature index tree is searched, and the label of the nearest neighbor data or the labels of the k-nearest neighbor data are voted to obtain the label of the to-be-predicted image.

[0112] The unsupervised fine-grained image classification model is obtained by the method of embodiment two.

[0113] Specifically, the obtained fine-grained recognition data set does not need to be labeled, and the following two feasible methods can be used for collection:

[0114] Method 1: Camera shooting collection. For example, fine-grained images of vehicles are collected directly by using a snapshot camera in a traffic environment; animal images are directly shot by using a monitoring camera in an animal farm; pedestrian images can be shot by using an access control system in a train station, an airport, etc.; military targets can be shot by using a drone, a remote sensing satellite, etc. in military reconnaissance; in the biological field, microscopic cell bodies and other targets can be directly shot, and similar methods can be used for collection in many other fields.

[0115] Method 2: Internet search collection. Common targets such as vehicles and animals can be directly downloaded from the Internet, and different fine-grained categories of data can be downloaded in batches by using a search engine query interface, and data can also be downloaded from a special website by using a search engine crawler interface. Since the data set does not need to be labeled, any vehicle or animal of a certain category can be collected without the need for artificial screening.

[0116] According to specific conditions, the obtained fine-grained image data can be used to obtain an unsupervised fine-grained image classification model by using the clustering-based unsupervised fine-grained image classification model training method of embodiment 1. The feature index tree can be constructed according to the labels of the images obtained in embodiment one.

[0117] Embodiment three:

[0118] The embodiment of the application provides a clustering-based unsupervised fine-grained image classification model training system, which comprises:

[0119] The first acquisition module is used for acquiring fine-grained image data.

[0120] The first feature extraction module is used for extracting features of fine-grained images and performing normalization processing by using a fine-grained image classification model pre-trained or trained in a training process, to obtain normalized features.

[0121] The clustering pseudo-label generation module is used for clustering the normalized features by using a predetermined clustering method, assigning pseudo-labels to corresponding image data according to clustering results, and constructing image data with pseudo-labels.

[0122] a training module configured to train the fine-grained image classification model using the image data with pseudo labels, update parameters in the model using a batch stochastic gradient descent algorithm, and update feature centers of each cluster using momentum;

[0123] a model output module configured to repeat the clustering and training processes, exit when the clustering result is unchanged for a plurality of times or reaches a preset maximum number of executions, and obtain an unsupervised fine-grained image classification model trained and pseudo labels of fine-grained images.

[0124] Embodiment Four

[0125] The embodiment of the present application provides an image classification system, which is characterized in that comprising:

[0126] a second acquisition module configured to acquire fine-grained image data;

[0127] a second feature extraction module configured to perform feature extraction on the acquired image data using the unsupervised fine-grained image classification model trained and obtain a label set of the acquired image data, and construct a feature index tree;

[0128] a label output module configured to perform feature extraction on images in a to-be-predicted data set using the unsupervised fine-grained image classification model trained, search for nearest neighbors or k nearest neighbors in the feature index tree, adopt a label of the nearest neighbor or vote for a label of the k nearest neighbors to make a decision, and obtain a label of a to-be-predicted image; wherein the unsupervised fine-grained image classification model is trained using the method of any one of claims 1 to 7.

[0129] Embodiment Five

[0130] The embodiment of the present application provides an unsupervised fine-grained image classification system based on clustering, comprising a processor and a storage medium.

[0131] The storage medium is configured to store instructions.

[0132] The processor is configured to operate according to the instructions to perform the steps of the method of embodiment one.

[0133] Embodiment Six

[0134] The embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the method of embodiment one.

[0135] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0136] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0137] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0139] The above description is only preferred embodiments of the application. It should be pointed out that for those skilled in the art, some improvements and modifications can be made without departing from the technical principles of the application, and these improvements and modifications should also be considered as falling within the scope of the application.

Claims

1. A method for training a clustering-based unsupervised fine-grained image classification model, the method comprising: The method comprises the following steps: obtaining fine-grained image data; extracting features of the fine-grained image and performing normalization processing by using a fine-grained image classification model in a pre-training or training process to obtain normalized features; performing clustering on the normalized features by using a predetermined clustering method, assigning pseudo-labels to corresponding image data according to a clustering result, and constructing image data with pseudo-labels; The pseudo-labeled image data is used to train a fine-grained image classification model, a batch stochastic gradient descent algorithm is used to update parameters in the model, and the feature centers of each cluster are updated with momentum. (3), In equation (3), For the first q Normalized features of each sample Indicates the first q The feature center of the class to which a sample's feature belongs. Indicates the first k Feature centers of each class t This indicates temperature hyperparameters. Represents the distance from the normalized feature to the class feature center, using Euclidean distance metric; when Its class feature center The closer the feature centers are to other classes, the smaller the loss. wherein the feature center of each cluster is obtained by calculating a weighted mean value of the normalized features corresponding to the image data with the same pseudo-label; The feature center of each cluster is obtained by the following formula: (1), In formula (1), is a feature center of the first k class, is the number of samples in the first k class, is the normalized feature of the i-th sample of the first k class, i is the i-th sample of the first class, is a corresponding weighted weight, obtained by the following formula: In formula (2), is the first k set of all sample features of the first class, denotes an exponential function, and distance metric is calculated by any one of the following three calculation methods: (1) Without considering the distance constraint, take ; (2) By minimum distance calculation: represents the calculation of the minimum distance of the feature to all other sample features of the class; (3) Average distance calculation: , which represents the average distance of the feature to all other samples of the class. within a single outer loop, updating all parameters of the fine-grained image classification model by using an InfoNCE loss function, and updating the feature center of each cluster by using momentum; between two outer loops, the feature center of each cluster is reconstructed, and the update strategy of a single outer loop is executed; The batch stochastic gradient descent algorithm is used to update the parameters in the model, the Adam optimizer is selected to update the parameters, and the momentum is used to update the feature centers of each cluster. The class feature center is updated by the mean of all sampling samples in the class, which is represented as follows: (4), In equation (4), For the first k A collection of sample indexes for a class. Indicates belonging to a set Sample characteristics, m This indicates an update in momentum. For the first k The class characteristic center of a class; repeating the clustering and training process, and exiting when the clustering result is unchanged for multiple times or reaches a preset maximum number of executions, to obtain a trained unsupervised fine-grained image classification model and pseudo-labels of the fine-grained image. 2.The cluster-based unsupervised fine-grained image classification model training method of claim 1, wherein, The fine-grained image classification model comprises a network layer, a fully connected layer, a global mean pooling layer with parameters, and a processing layer, and the unsupervised fine-grained image is sequentially input into the network layer, the fully connected layer, and the global mean pooling layer with parameters for feature extraction, and the extracted features are input into the processing layer for L2 normalization. 3.The cluster-based unsupervised fine-grained image classification model training method of claim 1, wherein, The predetermined clustering method comprises a DBSCAN clustering algorithm or an HDBSCAN clustering algorithm. 4.The method of claim 1, wherein, The batch stochastic gradient descent algorithm samples a batch of data for each iteration, and the sampling method comprises: randomly sampling P clusters; Randomly sample within each cluster Q one sample, for clusters with sample number greater than or equal to Q , use random selection without replacement to select Q samples; for clusters with sample number less than Q , use random selection with replacement to select Q samples; A batch of data consisting of PQ a number of samples is finally obtained.

5. An image classification method characterized by, The method comprises the following steps: obtaining fine-grained image data; extracting features of the fine-grained image and performing normalization processing by using a fine-grained image classification model in a pre-training or training process to obtain normalized features; performing clustering on the normalized features by using a predetermined clustering method, assigning pseudo-labels to corresponding image data according to a clustering result, and constructing image data with pseudo-labels; wherein the feature center of each cluster is obtained by calculating a weighted mean value of the normalized features corresponding to the image data with the same pseudo-label; 6. A clustering-based unsupervised fine-grained image classification model training system, which executes the clustering-based unsupervised fine-grained image classification model training method according to any one of claims 1 to 4. within a single outer loop, updating all parameters of the fine-grained image classification model by using an InfoNCE loss function, and updating the feature center of each cluster by using momentum; between two outer loops, the feature center of each cluster is reconstructed, and the update strategy of a single outer loop is executed; repeating the clustering and training process, and exiting when the clustering result is unchanged for multiple times or reaches a preset maximum number of executions, to obtain a trained unsupervised fine-grained image classification model and pseudo-labels of the fine-grained image. The fine-grained image classification model comprises a network layer, a fully connected layer, a global mean pooling layer with parameters, and a processing layer, and the unsupervised fine-grained image is sequentially input into the network layer, the fully connected layer, and the global mean pooling layer with parameters for feature extraction, and the extracted features are input into the processing layer for L2 normalization. The predetermined clustering method comprises a DBSCAN clustering algorithm or an HDBSCAN clustering algorithm. The batch stochastic gradient descent algorithm samples a batch of data for each iteration, and the sampling method comprises: randomly sampling P clusters; PQ The method comprises the following steps: obtaining fine-grained image data; extracting features of the fine-grained image and performing normalization processing by using a fine-grained image classification model in a pre-training or training process to obtain normalized features; performing clustering on the normalized features by using a predetermined clustering method, assigning pseudo-labels to corresponding image data according to a clustering result, and constructing image data with pseudo-labels; wherein the feature center of each cluster is obtained by calculating a weighted mean value of the normalized features corresponding to the image data with the same pseudo-label; within a single outer loop, updating all parameters of the fine-grained image classification model by using an InfoNCE loss function, and updating the feature center of each cluster by using momentum; between two outer loops, the feature center of each cluster is reconstructed, and the update strategy of a single outer loop is executed; repeating the clustering and training process, and exiting when the clustering result is unchanged for multiple times or reaches a preset maximum number of executions, to obtain a trained unsupervised fine-grained image classification model and pseudo-labels of the fine-grained image. The fine-grained image classification model comprises a network layer, a fully connected layer, a global mean pooling layer with parameters, and a processing layer, and the unsupervised fine-grained image is sequentially input into the network layer, the fully connected layer, and the global mean pooling layer with parameters for feature extraction, and the extracted features are input into the processing layer for L2 normalization. The model output module is configured to repeat the clustering and training process, exit when the clustering result is constant for multiple times or reaches a preset maximum execution number, and obtain a trained unsupervised fine-grained image classification model and pseudo labels of fine-grained images.

7. An image classification system characterized by, The method comprises the following steps: The second acquisition module is configured to acquire fine-grained image data. The second feature extraction module is configured to perform feature extraction on the acquired image data by using the trained unsupervised fine-grained image classification model, and obtain a label set of the acquired image data, and construct a feature index tree. The label output module is configured to perform feature extraction on images in a to-be-predicted data set by using the trained unsupervised fine-grained image classification model, search for nearest neighbor data or k nearest neighbor data in the feature index tree, adopt a label of the nearest neighbor data or vote for labels of the k nearest neighbor data to make a decision, and obtain a label of a to-be-predicted image; wherein the unsupervised fine-grained image classification model is trained by using the method in any one of claims 1 to 4.

Images