A Fine-Grained Image Classification Method and System

Abstract

The present application discloses a fine-grained image classification method and system thereof. The fine-grained image classification method specifically includes the following steps: initializing a hierarchical vision transformer network with pre-trained parameters; performing sampling of data; performing bottom-layer feature contrast learning based on the sampled data to obtain a first prediction loss; performing deep-layer feature balance processing based on the sampled data to obtain a second prediction loss; obtaining an overall loss according to the first prediction loss and the second prediction loss; updating network parameters through backpropagation; repeating the above steps until the parameters of the hierarchical vision transformer network converge, and saving the updated network parameters; initializing the hierarchical vision transformer network using the saved network parameters; and predicting the image category according to the initialized hierarchical vision transformer network. When optimizing the fine-grained image classification task, the present application simultaneously optimizes the problems of fine-grained features and long-tail distribution.

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically, to a fine-grained image classification method and system thereof. Background Art

[0002] The fine-grained image classification task refers to classifying images of multiple similar subclasses under the same category, such as industrial product defect classification, similar species classification, etc. Different categories are very similar and can usually only be distinguished by subtle features. At the same time, since the objects of the fine-grained image classification task usually come from the same field, there will usually be a certain degree of skew among different categories, such as the rarity of species, the probability of product defects, and the popularity of clothing, etc. During the process of collecting data for the fine-grained image classification task, this skew will lead to a huge difference in the data collection costs of different categories. And the difference in collection costs will result in a large number of samples in the actually collected data set only coming from a few categories, while the number of samples in most categories is small. Usually, the categories containing a large number of samples are called head categories, and the remaining categories are called tail categories, and this problem is called the long-tail distribution problem. The severe skew of the data set distribution leads to an obvious bias of the classification towards the head categories, and the classification boundary shifts towards the tail categories, making it easy for the tail data to be judged as the head categories. To sum up, the difficulties of the fine-grained image classification task mainly come from two aspects: 1) The difficulty of distinguishing different categories caused by fine-grained features; 2) The damage to the tail categories caused by the long-tail distribution. Early methods were usually designed based on convolutional neural network technology and achieved certain results on the above two problems. With the rise of Transformer (Vision Transformer Network) technology in the field of computer vision, Transformer technology has begun to be applied to improve the fine-grained image classification task.

[0003] The long-tail distribution problem is a common problem widely existing in many fields. However, compared with the conventional image classification task, the fine-grained image classification task has the characteristics of "large intra-class variance and small inter-class variance". Fine-grained features will further transform the problem of the head classes encroaching on the tail classes caused by the long-tail distribution into the problem of the head sub-classes encroaching on the tail similar sub-classes. This encroachment causes the distinguishable fine-grained features for differentiating the head and tail similar classes to be damaged, greatly increasing the classification difficulty. Current methods usually study the long-tail distribution and the fine-grained feature optimization as two independent directions. For the fine-grained image classification task, only the problem of difficult fine-grained feature classification is considered, and the long-tail distribution optimization method only considers the influence brought by the data distribution. However, in the real environment of the fine-grained image classification problem, the long-tail distribution and the fine-grained features exist simultaneously. Therefore, the existing methods perform poorly in solving the fine-grained image classification task in the real scenario. At the same time, a large number of existing studies are mainly designed based on convolutional neural networks, and the performance of convolutional neural networks has gradually tended to its limit in recent years, and the performance of related fine-grained image classification methods is also difficult to be further improved. Currently, the existing application methods of Transformer in the fine-grained image classification task have effectively improved the classification accuracy, proving its advantages in solving the fine-grained image classification task. However, the existing Transformer-based methods only consider the optimization of fine-grained features, but do not optimize for the long-tail distribution, and there is still a lack of effective optimization methods for the problem of their mutual coupling.

[0004] Therefore, how to provide a method that can solve the classification difficulty problem caused by fine-grained features and long-tail distribution is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of the present invention is to propose a fine-grained image classification method optimized based on Transformer. This method uses Transformer to solve the fine-grained image classification task. By optimizing the Transformer method, it simultaneously solves the classification difficulty problem caused by fine-grained features and long-tail distribution. The present invention optimizes the existing Transformer method, while improving the encroachment of the long-tail distribution problem on the tail class features, protects the learning ability of the overall features, and enhances the discrimination ability of Transformer for fine-grained features. The present invention aims to improve the encroachment of the fine-grained features and long-tail distribution problem on the tail classes during the classification learning process, and at the same time reduce the damage of the optimization method to the head classes. Moreover, the present invention further protects the learning of fine-grained features in this process, increasing the distance between different classes in the feature space. Finally, by optimizing the Transformer method, the overall classification accuracy is improved, and the robustness of the classification network is enhanced.

[0006] To solve the above problems, the present application provides a fine-grained image classification method, which specifically includes the following steps: S1, initializing a hierarchical vision transformer network with pre-trained parameters; S2, in response to the completion of the initialization of the hierarchical vision transformer network, performing data sampling; S3, performing bottom-layer feature contrast learning based on the sampled data to obtain a first prediction loss; S4, performing deep-layer feature balance processing based on the sampled data to obtain a second prediction loss; S5, obtaining an overall loss based on the deep-layer balance processing and the bottom-layer contrast learning; S6, in response to obtaining the overall loss, updating the network parameters through backpropagation; repeating steps S1-S6 until the parameters of the hierarchical vision transformer network converge, and then performing step S7; S7, saving the updated network parameters; S8, initializing the hierarchical vision transformer network using the saved network parameters; S9, predicting the image category based on the initialized hierarchical vision transformer network.

[0007] As described above, the pre-trained parameters are the parameters pre-trained by the hierarchical vision transformer.

[0008] As described above, performing data sampling includes using a positive sampler, a negative sampler, and a confusion data sampler to perform data sampling in the training dataset, and finally generating a triple data composed of the above three samplers.

[0009] As described above, the sampling process of the negative sampler includes: obtaining the weight of each category; calculating the sampling probability of each category according to the weight of each category; the weight w i is expressed as: N i represents the number of samples of the i-th category in the dataset set; the sampling probability p i is specifically expressed as: where w i represents the weight of the i-th category, w j represents the weight of the j-th category, and n represents the number of categories.

[0010] As described above, after obtaining the sampling probability p i the negative sampler extracts a category from the training dataset according to p i and then randomly samples a sample from the data of this category to complete the data sampling of the negative sampler.

[0011] As described above, in which, based on the sampled data, bottom-layer feature contrast learning is performed to obtain the first prediction loss, including the following sub-steps: feature extraction is performed on the data of positive sampling and confused data sampling to obtain intermediate features; the obtained intermediate features are mixed to obtain bottom-layer mixed features; the bottom-layer mixed features are successively passed through the subsequent network layers of the vision transformer network to obtain final mixed features; the final mixed features are fed into a classifier for prediction to obtain a prediction probability distribution; the first prediction loss is obtained according to the prediction probability distribution.

[0012] As described above, in which, a pair of data (x, y), (x', y') is obtained through positive sampling and confused data sampling, where x, x' are image sample data, and y, y' are corresponding true class labels, and x, x' are successively passed through the shallow network of the vision transformer network to obtain intermediate features h, h'.

[0013] As described above, in which, based on the sampled data, deep-layer feature balance processing is performed to obtain the second prediction loss, including the following steps: feature extraction is performed on the data of positive sampling and negative sampling to obtain deep-layer features; predictions are made on the deep-layer features to obtain prediction results, and the prediction results are dynamically mixed to obtain a mixed prediction result and a mixed prediction probability distribution; in response to obtaining the mixed prediction probability distribution, the prediction labels obtained by negative sampling are label-smeared to obtain smeared labels; the prediction loss of the mixed prediction result is obtained according to the smeared labels.

[0014] As described above, in which, the deep-layer features include head deep-layer features and tail deep-layer features, and a classifier is used for the head deep-layer features to make predictions, and a classifier is used for the tail deep-layer features to make predictions, and the results predicted by the two classifiers are mixed for prediction. The mixed prediction result f is specifically expressed as:

[0015] f = αWz+(1 - αW”z”)

[0016] where α is used to control the proportion of the positive sampling branch and the negative sampling branch.

[0017] A fine-grained image classification system specifically includes a first initialization unit, a data mixed sampling unit, a low-level feature contrast learning unit, a deep feature balancing unit, an overall loss obtaining unit, a parameter updating unit, a saving unit, a second initialization unit, and a predicted image category unit. The initialization unit is used to initialize the hierarchical vision transformer network with pre-trained parameters. The data mixed sampling unit is used to perform data sampling. The low-level feature contrast learning unit is used to perform low-level feature contrast learning based on the sampled data to obtain a first prediction loss. The deep feature balancing unit is used to perform deep feature balancing processing based on the sampled data to obtain a second prediction loss. The overall loss obtaining unit is used to obtain an overall loss based on the deep balancing processing and the low-level contrast learning. The parameter updating unit is used to update the pre-trained parameters through backpropagation. The saving unit is used to save the pre-trained parameters after training is completed. The second initialization unit is used to initialize the hierarchical vision transformer network with the saved parameters. The predicted image category unit is used to predict the image category based on the hierarchical vision transformer network initialized with the saved parameters.

[0018] The present application has the following beneficial effects:

[0019] (1) When optimizing the fine-grained image classification task, the method proposed in the present application simultaneously optimizes the problems of fine-grained features and long-tail distribution. Existing research has not optimized the above two problems simultaneously. Therefore, the present application makes up for the deficiencies of existing research and effectively solves this problem.

[0020] (2) The present application successfully applies the Transformer to the fine-grained image classification task. Currently, in the field of fine-grained image classification, the application research of the Transformer is less. The present invention effectively improves the image classification accuracy in this scenario by improving the existing Transformer method.

[0021] (3) The method proposed in the present application can be flexibly applied to various Transformer variant networks and has good migration ability. In the actual use process, the Transformer network can be flexibly replaced according to the actual conditions, which greatly reduces the application difficulty. Description of the Drawings

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings described below are only some embodiments recorded in the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0023] Figure 1It is the internal structure diagram of the fine-grained image classification system provided by the embodiments of the present application;

[0024] Figure 2 It is the flowchart of the fine-grained image classification method provided by the embodiments of the present application. Detailed implementation manners

[0025] The following combines the accompanying drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts fall within the protection scope of the present application.

[0026] The present invention proposes a fine-grained image classification method optimized based on Transformer. This method is optimized and improved on the basis of Transformer.

[0027] Embodiment 1

[0028] As Figure 1 shown, it is a fine-grained image classification system provided by the present application, specifically including: a first initialization unit 101, a data hybrid sampling unit 102, a low-level feature contrast learning unit 103, a deep feature balancing unit 104, an overall loss acquisition unit 105, a parameter update unit 106, a storage unit 107, a second initialization unit 108, and a predicted image category unit 109.

[0029] The initialization unit 101 is used to initialize the hierarchical vision transformer network with pre-trained parameters.

[0030] The data hybrid sampling unit 102 is connected to the initialization unit 101 and is used for data sampling.

[0031] Specifically, the data hybrid sampling unit 102 is composed of a positive sampler, a negative sampler, and a confused data sampler, and this sampler will generate triple data composed of the above three samplers each time during the training process.

[0032] The low-level feature contrast learning unit 103 is connected to the data hybrid sampling unit 102 and is used for low-level feature contrast learning according to the sampled data to obtain the first prediction loss.

[0033] The deep feature balancing unit 104 is connected to the data hybrid sampling unit 102 and is used for deep feature balancing processing according to the sampled data to obtain the second prediction loss.

[0034] The overall loss acquisition unit 105 is respectively connected to the underlying feature contrast learning unit 103 and the deep feature balancing unit 104, and is used to obtain the overall loss according to the deep balancing process and the underlying contrast learning.

[0035] The parameter update unit 106 is connected to the overall loss acquisition unit 105, and is used to update the pre-trained parameters through backpropagation.

[0036] The saving unit 107 is connected to the parameter update unit 106, and is used to save the pre-trained parameters after the training is completed.

[0037] The second initialization unit 108 is connected to the saving unit 107, and is used to initialize the hierarchical vision transformer network using the saved parameters.

[0038] The predicted image category unit 109 is connected to the second initialization unit 108, and is used to predict the image category according to the hierarchical vision transformer network initialized with the saved parameters.

[0039] Embodiment 2

[0040] As Figure 2 shown, it is a fine-grained image classification method provided by the present application. In this example, a hierarchical vision transformer network (abbreviated as Swin Transforemr network) is used as the backbone network, which is one of the variant networks of Transformer and can gradually downsample during the forward propagation of the network to provide multi-scale feature information.

[0041] In this example, the training dataset used is IP102, which is a large-scale fine-grained image classification dataset of pests, including 102 sub-categories of pests, including 45,095 training pictures, 7,508 validation pictures, and 22,619 test pictures. Insects themselves are very small, and many categories need to be distinguished by experts in related fields. At the same time, this training dataset presents a natural long-tail distribution, and its maximum imbalance rate is 80.8, that is, the maximum number of samples in a category is 80.8 times that of the minimum category.

[0042] Step S210: Initialize the hierarchical vision transformer network using the pre-trained parameters.

[0043] Wherein the pre-trained parameters are the parameters pre-trained by the hierarchical vision transformer.

[0044] Step S220: In response to the completion of the initialization of the hierarchical vision transformer network, perform data sampling.

[0045] Among them, a positive sampler, a negative sampler, and a confusing data sampler are used to perform data sampling in the training dataset, and finally generate a triple data composed of the above three samplers.

[0046] Specifically, during one training loop, the positive sampler samples all samples in the training dataset with the same probability once. The input data obtained through positive sampling completely retains the original distribution characteristics of the training dataset and does not damage the diversity of the data. Therefore, it is beneficial to the feature learning process.

[0047] The main role of the negative sampler is to increase the sampling probability of tail data, thereby reducing the influence of head classes on tail classes during training. Under the negative sampler, the probability of each class being drawn is inversely proportional to the total number of samples of that class, that is, the more samples a class has, the lower the likelihood of that class being sampled. The following is the sampling process of the negative sampler:

[0048] For a dataset S = {S1, S2, …, S n}, where S i represents the data set of class i, and N i represents the number of samples of the i-th class in the dataset set, then the weight w i of this class is:

[0049]

[0050] Then, calculate the sampling probability p i of each class specifically as:

[0051]

[0052] where w i represents the weight of the i-th class, w j represents the weight of the j-th class, and n represents the number of classes.

[0053] After obtaining the sampling probability p i , the negative sampler first draws a class from the training dataset according to p i , and then randomly samples a sample from the data of that class to complete the data sampling of the negative sampler. Compared with the positive sampler, the sampling of the negative sampler is sampling with replacement.

[0054] The design purpose of the mixed data sampling is to improve the ability to extract fine-grained features on the basis of protecting the learning of common features. Therefore, the mixed data sampler needs to sample in pairs with the positive sampler. According to the samples obtained by positive sampling, sample the data of its mixed classes from the training dataset. The sampling process of the mixed data sampler is as follows:

[0055] First, in order to obtain the mixed classes, the class confusion matrix predicted by the hierarchical vision transformer network for the dataset S is required where Mi Represents the prediction statistical result of the i-th category, The value of represents the probability that the i-th category is predicted as the j-th category, representing the degree of confusion between categories. However, this confusion is usually not symmetric, that is, the probability of confusing category i as category j is usually not equal to the probability of confusing category j as category i, which also indirectly shows that the infringement between categories is not symmetric. During the training process, M comes from the prediction statistical result of the previous training cycle. Specifically, in the first training cycle, M i Is initialized to a uniform distribution, that is, it is assumed that the degree of confusion between all categories is the same. Starting from the second training cycle, the prediction statistical result of the previous round of positive sampler data is used as the confusion matrix.

[0056] For the collected sample x, the prediction result can be obtained after passing through the backbone network and the classifier Pass z through softmax to obtain the prediction probability for the corresponding category i ∈ {1, 2, …, n}

[0057]

[0058] The prediction probability distribution corresponding to the sample x is Then, count the average value of the prediction probability distributions of all samples in the i-th category dataset S i And use the obtained average result to update M i :

[0059]

[0060] Where N i Represents the number of samples in the i-th category in the dataset collection, and S i Represents the data set of category i,

[0061] In the sampling stage, in order to generate paired data, the confusion data sampler will be synchronized with the positive sampler. First, obtain the data category sampled by the positive sampler. Taking category j as an example, since the data sampled by the positive sampler is mainly the head category, the confusion data sampling will draw one category from the C - 1 categories that are easily confused as category j according to the confusion weight Extract one category. Different from anti-sampling, the confusion data sampling will not draw the category corresponding to itself. If there is no easily confused category, it will be obtained through random sampling. Then, randomly draw a data from the data set S corresponding to this category j With replacement, and this data will be paired with the data sampled by the positive sampler for training.

[0062] Step S230: Perform bottom-layer feature contrast learning based on the sampled data to obtain the first prediction loss.

[0063] Among them, the underlying feature contrast learning is to compare the differences in the underlying features of easily confused categories in the Transformer, and guide the Transformer to learn distinguishable fine-grained features, thereby improving the recognition of fine-grained features. The specific steps of step S130 include the following sub-steps:

[0064] Step S2301: Extract features from the data of positive sampling and confused data sampling to obtain intermediate features.

[0065] Specifically, a set of paired data (x, y), (x', y') is obtained through positive sampling and confused data sampling, where x and x' are the image sample data obtained by positive sampling and confused sampling respectively, and y and y' are the true class labels corresponding to the image sample data obtained by positive sampling and confused sampling respectively. Pass x and x' through the shallow network of the Transformer in sequence to obtain the intermediate features h and h' corresponding to positive sampling and confused data sampling.

[0066] Furthermore, y and y' are also the true class labels corresponding to the intermediate features h and h'.

[0067] There is a corresponding relationship among x, x', y, y', h, h', and x, y, h, and there is also a corresponding relationship among x′, y′, h′.

[0068] Step S2302: Mix the obtained intermediate features to obtain the underlying mixed features.

[0069] The underlying mixed features specifically include mixed features and mixed labels.

[0070] Among them, the intermediate features h, h' obtained in S2301 and their corresponding labels are fused using the Mixup method to obtain the mixed features of the paired data and the mixed labels

[0071]

[0072]

[0073] Among them, λ is generated by the generator and is the mixing coefficient of the intermediate features and their corresponding labels. During the training process, the generator samples a λ from the Beta distribution in each batch, λ ∼ Beta(β, β).

[0074] Among them, β is a hyperparameter used to control the distribution of λ, and the probability density function f(z) of the Beta(β, β) distribution is specifically expressed as:

[0075]

[0076] Since the values of the Beta distribution are between 0 and 1, it is suitable to provide parameters for the mixing ratio, which can ensure that the generated λ meets the requirements. Since the data sampled positively is relatively important in the early stage of training, λ is taken in stages, and the staged value-taking is specifically expressed as:

[0077]

[0078] where epoch represents the current number of training times, and t is used to control the segmentation position of λ. In the early stage of training, the range of λ is [0.5, 1]. That is, in the mixing process, the feature h corresponding to the positively sampled data x has a larger weight ratio. While controlling the contrastive learning, λ will be more biased towards the head category during parameter update to help the model better model the common features from a large amount of data. In the later stage of training, the range of λ is [0, 0.5], and the feature h' corresponding to the paired confusing data x' will have a larger weight, so as to better strengthen the learning of fine-grained features and improve the confusion situation.

[0079] Step S2303: Pass the underlying mixed features through the subsequent network layers of the Transformer in sequence to obtain the final mixed features.

[0080] Specifically, pass the underlying mixed features through the subsequent network layers of the Transformer in sequence to obtain the final mixed features

[0081] Step S2304: Feed the final mixed features into the classifier for prediction to obtain the predicted probability distribution.

[0082] Specifically, flatten the final mixed features and obtain them through global average pooling Then feed the final mixed features after global average pooling into the classifier for prediction, and pass the prediction result through softmax to obtain the predicted probability of each category:

[0083]

[0084] The predicted probability distribution of the mixture of the features of samples x and x' is

[0085] Step S2305: Calculate the first prediction loss according to the predicted probability distribution.

[0086] where the prediction loss is the distance loss between the prediction distribution and the mixed label. Since the mixture results in no longer being a one-hot form probability distribution, the KL divergence is used to calculate the prediction distribution The distance loss between the mixed tags, i.e., the first prediction loss:

[0087]

[0088] Step S240: Perform deep feature balancing processing on the sampled data to obtain the second prediction loss.

[0089] Among them, deep feature balancing is actually to improve the learning of overall features and alleviate the problem of the long-tail distribution affecting the tail categories.

[0090] Among them, step S240 specifically includes the following steps:

[0091] Step S2401: Extract features from the data of positive sampling and negative sampling to obtain head deep features and tail deep features.

[0092] Specifically, a set of data (x, y), (x”, y”) is obtained through positive sampling and negative sampling, and x, x” are input into the Transformer to extract features.

[0093] Specifically, the Stage4 network layers of two Swin Transformers with the same structure are used for feature extraction on the positive sampling branch and the negative sampling branch to obtain head deep features and tail deep features.

[0094] Among them, the branch for calculating positive sampling data is the positive sampling branch, and the branch for calculating negative sampling data is the negative sampling branch.

[0095] In order to model the deep features of tail categories and head categories, different deep networks will be used for feature extraction on the positive sampling branch and the negative sampling branch, and head deep features and tail deep features

[0096] Step S2402: Predict the obtained head deep features and tail deep features to obtain prediction results, and perform dynamic mixing on the prediction results to obtain mixed prediction results and mixed prediction probability distributions.

[0097] Among them, two different classifiers will be used on the positive sampling branch and the negative sampling branch respectively and for prediction.

[0098] Specifically, the head deep features are predicted using the head classifier W, and the tail deep features are predicted using the tail classifier W”.

[0099] Among them, the head classifier W tends to predict the input as a certain head category according to z, while the tail classifier W” tends to predict the input as a certain tail category according to z”. Therefore, in order to enable the network to comprehensively consider the depth features and inference biases of the head category and the tail category during the inference process, the inference results of the two classifiers are weighted to obtain the mixed prediction result f, which is specifically expressed as:

[0100] f = αWz+(1 - α)W”z”

[0101] Among them, α represents the mixing coefficient of the data collected by positive sampling and negative sampling, which is used to control the proportion of the positive sampling branch and the negative sampling branch, and can directly affect the attention of the model to the head category and the tail category during the training process. In the early stage of training, relying on its diversity and rich features, the positive sampling data helps to model the features of the head category and the general features of the head category and the tail category. In the early stage, through parameter sharing, the shallow features of the negative sampling branch can be optimized through the positive sampling branch. As the training progresses and the learning rate decreases, the parameters of the underlying layer are almost no longer updated. Increasing the weight of the negative sampling branch helps to model the deep features of the tail data and will not overly affect the general features that have been learned, improving the robustness of the deep features.

[0102] Furthermore, in order to dynamically obtain α during the training process, an exponential function generator is designed in this embodiment. Using m to represent the number of the current training cycle and M to represent the total number of training times, then the α of the current training cycle is:

[0103]

[0104] where e m represents the exponential function with base e and exponent m, and e M represents the exponential function with base e and exponent M.

[0105] Then, the mixed prediction result f is normalized through softmax to obtain the mixed prediction probability distribution of the samples x, x” as where:

[0106]

[0107] Step S2403: In response to obtaining the mixed prediction probability distribution, perform label smoothing on the predicted labels obtained by negative sampling to obtain smoothed labels.

[0108] It should be noted that there seems to be a small error in the formula in your original text. It should be "f = αWz+(1 - α)W”z”" instead of "f = αWz+(1 - αW”z”)". I have corrected it in the translation.During the test phase, set α to 0.5. The positive sampling branch and the negative sampling branch will mix and predict the same input data with the same proportion. However, due to the lack of tail category data, the negative sampling branch can easily fit all the tail training data relying on the powerful feature extraction ability of the Transformer, resulting in overfitting of the negative sampling branch. Therefore, during the training process, the network should not be overconfident in the negative sampling data, but should leave room for the prediction results. To address this issue, this embodiment proposes to use label smoothing to process the prediction labels of the negative sampling branch data to obtain smoothed labels.

[0109]

[0110] Where ε is used to control the degree of label smoothing. Through label smoothing, the prediction target of the network is no longer to predict the probability corresponding to the correct category as 1 as much as possible, but to require the network to predict it as 1 - ε, and predict other categories as ε / (n - 1). The distribution of deep features will be more balanced, thus alleviating the overfitting problem.

[0111] Step S2404: Obtain the prediction loss of the mixed prediction result according to the smoothed label.

[0112] The prediction loss of the mixed prediction result is the second prediction loss. Specifically, mix the smoothed label of the negative sampling branch with the label y of the positive sampling branch to obtain the mixed label of the two branches:

[0113]

[0114] Finally, calculate the distance loss between the prediction distribution and the mixed label through the KL divergence, that is, obtain the prediction loss, where the second prediction loss is specifically expressed as:

[0115]

[0116] It should be noted that the execution processes of steps S230 and S240 are executed in parallel.

[0117] Step S250: Obtain the overall loss according to the first prediction loss obtained by the deep balance processing and the second prediction loss obtained by the bottom contrast learning.

[0118] During the training process, the bottom feature contrast learning and the deep feature balance learning are carried out in parallel. The mixed data sampler obtains triplet data through positive sampling, negative sampling and confused data sampling respectively. Then, the positive sampling data is respectively subjected to balance learning and contrast learning with the negative sampling data and the confused data. The overall loss loss of the training process consists of two parts:

[0119] loss = loss1 + loss2

[0120] Among them, loss1 represents the first prediction loss, and loss2 represents the second prediction loss.

[0121] Step S260: In response to obtaining the overall loss, update the network parameters through backpropagation.

[0122] Repeat the above steps until the parameters of the Swin Transformr network converge and the parameter training is completed, then execute step S270. Step S270: Save the updated network parameters.

[0123] The specific condition for convergence is that the parameters reach the specified number of training times or the overall loss no longer decreases and tends to be stable, then the parameter training is completed and step S270 is executed.

[0124] Among them, steps S210 - S270 are the training process of this embodiment, and the following steps are the specific execution process after the pre-training parameter training is completed.

[0125] Step S280: Initialize the hierarchical vision transformer network using the saved updated network parameters.

[0126] Specifically, load the Swin Transformer network and initialize the Swin Transformer network using the trained network parameters.

[0127] Step S290: Perform image category prediction according to the initialized hierarchical vision transformer network.

[0128] Specifically, send the image to be predicted into the Swin Transformer network to obtain the prediction results of the head category classifier and the tail category classifier in the deep feature balance respectively. Mix the prediction results of the two classifiers according to the formula f = αWz + (1 - αW”z”), where α takes the value of 0.5, and finally obtain the prediction result of the image category.

[0129] This application has the following beneficial effects:

[0130] (1) When optimizing the fine-grained image classification task, the method proposed in this application simultaneously optimizes the fine-grained features and the long-tail distribution problem. Existing research has not optimized the above two problems simultaneously. Therefore, the present invention makes up for the deficiencies of existing research and effectively solves this problem.

[0131] (2) This application has successfully applied Transformer to the fine-grained image classification task. Currently, in the field of fine-grained image classification, there is little research on the application of Transformer. The present invention effectively improves the image classification accuracy in this scenario by improving the existing Transformer method.

[0132] (3) The method proposed in this application can be flexibly applied to various Transformer variant networks and has good migration ability. During actual use, the Transformer network can be flexibly replaced according to actual conditions, greatly reducing the application difficulty.

[0133] Although the examples referred to in the current application are described, they are for explanatory purposes only and not a limitation of the present application. Changes, additions, and / or deletions to the embodiments can be made without departing from the scope of the present application.

[0134] As described above, the above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A fine-grained image classification method, characterized in that, Specifically, it includes the following steps: S1. Initialize the hierarchical vision transformer network using pre-trained parameters; S2. In response to the completion of the initialization of the hierarchical vision transformer network, perform data sampling; S3. Based on the sampled data, conduct bottom-layer feature contrast learning to obtain the first prediction loss; S4. Based on the sampled data, perform deep-layer feature balance processing to obtain the second prediction loss; S5. Based on the first prediction loss obtained from the deep-layer balance processing and the second prediction loss obtained from the bottom-layer contrast learning, obtain the overall loss; S6. In response to obtaining the overall loss, update the network parameters through backpropagation; Repeat steps S1 - S6 until the parameters of the hierarchical vision transformer network converge, and save the updated network parameters; Use the saved network parameters to initialize the hierarchical vision transformer network; Based on the initialized hierarchical vision transformer network, predict the image category; Among them, the data sampling includes using a positive sampler, a negative sampler, and a confounding data sampler to perform data sampling in the training dataset, and finally generating triple data composed of the above three samplers; Among them, based on the sampled data, conduct bottom-layer feature contrast learning to obtain the first prediction loss, including the following sub-steps: Extract features from the data of positive sampling and confounding data sampling to obtain intermediate features; Mix the obtained intermediate features to obtain bottom-layer mixed features; Pass the bottom-layer mixed features through the subsequent network layers of the vision transformer network in sequence to obtain the final mixed features; Send the final mixed features into the classifier for prediction to obtain the prediction probability distribution; Obtain the first prediction loss based on the prediction probability distribution; Among them, based on the sampled data, conduct deep-layer feature balance processing to obtain the second prediction loss, including the following steps: Extract features from the data of positive sampling and negative sampling to obtain deep-layer features; Predict the deep-layer features to obtain the prediction results, and dynamically mix the prediction results to obtain the mixed prediction results and the mixed prediction probability distribution; In response to obtaining the mixed prediction probability distribution, smooth the prediction labels obtained from negative sampling to obtain the smoothed labels; Obtain the prediction loss of the mixed prediction results based on the smoothed labels.

2. The fine-grained image classification method according to claim 1, wherein The pre-trained parameters are the parameters pre-trained for the hierarchical vision transformer.

3. The fine-grained image classification method according to claim 1, characterized in that The sampling process of the negative sampler includes: Obtain the weights of each category; Calculate the sampling probability of each category according to the weights of each category; The weight w of the category i is expressed as: N i represents the number of samples in the i-th class of the dataset collection; Sampling probability p of the category i Specifically expressed as: where w i represents the weight of the i-th category, and w j represents the weight of the j-th category, and n represents the number of categories.

4. The fine-grained image classification method according to claim 3, wherein Obtain the sampling probability p i After that, the inverse sampler is based on p i Extract a category from the training dataset, and then randomly sample a sample from the data of this category to complete the data sampling of the inverse sampler.

5. The fine-grained image classification method according to claim 1, wherein Through positive sampling and confounding data sampling, obtain a pair of data (x, y), (x′, y′), where x, x′ are image sample data, and y, y′ are the corresponding true category labels. Pass x, x′ through the shallow network of the vision transformer network in sequence to obtain intermediate features h, h′.

6. The fine-grained image classification method according to claim 1, wherein The deep features include the head deep features and the tail deep features, and a classifier is used to predict the head deep features and a classifier is used to predict the tail deep features and the results predicted by the two classifiers are mixed and predicted. The mixed prediction result f is specifically expressed as: f = αWz+(1 - αW″z″) Among them, α is used to control the proportion of the positive sampling branch and the negative sampling branch.

7. A fine-grained image classification system, characterized in that, Specifically, it includes the first initialization unit, the data mixing sampling unit, the bottom-layer feature contrast learning unit, the deep-layer feature balance unit, the overall loss obtaining unit, the parameter update unit, the saving unit, the second initialization unit, and the unit for predicting the image category; The initialization unit is used to initialize the hierarchical vision transformer network using pre-trained parameters; A data hybrid sampling unit for data sampling; A low-level feature contrast learning unit for performing low-level feature contrast learning based on the sampled data to obtain a first prediction loss; A deep feature balancing unit for performing deep feature balancing processing based on the sampled data to obtain a second prediction loss; An overall loss obtaining unit for obtaining an overall loss based on deep balancing processing and low-level contrast learning; A parameter updating unit for updating pre-trained parameters through backpropagation; A saving unit for saving the pre-trained parameters after training is completed; A second initialization unit for initializing a hierarchical vision transformer network using the saved parameters; A predicted image category unit for predicting the image category based on the hierarchical vision transformer network initialized with the saved parameters; The data hybrid sampling unit performs data sampling, including using a positive sampler, a negative sampler, and a confused data sampler to perform data sampling in the training dataset, and finally generating triple data composed of the above three samplers; The low-level feature contrast learning unit performs low-level feature contrast learning based on the sampled data to obtain a first prediction loss, including the following sub-steps: Extract features from the data of positive sampling and confused data sampling to obtain intermediate features; Mix the obtained intermediate features to obtain low-level mixed features; Pass the low-level mixed features through the subsequent network layers of the vision transformer network in sequence to obtain final mixed features; Send the final mixed features into a classifier for prediction to obtain a predicted probability distribution; Obtain a first prediction loss based on the predicted probability distribution; The deep feature balancing unit performs deep feature balancing processing based on the sampled data to obtain a second prediction loss, including the following steps: Extract features from the data of positive sampling and negative sampling to obtain deep features; Perform prediction on the deep features to obtain a prediction result, dynamically mix the prediction results to obtain a mixed prediction result and a mixed prediction probability distribution; In response to obtaining the mixed prediction probability distribution, smooth the predicted labels obtained by negative sampling to obtain smoothed labels; Obtain the prediction loss of the mixed prediction result based on the smoothed labels.

Images