A training method and an image classification method of an image classification model

Abstract

The application provides a training method and an image classification method of an image classification model. The training method comprises: obtaining an image-based multi-level classification training set; obtaining an image classification model comprising a feature extractor for extracting image features from an input image and a stacked multi-layer bidirectional logical tree; obtaining a preset adaptive intra-granularity difference learning network, the learning network comprising a plurality of learning matrices, each learning matrix being configured to map a first logical value output by a classifier in the same layer to obtain a second logical value; and training the image classification model and the learning network using the training set and a preset total loss function, wherein the total loss function is configured to guide the model to learn classification knowledge of each layer according to the first logical value and a classification label, and guide the model to learn similarity information between each class in the same layer and a real class for each layer according to the second logical value and a smoothed label of a sample, so as to better improve the performance of the classifier.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to the field of image classification, and more particularly to a training method of an image classification model and an image classification method. BACKGROUND

[0002] Fine-Grained Visual Categorization (FGVC) [1][2] is one of the long-term concerns and challenging research fields in the deep learning community. Fine-Grained Visual Categorization requires the model to capture subtle differences between categories. In recent years, due to the hierarchical label structure naturally existing in the category information, many studies have confirmed that it can significantly enhance the understanding and performance of the model in the FGVC task. Therefore, a large amount of work has begun to explore how to effectively integrate hierarchical information into the FGVC task. The main goal is to ensure that the model's prediction is consistent with the hierarchical structure of the data, and even if there are errors, the prediction result is as close as possible to its related category, thereby weakening the severity of the error.

[0003] However, when hierarchical semantic information is integrated, a key challenge is the problem of granularity competition in learning. In fact, coarse-grained semantics is usually easier for the model to distinguish than fine-grained semantics. Therefore, if the classifier for all granularities is not constrained, the learning process of the model is likely to be excessively affected by coarse-grained classification, resulting in the model only focusing on coarse-grained information and ignoring fine-grained information. In order to solve this problem, document [3] decouples the learning method to correspond the classifiers of different granularities to different granularity features defined by humans in advance, and uses fine-grained features to enhance the performance of coarse-grained classifiers. Document [4] proposes a soft balancing strategy to adjust the alignment of coarse-grained classification to fine-grained classification in the process of multi-granularity classification learning. However, these existing methods mainly emphasize the negative impact of coarse-grained classification on fine-grained learning, and do not better solve the problem of granularity competition.

[0004] It can be seen that in the multi-level classification scenario, the problem of granularity competition affecting the performance of fine-grained classification needs to be solved.

[0005] The information of the above references is as follows:

[0006] [1] Xin Jiang, Hao Tang, Junyao Gao, Xiaoyu Du, Shengfeng He and Zechao Li. Delving into Multimodal Prompting for Fine-Grained Visual Classification. In AAAI, pages 2570-2578, 2024.

[0007] [2] Shu-Lin Xu, Yifan Sun, Faen Zhang, Anqi Xu, Xiu-Shen Wei, Yi Yang. Hyperbolic Space with Hierarchical Margin Boosts Fine-Grained Learning from Coarse Labels. In NeurIPS, pages 71263-71274, 2023.

[0008] [3] Dongliang Chang, Kaiyue Pang, Yixiao Zheng, Zhanyu Ma, Yi-Zhe Song, and Jun You’re your “flamingo” is my “bird”: Fine-grained, or not. In CVPR, pages 11476-11485, 2021.

[0009] [4] Ashima Garg, Depanshu Sani, and Saket Anand. Learning hierarchy aware features for reducing mistake severity. In ECCV, pages 252-267, 2022.

[0010] It should be noted that: the background art is only used to introduce the related information of the present application, in order to help understand the technical scheme of the present application, but it does not mean that the related information must be prior art. The related information is submitted and disclosed together with the present application scheme, and in the absence of evidence showing that the related information has been disclosed before the filing date of the present application, the related information should not be regarded as prior art. SUMMARY

[0011] Therefore, the purpose of the present application is to overcome the defects of the above-mentioned prior art, and to provide a training method of image classification model and image classification method.

[0012] The object of the present application is achieved by the following technical solutions.

[0013] According to a first aspect of the present application, a method for training an image classification model is provided, comprising: obtaining a multi-level classification training set, the training set comprising: a plurality of samples, each sample comprising an image and a classification label, the classification label indicating a real class to which the image in the sample belongs in each level of a plurality of preset levels; obtaining an image classification model comprising a feature extractor for extracting image features from an input image and a stacked multi-layer bidirectional logical tree, wherein each layer of the bidirectional logical tree is a classifier, the input of the first layer classifier is the image features, the input of each subsequent layer classifier is the output of the previous layer classifier, and the output of each layer classifier is a first logical value used to determine the classification result of the layer; obtaining a preset adaptive intra-granularity difference learning network, wherein the learning network comprises a plurality of learning matrices, each learning matrix is used to map the first logical value output by the classifier in the same layer to obtain a second logical value; and training the image classification model and the learning network using the training set and a preset total loss function, wherein the total loss function is configured to guide the model to learn the classification knowledge of each layer according to the first logical value and the classification label, and to guide the model to learn the similarity information between each class in the same layer and the real class for each layer according to the second logical value and the smoothed label of the sample. The technical scheme of this embodiment can at least achieve the following beneficial technical effects: on the one hand, the present scheme constructs a classification learning framework based on a stacked multi-layer bidirectional logical tree, which can improve the priority of learning fine-grained features and promote the semantic information flow between the outputs of different levels (granularity) during training, so as to alleviate the granularity competition problem end-to-end by regarding the coarse-grained level error as auxiliary supervision signal for the fine-grained level; on the other hand, the total loss function of the present scheme not only learns the knowledge of multi-level classification, but also uses the output of the adaptive intra-granularity difference learning network and the preset smoothed label to guide the model to learn the similarity information between each class in the same layer and the real class for each layer, which can learn to perceive the subtle differences between different sub-classes in the same level, correct the distribution of the logical values output by the bidirectional logical tree, and thus better improve the performance of the classifier.

[0014] Optionally, in the multi-layer bidirectional logical tree, the classifiers closer to the feature extractor are used to classify the finer-grained levels in the multi-level classification. The technical scheme of this embodiment can at least achieve the following beneficial technical effects: the present application proposes a hierarchical classification learning framework based on a bidirectional logical tree, which adopts a method of sequentially constructing classifiers from fine-grained to coarse-grained, rather than constructing a series of classifiers in parallel based on the same input features. In this setting, the output of the fine-grained classifier serves as the input of the coarse-grained classifier, promoting the flow of hierarchical semantic information between different granularities, and also assisting in improving the learning of fine-grained classification with the label information of coarse-grained classification, thereby better improving the performance of the model.

[0015] Optionally, the smooth label corresponding to one sample is obtained by aggregating the distance between each two classes and the shortest common ancestor of the classification label of the sample, the weight vector of the row corresponding to the real class indicated in the classification label of the sample in the learning matrix of the learning network, and the classification label of the sample, wherein the weight vector is the weight for weighting the first logic value of each class when the learning network currently learns the second logic value of the real class. The technical scheme of this embodiment can at least achieve the following beneficial technical effects: on the basis of the original classification label, on the one hand, the attention degree of the real class learned by the learning network to the first logic value of each class is integrated, and on the other hand, the distance between each two classes and the shortest common ancestor is integrated, so that the smooth label in the form of non-unique hot code can be formed, the value of the class more similar to the real class in the smooth label is larger, and the model training is guided by the smooth label with the similarity information integrated, so that the performance of the model is improved.

[0016] Optionally, the total loss function is a weighted sum of a plurality of sub-loss functions, wherein the plurality of sub-loss functions include a first sub-loss function for calculating the loss between the first logic value of each layer and the classification label and a second sub-loss function for calculating the loss between the second logic value of each layer and the smooth label.

[0017] Optionally, the total loss function is:

[0018]

[0019] wherein h represents the hth layer, represents the first sub-loss function for calculating the loss between the first logic value of the hth layer and the classification label, represents the second sub-loss function for calculating the loss between the second logic value of the hth layer and the smooth label, and a represents the weighted value of . represents the weighted value of .

[0020] Optionally, the total loss function is:

[0021]

[0022] wherein h represents the hth layer, and λ h represents the hyperparameter set for each layer of the sub-loss, and the value of λ h set for the layer farther away from the trunk network is smaller. represents the first sub-loss function for calculating the loss between the first logic value of the hth layer and the classification label, represents the second sub-loss function for calculating the loss between the second logic value of the hth layer and the smooth label, and a represents the weighted value of . represents the weighted value of The weighted value. The technical solution of this embodiment can at least achieve the following beneficial technical effects: through λ h The supervision strength of the sub-loss function for classifiers at different levels is set hierarchically, and the λ of the layers further away from the backbone network (i.e., the coarser-grained classification layers) is increased. h The smaller the value, the less the model is affected by coarse-grained classification labels during training, reducing granularity competition and improving model performance.

[0023] Optionally, the second sub-loss function is:

[0024]

[0025] Where N represents the number of samples used in a training batch. Represents the smooth label of the i-th sample The components of the h-th layer, The learning matrix Δ represents the learning network. h The h-th layer The row weight vector, where β represents the weighting coefficient. This represents the h-th level sub-table in the shortest common ancestor distance table D constructed based on the label trees corresponding to the category labels. The number of rows, Let represent the second logical value output by layer h for the i-th sample, and Softmax(·) represent the Softmax function. This embodiment achieves at least the following beneficial technical effects: the combination of the second sub-loss function and the adaptive intra-granularity difference learning network helps the model learn subtle differences between categories and similarity to the true category within each layer (intra-granularity), thereby further improving model performance.

[0026] Optional, the smoothed label of the i-th sample The components of the h-th layer are:

[0027]

[0028] in, This represents the component of the h-th layer in the classification label of the i-th sample. Represents the learning matrix Δ h The hth layer The element corresponding to row j-th column, This represents the h-th level sub-table in distance table D. The element corresponding to the j-th column in row, exp′(·) represents the function used to calculate each element of the vector within the parentheses as a power of e and then reassemble the results into a vector in order, exp(·) represents the natural exponential function, γ represents the temperature coefficient, ∈ represents the weighting coefficient, C hThe total number of all categories of the hth layer is represented. The technical solution of this embodiment can achieve at least the following beneficial technical effects: the smooth label combines the original classification label, the attention degree of the real category learned by the learning network to the first logic value of each category, and the distance between the related category and the shortest common ancestor in the distance table, and can form a smooth label in the form of non-unique hot code, that is, the label value can be a value in [0, 1], instead of the form of 0 or 1. The label value of the category that is more similar to the real category in the smooth label is larger. In this way, on the basis of the classification label, the similarity information is additionally integrated to obtain the smooth label, and the model training is guided by the smooth label to improve the performance of the model.

[0029] According to a second aspect of the present application, an image classification method is provided, comprising: obtaining a trained image classification model trained by the method according to the first aspect; obtaining an image to be classified, and inputting the image to be classified into the trained image classification model to obtain the first logic value output by each layer classifier; and determining the classification result of each layer according to the first logic value output by each layer classifier.

[0030] According to a third aspect of the present application, a computer program product is provided, comprising computer programs / instructions, which, when executed by a processor, implement the steps of the method according to the first and / or second aspect.

[0031] According to a fourth aspect of the present application, an electronic device is provided, comprising: one or more processors; and a memory, wherein the memory is configured to store executable instructions; and the one or more processors are configured to implement the steps of the method according to the first and / or second aspect by executing the executable instructions. BRIEF DESCRIPTION OF DRAWINGS

[0032] The embodiments of the present application will be further described below with reference to the accompanying drawings, in which:

[0033] Figure 1 A flowchart of the training method of the image classification model according to the embodiments of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0035] As mentioned in the background section, under the multi-level classification scenario, the problem of granularity competition needs to be solved to affect the performance of fine-grained classification. Existing methods mainly emphasize the negative impact of coarse-grained classification on fine-grained learning, ignoring the key fact that coarse-grained classification can improve fine-grained learning. Therefore, it should be considered to improve the performance of fine-grained classification on the basis of avoiding granularity competition. The existing methods for fusing multi-granularity label semantic information to improve the performance of fine-grained classification usually have a limitation: developing independent hierarchical perception classifiers based on shared features, which makes the model tend to preferentially learn coarse-grained features, thereby hindering the learning of fine-grained features. In view of this, the method of the present application proposes a hierarchical classification learning framework based on a bidirectional logical tree, which adopts a method of sequentially constructing classifiers from fine-grained to coarse-grained, rather than constructing a series of classifiers in parallel based on the same input features. In this setting, the output of the fine-grained classifier is used as the input of the coarse-grained classifier, which promotes the flow of hierarchical semantic information between different granularities. In addition, the present application also introduces an adaptive intra-granularity difference learning network to enable the model to learn the subtle semantic differences (similarity information) between classes within the same granularity. Even if the model makes a classification error, the resulting classification result is more semantically close to the true class, thereby better improving the performance of the model.

[0036] To better illustrate the technical solutions of the present application, the following will be introduced from the aspects of sample setting, distance table, smoothed label, model structure, adaptive intra-granularity difference learning network, loss function, model training and application scenario.

[0037] I. Sample setting

[0038] The sample is a pre-collected image and a multi-level classification label annotated for it, which is used to train, verify and / or test the model.

[0039] According to an embodiment of the present application, the sample can come from a pre-collected multi-level classification dataset. The multi-level classification dataset includes: a plurality of samples, each sample including an image and a classification label, the classification label indicating the true class to which the image in the sample belongs in each layer of a preset plurality of layers. In the dataset, a certain sub-class of the hth layer of finer granularity only belongs to a single parent class of the h+1th layer. That is, the annotated hierarchical label has only a one-to-many relationship from coarse to fine, and a fine class does not belong to multiple coarse classes. The multi-level classification dataset can be an existing dataset, such as the public dataset FGVC-Aircraft, CIFAR100 or iNaturalist2019. Alternatively, the implementer can also collect images and annotate multi-level classification labels. The multi-level classification dataset can be divided into a training set, a verification set and a test set, or a training set and a test set.

[0040] It should be noted that multi-level classification labels are relative to single-level labels. Traditional single-level labels only label one level of category for an image, such as labeling an image as a cat, bird, or dog in an animal classification dataset. Multi-level classification labels, however, start from the root category and perform multiple levels of category labeling. For example, in the CIFAR100 dataset, for images of "beaver" and "table," the labeled classification labels should be: {beaver, aquatic mammal, aquatic organism, animal, organism} and {table, furniture, everyday item, man-made object, non-living object}, respectively. In the method of this invention, unless otherwise specified, the classification labels refer to multi-level classification labels (or hierarchical labels or multi-level labels).

[0041] II. Distance Table

[0042] To enable the model to better learn intra-layer (intra-granular) similarity information, the method of this invention also constructs a distance table by building a label tree based on the classification labels.

[0043] According to one embodiment of the present invention, the distance table D has a sub-table D for each layer h. h Sub-table D h The specification is N×N, where N is the total number of categories in the label space of this layer (that is, the total number of categories that the classifier of the corresponding layer needs to divide). Subtable D h The The element corresponding to the j-th column in the row, whose value is the element in the tag space. The distance (LCA) between the labels of class j and the shortest common ancestor (LCA) of the two classes is calculated. This is the distance between the labels of the two classes and their nearest common ancestor in the label tree. If the distance is considered to be the number of levels from the LCA to the labels of the two classes, then a preset distance value (e.g., 5) is used if there is no LCA. For example, in the finest level of the CIFAR100 dataset, the LCA of "table" and "chair" is "furniture," so the corresponding distance is 1; the LCA of "bird" and "beaver" is "animal," so the corresponding distance is 3; and the distance between "table" and "beaver" is 5 because there is no LCA. It should be understood that the above examples are for illustrative purposes only, and the specific distance values ​​can be adjusted according to the implementer's needs.

[0044] III. Smooth Labels

[0045] Smoothing labels are an improvement made to reduce granularity competition and to reduce the amount of information the model learns only from hard labels (i.e., classification labels). Since smoothing labels are added to the weight vector in the learning matrix, and this weight vector is updated along with the model during training, smoothing labels are also dynamically calculated using the weight vector in the latest learning matrix during each batch of training.

[0046] According to one embodiment of the present invention, the smooth label corresponding to a sample is obtained by aggregating the sample's classification label, the weight vector of the row in the learning matrix of the learning network corresponding to the true category indicated in the sample's classification label, and the distance between each pair of categories involved in the sample's classification label and the shortest common ancestor, wherein the weight vector is the weight of the first logical value of each category when the learning network calculates the second logical value of the true category.

[0047] According to one embodiment of the present invention, the smoothing label of the i-th sample The components of the h-th layer are:

[0048]

[0049] in, This represents the component of the h-th layer in the classification label of the i-th sample. Represents the learning matrix Δ h The hth layer The element corresponding to row j-th column, This represents the h-th layer in distance table D. The element corresponding to the j-th column in row, exp′(·) represents the function used to calculate each element of the vector within the parentheses as a power of e and then reassemble the results into a vector in order, exp(·) represents the natural exponential function, γ represents the temperature coefficient, ∈ represents the weighting coefficient, C h This represents the total number of categories in layer h. For example, in the CIFAR100 dataset, the total number of categories in layer 5 is 100, and the total number of categories in layer 4 is 20. The technical solution of this embodiment can achieve at least the following beneficial technical effects: the smooth label integrates the original classification label, the attention of the real category learned by the learning network to the first logical value of each category, and the distance between the related categories and the shortest common ancestor in the distance table, which can form a smooth label in the form of a non-one-hot code, that is, the label value can be a value in [0,1], rather than the form of 0 or 1. The smooth label has a larger label value for the category that is closer to the real category (similarity). In this way, similarity information is added on the basis of the classification label to obtain the smooth label. The smooth label guides the model training to improve the performance of the model.

[0050] It should be noted that practitioners in this field may also adjust the smoothing label of the i-th sample above. The expression for the h-th layer component can be adjusted, or alternative implementations with similar effects can be obtained based on this idea. For example:

[0051]

[0052] Where β represents The weighting coefficients; or

[0053]

[0054] wherein β represents a weighting coefficient of and γ represents a weighting coefficient of .

[0055] IV. Model structure

[0056] In order to better utilize coarse-grained label information to improve the performance of fine-grained classification of the model, the model structure is improved.

[0057] According to an embodiment of the present application, the image classification model comprises a feature extractor for extracting image features from an input image and a stacked multi-layer bidirectional logical tree, wherein each layer of the bidirectional logical tree is a layer of classifier, the input of the first layer of classifier is the image features, the input of each subsequent layer of classifier is the output of its previous layer of classifier, and the output of each layer of classifier is a first logical value used to determine the classification result of the layer. Among them, in the multi-layer bidirectional logical tree, the classifiers closer to the feature extractor are used to classify the finer-grained levels in the multi-level classification. The embodiment of the present application adopts a method of sequentially constructing classifiers from fine-grained to coarse-grained, rather than constructing a series of classifiers in parallel based on the same input features. In this setting, the output of the fine-grained classifier is used as the input of the coarse-grained classifier, which facilitates the flow of hierarchical semantic information between different granularities, and can utilize the coarse-grained label semantics to assist the learning process of the model for fine-grained label semantics, thereby improving the performance of the model.

[0058] Illustratively, the architecture of the multi-layer bidirectional logical tree means that the multi-level classifiers are sequentially connected in the order from fine to coarse:

[0059]

[0060] wherein, corresponding to the output of the h-1th layer of classifier, f h represents the hth layer of classifier, h∈[1,H] is the level. h=1 represents the finest-grained level.

[0061] Optionally, the feature extractor can use the existing neural network backbone network, such as the backbone network of ResNet50 or ViT-B model, or can be constructed by the implementer using convolutional layers, pooling layers, residual layers and attention layers or their combinations.

[0062] Preferably, the h-th layer classifier comprises a stacked batch normalization layer (BN), a linear layer (Linear), a batch normalization layer (BN) and an ELU activation function. Of course, the h-th layer classifier can also adopt other manners, such as: a stacked batch normalization layer (BN), a linear layer (Linear), a batch normalization layer (BN) and a ReLU activation function.

[0063] V. Adaptive intra-granularity difference learning network

[0064] The purpose of setting the adaptive intra-granularity difference learning network (hereinafter referred to as the learning network) is to specify the model to learn the subtle differences between different sub-classes in the same level, and to correct the output probability distribution of the bidirectional logical tree, so as to better improve the performance of the classifier.

[0065] Preferably, the learning network can be implemented by using a full connection layer without a bias term. Using a full connection layer without a bias term can reduce the influence of the bias term on the correlation learning between classes in the same layer, and improve the performance of the model. Of course, it is still feasible in principle to implement the learning network by using a full connection layer with a bias term, but the effect is slightly worse.

[0066] VI. Loss function

[0067] The loss function is a function used to specify the model training. In order to optimize the training effect, in addition to the conventional cross-entropy loss function (hereinafter referred to as the first sub-loss function), another smoothing loss function (hereinafter referred to as the second sub-loss function) is additionally set.

[0068] According to an embodiment of the present application, the total loss function is a weighted sum of a plurality of sub-loss functions, which includes a first sub-loss function for calculating the loss between the first logical value of each layer and the classification label, and a second sub-loss function for calculating the loss between the second logical value of each layer and the smoothing label.

[0069] The first sub-loss function and the second sub-loss function can be simply weighted and summed. According to an embodiment of the present application, the total loss function is:

[0070]

[0071] wherein h represents the h-th layer, represents the first sub-loss function for calculating the loss between the first logical value of the h-th layer and the classification label, represents the second sub-loss function for calculating the loss between the second logical value of the h-th layer and the smoothing label, and a represents the weighted value of represents the weighted value of

[0072] Since the category at the coarse-grained level is relatively easier for the model to accurately determine, the total loss function can be further improved to appropriately reduce the model's focus on the loss at the coarse-grained level, thereby optimizing the granularity competition and improving model performance. According to another embodiment of the present invention, the total loss function is:

[0073]

[0074] Where h represents the h-th layer, λ h This represents the hyperparameter set for the sub-loss of each layer, with λ set for layers further away from the backbone network. h The smaller the value, This represents the first sub-loss function used to calculate the loss between the first logistic value and the classification label at layer h. This represents the second sub-loss function for calculating the second logical value of the h-th layer and smoothing the inter-label loss, where α represents... The weighted value, express The weighted value. Preferably, λ h =exp(α·(1-h)) is the hyperparameter of the loss corresponding to each layer, where h represents the current layer number, and α is the hyperparameter used to control the supervision intensity of each layer. As h increases, the granularity becomes coarser, and λ... h The smaller the value of α, the more it controls the supervisory strength of the coarse-grained level. In experiments, the typical value range is 0.5 to 2. Considering that coarser-grained labels provide less help for fine-grained classification, the specific value of α is related to the quality of the coarse-grained labeling and can be set by the implementer as needed. The technical solution of this embodiment can achieve at least the following beneficial technical effects: through α... h The supervision strength of the sub-loss function for classifiers at different levels is set hierarchically, and the λ of the layers further away from the backbone network (i.e., the coarser-grained classification layers) is increased. h The smaller the value, the less the model is affected by coarse-grained classification labels during training, reducing granularity competition and improving model performance.

[0075] According to one embodiment of the present invention, the first sub-loss function is:

[0076]

[0077] Where n represents the number of samples used in a training batch. This represents the component of the h-th layer in the classification label of the i-th sample. This represents the first logical value output by the classifier at layer h for the i-th sample.

[0078] According to one embodiment of the present invention, the second sub-loss function is:

[0079]

[0080] wherein N represents the number of samples used in a training batch, represents the smoothed label of the i-th sample the component of the h-th layer in represents the learning matrix Δ of the learning network h the h-th layer in the weight vector of the h-th row, and β represents the weighting coefficient, represents the component of the h-th row in the shortest common ancestor distance table D constructed according to the label tree corresponding to the classification label represents the second logic value output by the learning network of the h-th layer for the i-th sample, and Softmax(·) represents the Softmax function. The technical scheme of this embodiment can at least achieve the following beneficial technical effects: the second sub-loss function and the adaptive intra-granularity difference learning network can help the model learn the subtle differences between categories and the similarity with the real category in each layer (intra-granularity), so as to further improve the performance of the model.

[0081] Seven, model training

[0082] According to an embodiment of the present application, a method for training an image classification model is provided, as shown in Figure 1 , comprising S1, S2, S3, S4. In order to better understand the present application, each step will be described in detail below in combination with specific embodiments.

[0083] Step S1: Obtain a multi-level classification training set.

[0084] According to an embodiment of the present application, the data set of the foregoing embodiment can be divided into a training set, a validation set and a test set in a ratio of 7:2:1 or 8:1:1, and the training set is used to perform the subsequent training process.

[0085] Step S2: Obtain an image classification model, which includes a feature extractor for extracting image features from an input image and a stacked multi-layer bidirectional logical tree, wherein each layer of the bidirectional logical tree is a classifier, the input of the first layer classifier is the image features, the input of each subsequent layer classifier is the output of the previous layer classifier, and the output of each layer classifier is a first logic value for determining the classification result of the layer.

[0086] According to an embodiment of the present application, the image classification model constructed in the above embodiment is obtained, and the trainable parameters of the model are initialized in a random initialization manner. In the subsequent process, the model is iteratively trained, i.e., each training is based on the model parameters obtained from the previous training.

[0087] ​Step S3: obtaining a preset adaptive intra-granularity difference learning network, wherein the learning network comprises a plurality of layers of learning matrices, and each layer of learning matrices is configured to map a first logic value output by a classifier in the same layer to obtain a second logic value.

[0088] According to an embodiment of the present application, the adaptive intra-granularity difference learning network constructed in the above embodiment is obtained, and the trainable parameters of the learning network are initialized in a random initialization manner. In the subsequent process, the learning network is iteratively trained, i.e., each training is based on the learning matrix of the learning network obtained in the previous training to update the learning matrix.

[0089] Step S4: training the image classification model and the learning network using the training set and the preset total loss function, wherein the total loss function is configured to guide the model to learn the classification knowledge of each layer according to the first logic value and the classification label, and guide the model to learn the similarity information of each class in the same layer to the real class according to the second logic value and the smoothed label of the sample.

[0090] According to an embodiment of the present application, the training process can use any total loss function of the foregoing embodiments to guide the training process of the model and the learning network.

[0091] Eight, application scenarios

[0092] According to an embodiment of the present application, an image classification method is provided, comprising: obtaining a trained image classification model trained by the training method of the image classification model according to the foregoing embodiments; obtaining an image to be classified, inputting the image to be classified into the trained image classification model to obtain a first logic value output by each layer of classifier; and determining a classification result of each layer according to the first logic value output by each layer of classifier. For example: inputting the first logic value of the hth layer into a Softmax function to obtain a confidence degree of the image to be classified belonging to each class of the hth layer, and taking the confidence degree of the image to be classified belonging to each class of the hth layer as the classification result. Or, selecting the class with the highest confidence degree as the classification result of the hth layer. Collecting the classification results of all layers, i.e., the total classification result.

[0093] In order to verify the effect of the method of the present application, the inventors have carried out the following comparative experiments:

[0094] Comparison method:

[0095] CE, CrossEntropy, refers to a method based on only a conventional cross-entropy loss function and an existing model.

[0096] HXE, reference: Bertinetto, Luca, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, and Nicholas A. Lord. "Making better mistakes: Leveraging class hierarchies with deep networks." In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pp. 12506-12515. 2020.

[0097] Soft-Labels, reference: Bertinetto, Luca, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, and Nicholas A. Lord. "Making better mistakes: Leveraging class hierarchies with deep networks." In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pp. 12506-12515. 2020.

[0098] Flamingo, reference: Chang, Dongliang, Kaiyue Pang, Yixiao Zheng, Zhanyu Ma, Yi-Zhe Song, and Jun Guo. "Your"flamingo"is my"bird": Fine-grained, or not." In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 11476-11485. 2021.

[0099] CRM, Ref: Karthik, Shyamgopal, Ameya Prabhu, Puneet K. Dokania, and Vineet Gandhi. "No Cost Likelihood Manipulation at Test Time for Making Better Mistakes in Deep Networks." In International Conference on Learning Representations.

[0100] HAF, Ref: Garg, Ashima, Depanshu Sani, and Saket Anand. "Learning hierarchy aware features for reducing mistake severity." In European Conference on Computer Vision, pp. 252-267. Cham: Springer Nature Switzerland, 2022.

[0101] HAFrame, Ref: Liang, Tong, and Jim Davis. "Inducing neural collapse to a fixed hierarchy-aware frame for reducing mistake severity." In Proceedings of the IEEE / CVF International Conference on Computer Vision, pp. 1443-1452. 2023.

[0102] HiE, Ref: Jain, Kanishk, Shyamgopal Karthik, and Vineet Gandhi. "Test-time amendment with a coarse classifier for fine-grained classification." Advances in Neural Information Processing Systems 36 (2024).

[0103] The total loss function used in the method of the present application during the experiment:

[0104]

[0105] The smooth label calculation method used in the method of the application during the experiment is as follows:

[0106]

[0107] The data set used in the experiment: The experiment was carried out on three public data sets FGVC-Aircraft, CIFAR100 and iNaturalist2019.

[0108] Experimental indexes: Mistakes severity, Hier dist@5, Hier dist@20 and Accuracy. Among them, Mistakes severity refers to the average of the shortest common ancestor distance (LCA) distance between the predicted error sample label and the true value label, Hier dist@k refers to the average of the LCA distance between the Top-k prediction of the model and the true value label, and Accuracy refers to the ratio of the samples predicted correctly by the model to all samples.

[0109] The experimental results are shown in Tables 1-3.

[0110] Table 1: Performance of each method on FGVC-Aircraft data set

[0111]

[0112] Table 2: Performance of each method on CIFAR100 data set

[0113]

[0114] Table 3: Performance of each method on iNaturalist2019 data set

[0115]

[0116]

[0117] It can be seen that in the three public data sets FGVC-Aircraft, CIFAR100 and iNaturalist2019, the method of the application is optimal in most of the four indexes Mistakes severity, Hier dist@5, Hier dist@20 and Accuracy, which verifies the effectiveness of the method of the application.

[0118] In general, the core objective of the present application aims to solve the potential competition problem between fine-grained classification and coarse-grained classification in the multi-grained classification framework, and then explore a strategy to efficiently integrate the information advantage of coarse-grained classification as an auxiliary means to improve the performance of fine-grained classification. In the model learning process, the previous method only emphasizes the negative impact of coarse-grained classification on fine-grained learning, while ignoring the key fact that coarse-grained classification can assist to improve fine-grained learning. In order to solve this problem, the inventors propose a new hierarchical classification learning framework based on bidirectional logical tree (referred to as BiLT), which aims to effectively utilize hierarchical label information to construct hierarchical perception classifiers to alleviate the granularity competition problem. The hierarchical classification learning framework based on bidirectional logical tree achieves this goal by constructing classifiers in order from fine-grained to coarse-grained, and the input of the coarse-grained classifier comes from the output of the finer-grained classifier. In addition, an adaptive internal granularity difference learning method (referred to as AIGDL) is developed (mainly based on the learning network and the second sub-loss function in the total loss function) to better exploit the subtle semantic differences between sub-classes under the same granularity and improve the performance of the model.

[0119] It should be noted that although the above describes the steps in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of these steps can be performed concurrently or even in a different order, as long as the desired function can be achieved.

[0120] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0121] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch cards or punched tape, and any suitable combination of the foregoing. A non-transitory, computer-readable storage medium does not include a signal.

[0122] Having described various embodiments of the application, it is to be understood that the above description is meant not to limit and not to encompass all of the possible embodiments covered by the claims. Many modifications and variations of this application can be apparent to those of ordinary skill in the art without departing from the spirit and scope of the described embodiments. It is intended that the scope of the application should only be limited by the appended claims.

Claims

1. A training method for an image classification model, characterized in that, include: Obtain a multi-level classification training set, which includes: multiple samples, each sample including an image and a classification label, the classification label indicating the true category to which the image in the sample belongs in each of the preset multiple levels; An image classification model is obtained, which includes a feature extractor for extracting image features from an input image and a stacked multi-level bidirectional logic tree. Each layer of the bidirectional logic tree is a classifier of one layer. The input of the first layer classifier is the image features, and the input of each subsequent layer classifier is the output of the previous layer classifier. The output of each layer classifier is a first logical value used to determine the classification result of that layer. In the multi-level bidirectional logic tree, the classifier closer to the feature extractor is used to classify the finer-grained levels in the multi-level classification. Obtain a preset adaptive intragranular differential learning network, wherein the learning network includes multiple learning matrices, and each learning matrix is ​​used to map the first logical value output by the classifier in the same layer to obtain a second logical value; The training set and a preset total loss function are used to guide the training of the image classification model and the learning network. The total loss function is configured to guide the model to learn the classification knowledge of each layer based on a first logical value and classification label, and to guide the model to learn the similarity information between each category in the same layer and the real category based on a second logical value and the smooth label of the sample.

2. The method according to claim 1, characterized in that, The smooth label corresponding to a sample is obtained by aggregating the sample's classification label, the weight vector of the row in the learning matrix of the learning network that corresponds to the true class indicated in the sample's classification label, and the distance between each pair of classes involved in the sample's classification label and the shortest common ancestor. The weight vector is the weight of the first logical value of each class when the learning network calculates the second logical value of the true class.

3. The method according to claim 2, characterized in that, The total loss function is a weighted sum of multiple sub-loss functions, including a first sub-loss function for calculating the first logical value and the loss between classification labels for each layer, and a second sub-loss function for calculating the second logical value and the loss between smoothing labels for each layer.

4. The method according to claim 3, characterized in that, The total loss function is: in, Indicates the first layer, To find the first... The first sub-loss function between the first logical value of the layer and the classification label. To find the first... The second logical value of the layer and the second sub-loss function of the smooth label loss, express The weighted value, express The weighted value.

5. The method according to claim 3, characterized in that, The total loss function is: in, Indicates the first layer, This represents the hyperparameters set for the sub-loss of each layer, with the settings increasing as the layer moves further away from the backbone network. The smaller the value, To find the first... The first sub-loss function between the first logical value of the layer and the classification label. To find the first... The second logical value of the layer and the second sub-loss function of the smooth label loss, express The weighted value, express The weighted value.

6. The method according to claim 4 or 5, characterized in that, The second sub-loss function is: in, This indicates the number of samples used in a training batch. Indicates the first Smooth label for each sample The Middle The components of the layer, = , The learning matrix representing the learning network The first in layer The weight vector of the row, Indicates the weighting coefficient. This represents the shortest common ancestor distance table constructed based on the label trees corresponding to the category labels. The Middle Sub-tables The number of rows, , Indicates the first Layer to the first The second logical value output by each sample express function.

7. The method according to claim 6, characterized in that, No. Smooth label for each sample The Middle The components of the layer are: in, Indicates the first The classification label of the sample is the first The components of the layer, Representation of the learning matrix The layer Line number The elements corresponding to the column, Distance table The Middle Sub-tables Line number The elements corresponding to the column, This represents a function that calculates each element of the vector within parentheses as a power of e, and then reassembles the results into a vector in sequence. This represents the natural exponential function. Indicates the temperature coefficient. Indicates the weighting coefficient. Indicates the first The total number of all categories in the layer.

8. An image classification method, comprising: Obtain a trained image classification model obtained by the method according to any one of claims 1-7; Obtain the image to be classified, input the image to be classified into the trained image classification model, and obtain the first logical value output by each layer of the classifier; The classification result of each layer is determined based on the first logical value output by each layer's classifier.

9. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1-8.

11. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1-8 by executing the executable instructions.

Images