Training method of image classification model and image classification method

Abstract

The invention provides a training method of an image classification model and an image classification method, and the training method comprises the steps: obtaining a multi-level classification training set based on an image; obtaining an image classification model comprising a feature extractor for extracting image features from the input image and a stacked multilayer bidirectional logic tree; a preset self-adaptive intra-granularity difference learning network is obtained, the learning network comprises multiple layers of learning matrixes, and each layer of learning matrix is used for mapping the first logic values output by the classifiers of the same layer to obtain second logic values; using the training set and a preset total loss function to guide an image classification model and a learning network to train, the total loss function being configured to guide the model to learn classification knowledge of each layer according to a first logic value and a classification label, and according to the second logic value and the smooth label of the sample, guiding the model to learn similarity information of each category and a real category in the same layer for each layer, thereby better improving the performance of the classifier.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, specifically to the field of image classification, and more specifically to a training method for an image classification model and an image classification method. Background Art

[0002] Fine-Grained Visual Categorization (FGVC) [1][2] It is one of the long-term and challenging research areas of the deep learning community. Fine-grained image classification requires the model to be able to capture the subtle differences between categories. In recent years, many studies have shown that the hierarchical label structure naturally exists in the category information, which can significantly enhance the understanding and performance of the model in the FGVC task. Therefore, a lot of work has begun to explore how to effectively integrate hierarchical information into the FGVC task. The main goal is to ensure that the model's predictions are consistent with the hierarchical structure of the data. Even if an error occurs, the prediction result is as close as possible to its related category, thereby reducing the severity of the error.

[0003] However, when hierarchical semantic information is integrated, a key challenge is the problem of granularity competition in learning. In fact, coarse-grained semantics are usually easier for models to distinguish than fine-grained semantics. Therefore, if there are no constraints on classifiers of all granularities, the learning process of the model is likely to be overly affected by coarse-grained classification, causing the model to only focus on coarse-grained information and ignore fine-grained information. To solve this problem, reference [3] uses a decoupled learning method to correspond classifiers of different granularities to different granularity features defined in advance, and uses fine-grained features to enhance the performance of coarse-grained classifiers. Reference [4] proposes a soft balancing strategy to adjust the alignment of coarse-grained classification to fine-grained classification during multi-granularity classification learning. However, these existing methods mainly emphasize the negative impact of coarse-grained classification on fine-grained learning, and do not solve the problem of granularity competition well.

[0004] It can be seen that in multi-level classification scenarios, it is necessary to solve the problem of granularity competition affecting the performance of fine-grained classification.

[0005] The information of the above references is as follows:

[0006] [1] Xin Jiang, Hao Tang, Junyao Gao, Xiaoyu Du, Shengfeng He and Zechao Li. Delving into Multimodal Prompting for Fine-Grained Visual Classification. In AAAI, pages 2570–2578, 2024.

[0007] [2] Shu-Lin Xu, Yifan Sun, Faen Zhang, Anqi Xu, Xiu-Shen Wei, Yi Yang. Hyperbolic Space with Hierarchical Margin Boosts Fine-Grained Learning from Coarse Labels. In NeurIPS, pages 71263–71274, 2023.

[0008] [3] Dongliang Chang, Kaiyue Pang, Yixiao Zheng, Zhanyu Ma, Yi-Zhe Song, and Jun You're your "flamingo" is my "bird": Fine-grained, or not. In CVPR, pages 11476-11485, 2021.

[0009] [4]Ashima Garg, Depanshu Sani, and Saket Anand. Learning hierarchy awarefeatures for reducing mistake severity. In ECCV, pages 252-267, 2022.

[0010] It should be noted that this background technology is only used to introduce the relevant information of the present invention to help understand the technical solution of the present invention, but it does not mean that the relevant information is necessarily the prior art. The relevant information is submitted and disclosed together with the present invention solution. If there is no evidence that the relevant information has been disclosed before the application date of the present invention, the relevant information shall not be regarded as the prior art. Summary of the invention

[0011] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide an image classification model training method and an image classification method.

[0012] The objective of the present invention is achieved through the following technical solutions:

[0013] According to a first aspect of the present invention, a method for training an image classification model is provided, comprising: obtaining a multi-level classification training set, the training set comprising: a plurality of samples, each sample comprising an image and a classification label, the classification label indicating the true category to which the image in the sample belongs at each of the preset multiple levels; obtaining an image classification model, which comprises a feature extractor for extracting image features from an input image and a stacked multi-level bidirectional logic tree, wherein each layer of the bidirectional logic tree is a classifier of a layer, the input of the first layer classifier is the image feature, the input of each subsequent layer classifier is the output of the previous layer classifier, and the output of each layer classifier is a feature extractor for extracting image features from an input image, wherein the first layer classifier is an ... A first logical value for determining the classification result of the layer; obtaining a preset adaptive intra-granularity difference learning network, wherein the learning network includes multiple layers of learning matrices, and each layer of the learning matrix is ​​used to map the first logical value output by the classifier of the same layer to obtain a second logical value; using the training set and the preset total loss function to guide the image classification model and the learning network for training, wherein the total loss function is configured to guide the model to learn the classification knowledge of each layer according to the first logical value and the classification label, and to guide the model to learn the similarity information between each category in the same layer and the true category according to the second logical value and the smooth label of the sample. The technical solution of this embodiment can at least achieve the following beneficial technical effects: on the one hand, this solution constructs a classification learning framework based on stacked multi-layer bidirectional logic trees, which can improve the priority of learning fine-grained features and promote the flow of semantic information between outputs of different levels (granularities) during training, so as to regard coarse-grained level errors as auxiliary supervision signals for fine-grained levels, thereby alleviating the granularity competition problem end-to-end; on the other hand, the total loss function of this solution not only learns the knowledge of multi-level classification, but also uses the output of the adaptive intra-granularity difference learning network and the preset smooth label guidance model to learn the similarity information between each category and the real category in the same layer for each layer. By learning to perceive the subtle differences between different subclasses at the same level, the distribution of the logical values ​​output by the bidirectional logic tree can be corrected, thereby better improving the performance of the classifier.

[0014] Optionally, in a multi-layer bidirectional logic tree, the classifier that is closer to the feature extractor is used to classify the finer-grained levels in the multi-level classification. The technical solution of this embodiment can at least achieve the following beneficial technical effects: The present invention proposes a hierarchical classification learning framework based on a bidirectional logic tree, which adopts a method of sequentially constructing classifiers from fine-grained to coarse-grained, rather than constructing a series of classifiers based on the same input features in parallel. In this setting, the output of the fine-grained classifier serves as the input of the coarse-grained classifier, which promotes the flow of hierarchical semantic information between different granularities, and can also assist in improving the learning of fine-grained classification with the help of the label information of the coarse-grained classification, thereby better improving the performance of the model.

[0015] Optionally, a smooth label corresponding to a sample is obtained by aggregating the classification label of the sample, the weight vector of the row corresponding to the real category indicated in the classification label of the sample in the learning matrix of the learning network, and the distance between each two categories involved in the classification label of the sample and the shortest common ancestor, wherein the weight vector is the weight of the first logical value of each category when calculating the second logical value of the real category currently learned by the learning network. The technical solution of this embodiment can at least achieve the following beneficial technical effects: on the basis of the original classification label, on the one hand, the attention of the real category learned by the learning network to the first logical value of each category is integrated, and on the other hand, the distance between each two categories and the shortest common ancestor is integrated to form a smooth label in the form of a non-unique hot code. The closer (similarity) the category in the smooth label is to the real category, the larger the value is, so as to use the smooth label incorporating similarity information to guide model training and improve model performance.

[0016] Optionally, the total loss function is a weighted sum of multiple sub-loss functions, including a first sub-loss function for calculating the first logical value of each layer and the loss between classification labels, and a second sub-loss function for calculating the second logical value of each layer and the loss between smoothed labels.

[0017] Optionally, the total loss function is:

[0018]

[0019] Among them, h represents the hth layer, It represents the first sub-loss function for finding the loss between the first logical value and the classification label of the h-th layer. It represents the second sub-loss function for finding the second logical value of the h-th layer and the loss between smooth labels, and α represents The weighted value of express The weighted value of .

[0020] Optionally, the total loss function is:

[0021]

[0022] Among them, h represents the hth layer, λ h Represents the hyperparameters set for the sub-loss of each layer, and the λ set for the layer farther away from the backbone network h The smaller the value, It represents the first sub-loss function for finding the loss between the first logical value and the classification label of the h-th layer. It represents the second sub-loss function for finding the second logical value of the h-th layer and the loss between smooth labels, and α represents The weighted value of express The technical solution of this embodiment can at least achieve the following beneficial technical effects: h The supervision strength of the sub-loss function of the classifiers at different levels is set hierarchically, and the λ of the layer farther away from the backbone network (i.e. the coarser classification layer) is set. h The smaller the value is set, the less likely the model will be overly affected by coarse-grained classification labels during training, which can reduce granularity competition problems and improve model performance.

[0023] Optionally, the second sub-loss function is:

[0024]

[0025] Where N represents the number of samples used in a training batch. represents the smoothed label of the i-th sample The h-th layer of the component, The learning matrix Δ represents the learning network h The hth layer in The weight vector of the row, β represents the weight coefficient, Represents the h-th level subtable in the shortest common ancestor distance table D constructed according to the label tree corresponding to the classification label The weight of the line, represents the second logical value output by the hth layer for the i-th sample, and Softmax(·) represents the Softmax function. The technical solution of this embodiment can at least achieve the following beneficial technical effects: the second sub-loss function and the adaptive intra-granularity difference learning network can help the model learn the subtle differences between categories and the similarity with the real category in each layer (intra-granularity) to further improve the performance of the model.

[0026] Optional, smoothed label for the ith sample The components of the hth layer in are:

[0027]

[0028] in, represents the h-th layer component of the classification label of the i-th sample, Denotes the learning matrix Δ h The hth layer The element corresponding to the jth column of the row, Represents the h-th level sub-table in the distance table D The element corresponding to the jth column of the row, exp′(·) represents the function used to treat each element of the vector in the brackets as the power calculation result of e and then reorganize it into a vector in order, exp(·) represents the natural exponential function, γ represents the temperature coefficient, ∈ represents the weighting coefficient, C hRepresents the total number of all categories in the hth layer. The technical solution of this embodiment can at least achieve the following beneficial technical effects: the smoothed label integrates the original classification label, the attention of the real category learned by the learning network to the first logical value of each category, and the distance between the relevant category and the shortest common ancestor in the distance table, and can form a smoothed label in the form of a non-unique hot code, that is, the label value can be a numerical value in [0,1], rather than in the form of 0 or 1. The closer (similarity) the category in the smoothed label is to the real category, the larger the label value is. In this way, on the basis of the classification label, additional similarity information is incorporated to obtain a smoothed label, and the model training is guided by the smoothed label to improve the performance of the model.

[0029] According to a second aspect of the present invention, there is provided an image classification method, comprising: obtaining a trained image classification model trained according to the method described in the first aspect; obtaining an image to be classified, inputting the image to be classified into the trained image classification model, and obtaining a first logical value output by each layer of the classifier; and determining the classification result of each layer according to the first logical value output by each layer of the classifier.

[0030] According to a third aspect of the present invention, there is provided a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method described in the first and / or second aspect.

[0031] According to a fourth aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the method described in the first and / or second aspects by executing the executable instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0033] Figure 1 4 is a flowchart of a method for training an image classification model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0035] As mentioned in the background technology section, in multi-level classification scenarios, it is necessary to solve the problem that granularity competition affects the performance of fine-grained classification. Existing methods mainly emphasize the negative impact of coarse-grained classification on fine-grained learning, ignoring the key fact of how coarse-grained classification improves fine-grained learning. Therefore, it should be considered to improve the performance of fine-grained classification on the basis of avoiding granularity competition. Existing methods for fusing multi-granularity label semantic information to improve fine-grained classification performance usually have a limitation: developing independent hierarchical perception classifiers based on shared features makes the model tend to learn coarse-grained features first, thereby hindering the learning of fine-grained features. In response to this, the method of the present invention proposes a hierarchical classification learning framework based on a bidirectional logic tree, which adopts a method of sequentially constructing classifiers from fine-grained to coarse-grained, rather than constructing a series of classifiers based on the same input features in parallel. In this setting, the output of the fine-grained classifier is used as the input of the coarse-grained classifier, which promotes the flow of hierarchical semantic information between different granularities. In addition, the present invention also introduces an adaptive intra-granularity difference learning network to enable the model to learn subtle semantic differences (similarity information) between categories within the same granularity. Even if the model classification is wrong, the obtained classification result is semantically closer to the true category, thereby better improving the performance of the model.

[0036] To better illustrate the technical solution of the present invention, the following introduces sample setting, distance table, smooth label, model structure, adaptive intra-granularity difference learning network, loss function, model training and application scenarios.

[0037] 1. Sample settings

[0038] Samples are pre-collected images and their multi-level classification labels, which are used to train, validate and / or test models.

[0039] According to one embodiment of the present invention, the sample may come from a pre-collected multi-level classification dataset. The multi-level classification dataset includes: multiple samples, each sample includes an image and a classification label, and the classification label indicates the true category to which the image in the sample belongs in each of the preset multiple levels. In the dataset, a subcategory of the h-th layer with finer granularity only belongs to a single parent category of the h+1-th layer. That is, the annotated hierarchical labels have only a one-to-many relationship from coarse to fine, and there will not be a fine category belonging to multiple coarse categories. The multi-level classification dataset may be an existing dataset, such as: the public dataset FGVC-Aircraft, CIFAR100 or iNaturalist2019. Alternatively, the implementer may also collect images and annotate multi-level classification labels on his own. The multi-level classification dataset may be divided into a training set, a validation set and a test set, or into a training set and a test set.

[0040] It should be noted that multi-level classification labels are relative to single-level labels. Traditional single-level labels only label one level of categories for an image, such as in an animal classification dataset: an image is labeled with a cat, bird, or dog label. In multi-level classification labels, multiple levels of category labeling will be performed starting from the root category. For example: in the CIFAR100 dataset, for the two categories of images "beaver" and "table", the labeled classification labels should be: {beaver, aquatic mammal, aquatic organism, animal, organism}, {table, furniture, daily necessities, man-made objects, non-biological objects}. In the method of the present invention, unless otherwise specified, the classification label defaults to the multi-level classification label (or hierarchical label or multi-level label).

[0041] 2. Distance table

[0042] In order to enable the model to better learn the similarity information within the layer (within the granularity), the method of the present invention also constructs a label tree based on the classification labels to construct a distance table.

[0043] According to an embodiment of the present invention, the distance table D is provided with a sub-table D for each layer h. h , subtable D h The specification is N×N, where N is the total number of categories in the label space of this layer (that is, the total number of categories that the classifier of the corresponding layer needs to divide). Represents subtable D h No. The element corresponding to the jth column of the row has the value of the first column in the label space. The distance between the label of the class and the jth class and the shortest common ancestor (LCA). That is, the distance between the labels of the two categories and the nearest common ancestor in the label tree. Assuming that the number of layers from the shortest common ancestor to the labels of the two categories is regarded as the distance, if there is no shortest common ancestor, it is regarded as a preset distance value (such as 5). For example: in the finest level of the CIFAR100 dataset, the shortest common ancestor of the table and the chair is furniture, then the corresponding distance is 1; the shortest common ancestor of the bird and the beaver is an animal, then the corresponding distance is 3, and the table and the beaver have no shortest common ancestor, then the distance is 5. It should be understood that the above examples are only for illustration, and the specific value of the distance can also be adjusted according to the needs of the implementer.

[0044] 3. Smooth Labels

[0045] Smoothing labels is an improvement made to reduce granularity competition and reduce the information contained in the model learning only hard labels (i.e. classification labels). Since smoothing labels will be added to the weight vector in the learning matrix, the weight vector will be updated during the model training process. Therefore, smoothing labels will also be dynamically calculated using the weight vector in the latest learning matrix during each batch training.

[0046] According to one embodiment of the present invention, a smoothed label corresponding to a sample is obtained by aggregating the classification label of the sample, the weight vector of the row corresponding to the true category indicated in the classification label of the sample in the learning matrix of the learning network, and the distance between each two categories involved in the classification label of the sample and the shortest common ancestor, wherein the weight vector is the weight of the first logical value of each category when calculating the second logical value of the true category currently learned by the learning network.

[0047] According to one embodiment of the present invention, the smoothed label of the i-th sample The components of the hth layer in are:

[0048]

[0049] in, represents the h-th layer component of the classification label of the i-th sample, Denotes the learning matrix Δ h The hth layer The element corresponding to the jth column of the row, Indicates the hth layer in the distance table D The element corresponding to the jth column of the row, exp′(·) represents the function used to treat each element of the vector in the brackets as the power calculation result of e and then reorganize it into a vector in order, exp(·) represents the natural exponential function, γ represents the temperature coefficient, ∈ represents the weighting coefficient, C h Represents the total number of all categories in the hth layer. For example: in the CIFAR100 dataset, the total number of categories in the 5th layer is 100, and the total number of categories in the 4th layer is 20. The technical solution of this embodiment can at least achieve the following beneficial technical effects: the smoothed label integrates the original classification label, the attention of the real category learned by the learning network to the first logical value of each category, and the distance between the relevant category and the shortest common ancestor in the distance table, and can form a smoothed label in the form of a non-unique hot code, that is, the label value can be a numerical value in [0,1], rather than in the form of 0 or 1. The closer (similarity) the category in the smoothed label is to the real category, the larger the label value is. In this way, on the basis of the classification label, additional similarity information is incorporated to obtain a smoothed label, and the model training is guided by the smoothed label to improve the performance of the model.

[0050] It should be noted that practitioners in this field may also use the smoothing label of the above i-th sample as The expression of the component of the hth layer in the example is adjusted or an alternative embodiment with similar effect is obtained based on this idea. For example:

[0051]

[0052] Among them, β represents or

[0053]

[0054] Among them, β represents The weighting coefficient of The weighting coefficient of .

[0055] IV. Model Structure

[0056] In order to better utilize coarse-grained label information to improve the performance of fine-grained classification of the model, the present invention improves the model structure.

[0057] According to one embodiment of the present invention, an image classification model includes a feature extractor for extracting image features from an input image and a stacked multi-layer bidirectional logic tree, wherein each layer of the bidirectional logic tree is a classifier of a layer, the input of the first layer classifier is the image feature, the input of each subsequent layer classifier is the output of the classifier of the previous layer, and the output of each layer classifier is a first logical value for determining the classification result of the layer. Among them, in the multi-layer bidirectional logic tree, the classifier closer to the feature extractor is used to classify the finer-grained levels in the multi-level classification. The embodiment of the present invention adopts a method of sequentially constructing classifiers from fine granularity to coarse granularity, rather than constructing a series of classifiers based on the same input features in parallel. In this setting, the output of the fine-grained classifier is used as the input of the coarse-grained classifier, which promotes the flow of hierarchical semantic information between different granularities, and the coarse-grained label semantics can be used to assist the supervision model in the learning process of the fine-grained label semantics, thereby improving the model performance.

[0058] Schematically, the architecture of a multi-layer bidirectional logic tree refers to the multi-level classifiers being connected in order from fine to coarse:

[0059]

[0060] in, Corresponding to the h-1th layer classifier output, f h represents the h-th layer classifier, h∈[1,H] is the level. h=1 represents the finest granularity layer.

[0061] Optionally, the feature extractor can adopt the backbone network of an existing neural network, such as the backbone network of ResNet50 or ViT-B model, or can be constructed by the implementer using convolutional layers, pooling layers, residual layers and attention layers or a combination thereof.

[0062] Preferably, the h-th layer classifier includes a stacked batch normalization layer (BN), a linear layer (Linear), a batch normalization layer (BN) and an ELU activation function. Of course, the h-th layer classifier may also be implemented in other ways, such as a stacked batch normalization layer (BN), a linear layer (Linear), a batch normalization layer (BN) and a ReLU activation function.

[0063] 5. Adaptive intra-granular difference learning network

[0064] The purpose of setting up an adaptive intra-granularity difference learning network (hereinafter referred to as the learning network) is to specify that the model learns to perceive the subtle differences between different subclasses at the same level and to modify the output probability distribution of the bidirectional logic tree to better improve the performance of the classifier.

[0065] Preferably, the learning network can be implemented using a fully connected layer without bias terms. Using a fully connected layer without bias terms can reduce the impact of bias terms on the correlation learning between categories in the same layer and improve the performance of the model. Of course, it is still feasible in principle to use a fully connected layer with bias terms to implement the learning network, but the effect is slightly worse.

[0066] 6. Loss Function

[0067] The loss function is a function used to specify model training. In order to optimize the training effect, the present invention sets another smoothing loss function (hereinafter referred to as the second sub-loss function) in addition to the conventional cross entropy loss function (hereinafter referred to as the first sub-loss function).

[0068] According to one embodiment of the present invention, the total loss function is a weighted sum of multiple sub-loss functions, including a first sub-loss function for calculating the first logical value of each layer and the loss between classification labels, and a second sub-loss function for calculating the second logical value of each layer and the loss between smoothed labels.

[0069] The first sub-loss function and the second sub-loss function can be simply weighted summed. According to one embodiment of the present invention, the total loss function is:

[0070]

[0071] Among them, h represents the hth layer, It represents the first sub-loss function for finding the loss between the first logical value and the classification label of the h-th layer. It represents the second sub-loss function for finding the second logical value of the h-th layer and the loss between smooth labels, and α represents The weighted value of express The weighted value of .

[0072] Since the categories at the coarse-grained level are relatively easier for the model to accurately judge, further improvements to the total loss function can be considered to appropriately reduce the model's attention to the loss at the coarse-grained level, so as to optimize the granularity competition and improve the performance of the model. According to another embodiment of the present invention, the total loss function is:

[0073]

[0074] Among them, h represents the hth layer, λ h Represents the hyperparameters set for the sub-loss of each layer, and the λ set for the layer farther away from the backbone network h The smaller the value, It represents the first sub-loss function for finding the loss between the first logical value and the classification label of the h-th layer. It represents the second sub-loss function for finding the second logical value of the h-th layer and the loss between smooth labels, and α represents The weighted value of express The weighted value of . Preferably, λ h =exp(α·(1-h)) is the hyperparameter of the loss corresponding to each layer, h represents the current number of layers, and α is a hyperparameter used to control the supervision strength of each layer. As h increases, the granularity becomes coarser, and λ h The smaller α is, the stronger the supervision of the coarse-grained level is controlled. In the experiment, the general value range is 0.5 to 2. Considering that the coarser-grained labels are less helpful for fine-grained classification, the specific value of α is related to the quality of the coarse-grained label marking and can be set by the implementer as needed. The technical solution of this embodiment can at least achieve the following beneficial technical effects: h The supervision strength of the sub-loss function of the classifiers at different levels is set hierarchically, and the λ of the layer farther away from the backbone network (i.e. the coarser classification layer) is set. h The smaller the value is set, the less likely the model will be overly affected by coarse-grained classification labels during training, which can reduce granularity competition problems and improve model performance.

[0075] According to one embodiment of the present invention, the first sub-loss function is:

[0076]

[0077] Where n represents the number of samples used in a training batch. represents the h-th layer component of the classification label of the i-th sample, Represents the first logical value output by the classifier of the h-th layer for the i-th sample.

[0078] According to one embodiment of the present invention, the second sub-loss function is:

[0079]

[0080] Where N represents the number of samples used in a training batch. represents the smoothed label of the i-th sample The h-th layer of the The learning matrix Δ represents the learning network h The hth layer in The weight vector of the row, β represents the weight coefficient, Represents the hth layer in the shortest common ancestor distance table D constructed based on the label tree corresponding to the classification label The weight of the line, represents the second logical value output by the learning network of the hth layer for the i-th sample, and Softmax(·) represents the Softmax function. The technical solution of this embodiment can at least achieve the following beneficial technical effects: the second sub-loss function and the adaptive intra-granularity difference learning network can help the model learn the subtle differences between categories and the similarity with the real category in each layer (intra-granularity) to further improve the performance of the model.

[0081] 7. Model Training

[0082] According to one embodiment of the present invention, a method for training an image classification model is provided. Figure 1 , including: S1, S2, S3, S4. In order to better understand the present invention, each step is described in detail below in conjunction with specific embodiments.

[0083] Step S1: Obtain a multi-level classification training set.

[0084] According to one embodiment of the present invention, the data set in the aforementioned embodiment can be divided into a training set, a validation set and a test set in a ratio of 7:2:1 or 8:1:1, and the training set can be used for subsequent training processes.

[0085] Step S2: Obtain an image classification model, which includes a feature extractor for extracting image features from an input image and a stacked multi-layer bidirectional logic tree, wherein each layer of the bidirectional logic tree is a classifier of one layer, the input of the first layer classifier is the image feature, the input of each subsequent layer classifier is the output of the classifier of the previous layer, and the output of each layer classifier is a first logical value used to determine the classification result of the layer.

[0086] According to one embodiment of the present invention, the image classification model constructed in the above embodiment is obtained, and the trainable parameters of the initial model are initialized by random initialization. In the subsequent process, the model is iteratively trained, that is, each training is based on the model parameters obtained in the previous training.

[0087] Step S3: obtaining a preset adaptive intra-granularity difference learning network, wherein the learning network includes multiple layers of learning matrices, and each layer of the learning matrix is ​​used to map the first logical value output by the classifier at the same layer to obtain a second logical value.

[0088] According to one embodiment of the present invention, the adaptive intra-granularity difference learning network constructed in the above embodiment is obtained, and the trainable parameters of the initial learning network are randomly initialized. In the subsequent process, the learning network is iteratively trained, that is, each training is updated based on the learning matrix of the learning network obtained in the previous training.

[0089] Step S4: Use the training set and the preset total loss function to guide the image classification model and the learning network for training, wherein the total loss function is configured to guide the model to learn the classification knowledge of each layer according to the first logical value and the classification label, and to guide the model to learn the similarity information between each category in the same layer and the true category according to the second logical value and the smooth label of the sample.

[0090] According to one embodiment of the present invention, the training process may use any total loss function of the aforementioned embodiment to guide the training process of the model and the learning network.

[0091] 8. Application Scenarios

[0092] According to one embodiment of the present invention, there is provided an image classification method, comprising: obtaining a trained image classification model trained according to the training method of the image classification model of the aforementioned embodiment; obtaining an image to be classified, inputting the image to be classified into the trained image classification model, and obtaining the first logical value output by each layer classifier; and determining the classification result of each layer according to the first logical value output by each layer classifier. For example: inputting the first logical value of the h-th layer into the Softmax function, obtaining the confidence that the image to be classified belongs to each category of the h-th layer, and the confidence that the image to be classified belongs to each category of the h-th layer can be used as the classification result. Alternatively, the category with the highest confidence is selected as the classification result of the h-th layer. Aggregating the classification results of all layers is the total classification result.

[0093] In order to verify the effect of the method of the present invention, the inventors conducted the following comparative experiments:

[0094] Comparison method:

[0095] CE, or CrossEntropy, refers to a method based only on the conventional cross entropy loss function and the existing model.

[0096] HXE, References: Bertinetto, Luca, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, and Nicholas A. Lord. "Making better mistakes: Leveraging class hierarchies with deep networks." In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pp. 12506 - 12515. 2020.

[0097] Soft-Labels, References: Bertinetto, Luca, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, and Nicholas A. Lord. "Making better mistakes: Leveraging class hierarchies with deep networks." In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pp. 12506 - 12515. 2020.

[0098] Flamingo, References: Chang, Dongliang, Kaiyue Pang, Yixiao Zheng, Zhanyu Ma, Yi-Zhe Song, and Jun Guo. "Your "flamingo" is my "bird": Fine-grained, or not." In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 11476 - 11485. 2021.

[0099] CRM, the reference is: Karthik, Shyamgopal, Ameya Prabhu, Puneet K. Dokania, and Vineet Gandhi. "No Cost Likelihood Manipulation at Test Time for Making Better Mistakes in Deep Networks." In International Conference on Learning Representations.

[0100] HAF, the reference is: Garg, Ashima, Depanshu Sani, and Saket Anand. "Learninghierarchy aware features for reducing mistake severity." In European Conference on Computer Vision, pp. 252-267. Cham: Springer Nature Switzerland, 2022.

[0101] HAFrame, reference: Liang, Tong, and Jim Davis. "Inducing neural collapse to a fixed hierarchy-aware frame for reducing mistake severity." InProceedings of the IEEE / CVF International Conference on Computer Vision, pp.1443-1452.2023.

[0102] HiE, the reference is: Jain, Kanishk, Shyamgopal Karthik, and Vineet Gandhi. "Test-time amendment with a coarse classifier for fine-grained classification." Advances in Neural Information Processing Systems 36 (2024).

[0103] The total loss function used in the method of the present invention during the experiment is:

[0104]

[0105] The smoothing label calculation method used by the method of the present invention during the experiment is:

[0106]

[0107] Datasets used in the experiment: The experiments are conducted on three public datasets: FGVC-Aircraft, CIFAR100, and iNaturalist2019.

[0108] Experimental indicators: Mistakes severity, Hier dist@5, Hier dist@20, and Accuracy. Mistakes severity refers to the mean of the shortest common ancestor distance (LCA) between the labels of the predicted incorrect samples and the true labels, Hier dist@k refers to the mean of the LCA distance between the model's Top-k predictions and the true labels, and Accuracy refers to the ratio of samples predicted correctly by the model to all samples.

[0109] The experimental results are shown in Table 1-3:

[0110] Table 1: Performance of each method on the FGVC-Aircraft dataset

[0111]

[0112] Table 2: Performance of each method on the CIFAR100 dataset

[0113]

[0114] Table 3: Performance of each method on the iNaturalist2019 dataset

[0115]

[0116]

[0117] It can be seen that in the three public datasets of FGVC-Aircraft, CIFAR100, and iNaturalist2019, in terms of four indicators, including error severity (Mistakes severity), Top-5 level distance (Hier dist@5), Top-20 level distance (Hier dist@20), and accuracy (Accuracy), the results of most indicators of the method of the present invention are optimal, which verifies the effectiveness of the method of the present invention.

[0118] In general, the core goal of the present invention is to solve the potential competition problem between fine-grained classification and coarse-grained classification in a multi-granularity classification framework, and then explore a strategy to efficiently integrate the information advantages of coarse-grained classification as an auxiliary means to improve the performance of fine-grained classification. In the process of model learning, previous methods only emphasized the negative impact of coarse-grained classification on fine-grained learning, while ignoring the key fact that coarse-grained classification can assist in improving fine-grained learning. In order to solve this problem, the inventors proposed a new hierarchical classification learning framework based on a bidirectional logic tree (called BiLT), which aims to effectively use hierarchical label information to construct a hierarchical perception classifier to alleviate the granularity competition problem. The hierarchical classification learning framework based on a bidirectional logic tree achieves this goal by sequentially constructing classifiers from fine-grained to coarse-grained, and the input of the coarse-grained classifier comes from the output of the finer-grained classifier. In addition, an adaptive internal granularity difference learning (called AIGDL) method (mainly based on the learning network and the second sub-loss function in the total loss function) is developed to better explore the subtle semantic differences between subclasses at the same granularity and improve the performance of the model.

[0119] It should be noted that although the above describes the various steps in a specific order, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0120] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0121] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0122] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A training method for an image classification model, characterized in that: include: A multi-level classification training set is obtained, the training set comprising: a plurality of samples, each sample comprising an image and a classification label, the classification label indicating the true category to which the image in the sample belongs in each of the plurality of preset levels; Obtain an image classification model, which includes a feature extractor for extracting image features from an input image and a stacked multi-layer bidirectional logic tree, wherein each layer of the bidirectional logic tree is a classifier of a layer, the input of the first layer classifier is the image feature, the input of each subsequent layer classifier is the output of the previous layer classifier, and the output of each layer classifier is a first logic value for determining the classification result of the layer; Obtaining a preset adaptive intra-granularity difference learning network, wherein the learning network includes a multi-layer learning matrix, and each layer of the learning matrix is ​​used to map a first logic value output by a classifier at the same layer to obtain a second logic value; The training set and the preset total loss function are used to guide the image classification model and the learning network for training, wherein the total loss function is configured to guide the model to learn the classification knowledge of each layer according to the first logical value and the classification label, and to guide the model to learn the similarity information between each category in the same layer and the true category according to the second logical value and the smooth label of the sample.

2. The method according to claim 1, characterized in that In the multi-layer bidirectional logic tree, the classifier closer to the feature extractor is used to classify the finer-grained levels in the multi-level classification.

3. The method according to claim 1, characterized in that The smoothed label corresponding to a sample is obtained by aggregating the classification label of the sample, the weight vector of the row corresponding to the true category indicated in the classification label of the sample in the learning matrix of the learning network, and the distance between each two categories involved in the classification label of the sample and the shortest common ancestor, wherein the weight vector is the weight of the first logical value of each category when calculating the second logical value of the true category currently learned by the learning network.

4. The method according to claim 3, characterized in that The total loss function is a weighted sum of multiple sub-loss functions, including a first sub-loss function for calculating the first logical value of each layer and the loss between classification labels, and a second sub-loss function for calculating the second logical value of each layer and the loss between smoothed labels.

5. The method according to claim 4, characterized in that The total loss function is: Among them, h represents the hth layer, It represents the first sub-loss function for finding the loss between the first logical value and the classification label of the h-th layer. It represents the second sub-loss function for finding the second logical value of the h-th layer and the loss between smooth labels, and α represents The weighted value of express The weighted value of .

6. The method according to claim 4, characterized in that The total loss function is: Among them, h represents the hth layer, λ h Represents the hyperparameters set for the sub-loss of each layer, and the λ set for the layer farther away from the backbone network h The smaller the value, It represents the first sub-loss function for finding the loss between the first logical value and the classification label of the h-th layer. It represents the second sub-loss function for finding the second logical value of the h-th layer and the loss between smooth labels, and α represents The weighted value of express The weighted value of .

7. The method according to claim 5 or 6, characterized in that: The second sub-loss function is: Where N represents the number of samples used in a training batch. represents the smoothed label of the i-th sample The h-th layer of the Represents the learning matrix α of the learning network h The hth layer in The weight vector of the row, β represents the weight coefficient, Represents the h-th level subtable in the shortest common ancestor distance table D constructed according to the label tree corresponding to the classification label The weight of the line, represents the second logical value output by the h-th layer for the i-th sample, and Softmax(·) represents the Softmax function.

8. The method according to claim 7, characterized in that The smoothed label of the i-th sample The components of the hth layer in are: in, represents the h-th layer component of the classification label of the i-th sample, Denotes the learning matrix Δ h The hth layer The element corresponding to the jth column of the row, Represents the h-th level sub-table in the distance table D The element corresponding to the jth column of the row, exp′(·) represents the function used to treat each element of the vector in the brackets as the power calculation result of e and then reorganize it into a vector in order, exp(·) represents the natural exponential function, γ represents the temperature coefficient, ∈ represents the weighting coefficient, C h Represents the total number of all categories in the hth layer.

9. An image classification method, comprising: Obtain a trained image classification model trained according to the method of any one of claims 1-8; Acquire an image to be classified, input the image to be classified into the trained image classification model, and obtain a first logical value output by each layer of classifier; The classification result of each layer is determined according to the first logical value output by the classifier of each layer.

10. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 9.

12. An electronic device, characterized in that: include: one or more processors; as well as A memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1-9 by executing the executable instructions.

Images