Image classification system and training method thereof, and image classification method

Abstract

An embodiment of the present invention provides an image classification system for classifying an input image, the system comprising: a feature extraction module for extracting features from the image to obtain a feature vector corresponding to the image; a feature rotation module for rotating the feature vector based on an orthogonal projection matrix to obtain a rotated feature vector, wherein the orthogonal projection matrix is ​​extracted from a preset orthogonal matrix; a classification module comprising: a fixed parameter linear layer for equalizing the rotated feature vector to obtain a logit value indicating that the image belongs to each category; and a classification layer for determining the confidence level that the image belongs to each category based on the logit value indicating that the image belongs to each category. The image classification system designed in the embodiment of the present invention has good generalization for long-tail image classification scenarios and the performance of the model itself is also good.

Description

Technical Field

[0001] The present invention relates to the field of multi-classification in the field of machine learning, and more specifically, to the field of long-tail classification tasks in which the categories to which samples in a training set belong are extremely unevenly distributed (or the labels are extremely unevenly distributed). More specifically, it relates to an image classification system and its training method, as well as an image classification method. Background Art

[0002] In image classification tasks, the long-tail learning problem is a challenge faced by deep neural network models. Long-tail distribution refers to image data where some categories (the head categories) have a large number of samples, while others (the tail categories) contain only a small number of samples. In this case, training a well-performing model is difficult because the model is dominated by the head categories and ignores the tail categories.

[0003] In traditional image classification and recognition tasks, training data from a long-tail distribution of images is often artificially balanced, meaning that the number of samples across different categories is not significantly different. A balanced dataset greatly simplifies the requirements for algorithm robustness and, to a certain extent, ensures the reliability of the resulting model. However, as the number of categories of interest increases, maintaining a balance across categories leads to exponentially increasing collection costs. For example, for an animal classification dataset, millions of images of common animals like cats and dogs can be easily collected. However, to maintain a balanced dataset, an equal number of samples must also be collected for rare animals like snow leopards. As the rarity of a category increases, the collection cost often increases exponentially. Therefore, it is essential to develop methods that can perform well on such imbalanced data.

[0004] Currently, the two simplest basic methods for addressing the long-tail distribution of image data classification are resampling and reweighting. These methods essentially leverage the known distribution of the dataset and perform a brute-force inverse weighting of the data distribution during the learning process (some literature refers to this as hacking). This approach aims to offset the long-tail effect by strengthening the learning of the tail classes. However, these methods often suffer from performance tradeoffs, improving the recognition performance of the tail classes at the expense of the performance of the head classes.

[0005] Recently, the phenomenon of neural collapse has attracted increasing attention in the field of deep learning. Since the last layer in a deep neural network model has a linear classification layer, neural collapse essentially reveals that the final layer classifier in a deep neural network model tends to have a simple symmetric structure to achieve its state-of-the-art performance. The following are several phenomena of neural collapse: (1) Collapse of variability. For data samples belonging to the same class, the features of the classifier will be concentrated around its class average, that is, the features of samples of the same class collapse to a point; (2) Features of different classes tend to form a simple equiangular tight frame (ETF). The features of each class reach maximum linear separability and tend to form a simple equiangular tight frame; (3) Self-dual alignment. Among the features of the same class, except for the features of the last layer that reach a point, the vectors of the linear classifier matrix of the last layer also form a simple equiangular tight frame in its dual vector space and converge to the simple equiangular tight frame of the last layer features; (4) Nearest neighbor classification. When converged, the inference result of the model will be the same as the result of nearest neighbor classification in the feature space of the last layer. Therefore, the neural collapse phenomenon refers to the training of a classifier on a label-balanced dataset. When the cross-entropy loss function reaches its minimum value, the features of the classifier (the activation values ​​of the last layer of the deep model) will learn a completely symmetrical structure, resulting in each category being highly symmetrical in the feature space, and ultimately forming an equiangular tight frame (ETF). Based on the neural collapse phenomenon, the industry has also proposed a method to solve the problem of classifying image data with long-tail distribution from the perspective of feature learning. One of the methods is to directly fix the classifier of the classification model to ETF to learn a balanced representation in the hope of achieving satisfactory results. Although this method can learn a balanced feature space so that the feature space of the classification problem reaches the structure of ETF. However, directly learning features in a fixed direction will impair the learning ability of the model, resulting in worse generalization performance.

[0006] In summary, existing methods for classifying long-tail image data, such as those based on data resampling and loss reweighting, suffer from a model-performance tradeoff: improving the recognition performance of the tail classes at the expense of the performance of the head classes. Methods that simply fix the classification model's classifier to the ETF to learn features in a fixed direction impair the model's learning ability, resulting in poor generalization performance. Therefore, there is an urgent need for an improved deep neural network model that generalizes well to long-tail image classification scenarios and maintains good model performance. Summary of the Invention

[0007] Therefore, the purpose of the present invention is to overcome the above-mentioned defects of the prior art and provide an image classification system and its training method, as well as an image classification method.

[0008] The purpose of the present invention is achieved through the following technical solutions:

[0009] According to a first aspect of the present invention, there is provided an image classification system for classifying an input image, the system comprising:

[0010] A feature extraction module is used to extract features from an image to obtain a feature vector corresponding to the image;

[0011] A feature rotation module, configured to rotate the feature vector based on an orthogonal projection matrix to obtain a rotated feature vector, wherein the orthogonal projection matrix is ​​extracted from a preset orthogonal matrix;

[0012] The classification module includes a fixed parameter linear layer and a classification layer based on a preset equiangular tight framework, wherein:

[0013] The fixed parameter linear layer is used to equalize the rotated feature vector to obtain the logit value of each category of the image;

[0014] The classification layer is used to determine the confidence that the image belongs to each category based on the logit value of the image belonging to each category.

[0015] In some embodiments of the present invention, the orthogonal projection matrix is ​​obtained according to a learnable orthogonal matrix, where:

[0016] An orthogonal matrix is ​​a matrix whose row and column dimensions are both max(N,d), where N represents the total number of categories that can be identified by the classification, and d represents the dimension of the feature vector obtained by the feature extraction module.

[0017] When N=d, the orthogonal projection matrix is ​​equal to the orthogonal matrix itself;

[0018] N≠d, the orthogonal projection matrix extracts a block matrix that meets the orthogonal matrix characteristics from the orthogonal matrix.

[0019] In some embodiments of the present invention, the parameters of the fixed parameter linear layer are set based on a preset equiangular tight frame as follows:

[0020]

[0021] Among them, the M * It is not updated when the system is trained for classification. I is the unit matrix, I∈(N×N), and E is a vector of all 1s, E∈(1,N).

[0022] According to a second aspect of the present invention, a method for training an image classification system is provided, the method comprising:

[0023] S1. Obtain a training set for image classification, which includes multiple sample images and a label corresponding to each sample image, wherein the label indicates the true value of the category of the corresponding sample image;

[0024] S2. Obtain an image classification system including a feature extraction module, a feature rotation module, and a classification module, and train the image classification system once or multiple times using the training set in the following manner to obtain a trained image classification system:

[0025] The feature extraction module extracts features from the input sample image to obtain a feature vector corresponding to the sample image;

[0026] The feature rotation module rotates the feature vector based on the orthogonal projection matrix to obtain a rotated feature vector corresponding to the sample image, wherein the orthogonal projection matrix is ​​extracted from a learnable orthogonal matrix;

[0027] The fixed parameter linear layer of the classification module equalizes the rotated feature vector corresponding to the sample image to obtain the logit value of the sample image belonging to each category, and the classification layer of the classification module determines the confidence that the sample image belongs to each category based on the logit value of the sample image belonging to each category;

[0028] The classification loss value is determined according to the confidence and label of the corresponding sample image in the training set belonging to each category, and the parameters in the feature extraction module and the learnable orthogonal matrix are updated according to the classification loss value.

[0029] In some embodiments of the present invention, the orthogonal projection matrix is ​​obtained according to a learnable orthogonal matrix, wherein,

[0030] An orthogonal matrix is ​​a matrix whose row and column dimensions are both max(N,d), where N represents the total number of categories that can be identified by the classification, and d represents the dimension of the feature vector obtained by the feature extraction module.

[0031] When N=d, the orthogonal projection matrix is ​​equal to the orthogonal matrix itself;

[0032] N≠d, the orthogonal projection matrix is ​​a block matrix taken out from the orthogonal matrix and satisfies the orthogonal matrix characteristics.

[0033] In some embodiments of the present invention, the parameters of the fixed parameter linear layer are set based on a preset equiangular tight frame as follows:

[0034]

[0035] Among them, the M * It is not updated when the system is trained for classification. I is the unit matrix, I∈(N×N), and E is a vector of all 1s, E∈(1,N).

[0036] In some embodiments of the present invention, in step S2, when training the image classification system, the orthogonal matrix of the feature rotation module is converted into a skew-symmetric matrix based on a preset algebraic definition, and the feature extraction module and the feature rotation module are updated in the following manner:

[0037]

[0038] R←exp(BB T )

[0039]

[0040] Among them, θ is the parameter of the feature extraction module, α is the learning rate, The loss gradient under the parameter θ, R is an orthogonal matrix, B is the skew-symmetric matrix transformed from R by the preset algebraic definition, is the loss gradient corresponding to B.

[0041] According to a third aspect of the present invention, the image classification method includes:

[0042] Get the image to be classified;

[0043] The image to be classified is input into the trained image classification system obtained by the training method of the image classification system described in the above embodiment for classification.

[0044] According to a fourth aspect of the present invention, the image classification method includes:

[0045] Get the image to be classified;

[0046] Obtaining an adjusted image classification system, which is obtained by replacing a classification layer in a trained image classification system obtained by the training method of the image classification system described in the above embodiment with the adjusted classification layer;

[0047] The image to be classified is classified using the adjusted image classification system, wherein the adjusted classification layer obtains the category prediction value corresponding to the image in the following manner:

[0048]

[0049] Among them, logit i is the logit value of the image belonging to the i-th category, P i is the proportion of samples belonging to the i-th category in the image.

[0050] According to the fifth aspect of the present invention, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the steps of the methods described in the second, third and fourth aspects.

[0051] Compared with the prior art, the advantages of the present invention are:

[0052] The solution of the present invention introduces an orthogonal matrix to learn the direction in the feature space. This method greatly improves the generalization ability of the image classification system in long-tail scenarios. When classifying image data, from the perspective of improving the decision-making ability of the classification module, it is further proposed to use the post-logit adjustment method during classification. This method can ensure that the image classification system can give full play to the ability of the learned balanced features, so that the image classification system can have better effects in long-tail classification scenarios. Therefore, the image classification system designed based on the solution of the present invention has good generalization for long-tail image classification scenarios and the performance of the system itself is also good. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0054] Figure 1 2. A schematic diagram of an image classification system framework according to an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of the improved ResNet-32 model structure according to an embodiment of the present invention;

[0056] Figure 3 Schematic diagram of the structure of the ResNeXt-50 model improved according to an embodiment of the present invention;

[0057] The reference numerals in the specific implementation manner are as follows: 100 - image classification system; 110 - feature extraction module; 120 - feature rotation module; 130 - classification module; 132 - fixed parameter linear layer; 134 - classification layer. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0059] As described in the background, existing methods for classifying long-tail image data, such as those based on data resampling and loss reweighting, suffer from a model-performance tradeoff: improving the recognition performance of the tail class at the expense of the performance of the head class. Similarly, methods that simply fix the image classification system's classifier to the ETF to learn features in a fixed direction impair the model's learning ability, resulting in poorer generalization performance. Therefore, there is an urgent need for an improved deep neural network model that generalizes well to long-tail image classification scenarios and offers excellent model performance.

[0060] To address these issues, this paper proposes a balanced learning framework (RBL, for short). Technically, this approach learns the feature space of the ETF and introduces an optimizable orthogonal matrix to learn the eigenvector directions.

[0061] In order to better understand the present invention, the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0062] According to one embodiment of the present invention, Figure 1 As shown, the present invention proposes an image classification system 100 for classifying an input image, wherein the system 100 includes: a feature extraction module 110 for extracting features from an image to obtain a feature vector corresponding to the image; a feature rotation module 120 for rotating the feature vector based on an orthogonal projection matrix to obtain a rotated feature vector, wherein the orthogonal projection matrix is ​​extracted from a preset orthogonal matrix; a classification module 130, which includes a fixed parameter linear layer 132 and a classification layer 134 set based on a preset equiangular tight frame, wherein: the fixed parameter linear layer 132 is used to equalize the rotated feature vector to obtain the logit value of the image belonging to each category; the classification layer 134 is used to determine the confidence that the image belongs to each category based on the logit value of the image belonging to each category.

[0063] In order to improve the classification effect of the above-mentioned image classification system 100, the present invention proposes a training method for the image classification system 100 described in the above-mentioned embodiment.

[0064] In order to more intuitively understand the training method of the image classification system 100 proposed in the present invention, the basic framework and basic working principle of the image classification system in the neural network are first introduced.

[0065] When performing image classification, the image classification system (i.e., classification model) is generally expressed as follows:

[0066] logit=Mf(x;θ)

[0067] Where x is the image data, θ is the parameter of the feature extraction module 110, f(x; θ) is the parameter of the feature extraction module 110, M is the ETF, that is, the parameters of the fixed parameter linear layer 132 of the classification module 130 are set based on the ETF, and logit represents the logit value of the image belonging to each category after the image data x is processed by the feature extraction module 110 and the fixed parameter linear layer 132 of the classification module 130. After obtaining the logit value of the image belonging to each category, the confidence level of the image belonging to each category is determined as follows:

[0068]

[0069] Wherein, N represents the total number of categories that can be identified by the classification. When training the image classification system 100, the image classification system 100 is updated by taking the cross entropy loss function as an example. The cross entropy loss function is expressed as:

[0070]

[0071] Among them, y∈(1,N), y is the category corresponding to the image data, S is the training set, logit y is the logit value of the image data belonging to category y, j∈(1,N), logit j Indicates the logit value that the image data belongs to category j.

[0072] In order to enable the image classification system to learn balanced features, the existing technology fixes the parameters of the fixed parameter linear layer 132 based on the equiangular tight frame (ETF), and the corresponding cross entropy loss function is expressed as:

[0073]

[0074] It can be seen that M is not included in the optimization formula, which means that the parameters of the fixed parameter linear layer 132 are not optimized during training.

[0075] The inventors have discovered that, in the prior art, the training method of setting the parameters of the fixed-parameter linear layer 132 in the classification module 130 to a fixed form can also learn balanced features when the training set is unbalanced. However, the fixed setting of the parameters of the fixed-parameter linear layer 132 based on ETF will reduce the learning ability of the image classification system, which will damage the generalization of the classification system. Therefore, in order to be able to learn balanced features, the direction of the feature can also be learned.

[0076] According to an embodiment of the present invention, the present invention proposes a training method for an image classification system 100 , which includes steps S1 - S2 . The implementation process of each step is described in detail below.

[0077] S1. Obtain a training set for image classification, which includes multiple sample images and a label corresponding to each sample image, where the label indicates the true value of the category of the corresponding sample image.

[0078] According to one embodiment of the present invention, the training set for image classification used in the embodiment of the present invention can be a subset extracted from the CIFAR10, CIFAR100 or ImageNet dataset. Take the CIFAR10 dataset as an example. The sample images are images containing airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships or trucks, and the labels are the true values ​​of the categories to which each sample image belongs. However, in order to simulate the effect of category imbalance. Schematically, a preset imbalance ratio can be used to extract sample images of each category from the dataset to form a long-tail training set, wherein the imbalance ratio is the ratio of the number of samples of the category with the most sample images to the number of samples of the category with the least sample images, which measures the degree of imbalance of the dataset. Schematically, the imbalance ratio is set to 50, 100, 200 or 256.

[0079] It should be understood that the space of labels corresponding to the sample images in the training set (or the set of categories) can adopt the label space of the existing data set, such as the above-mentioned CIFAR10, CIFAR100 (various aquatic mammals, fish, food containers, food containers, household appliances, etc.) or ImageNet data set (animal categories (dogs, cats, birds, fish, horses, etc.), plant categories (flowers, trees, grass, etc.), appliance categories (vehicles, furniture, electrical appliances, etc.), food categories (fruits, vegetables, desserts, meat, etc.), natural landscape categories (beaches, mountains, forests, rivers, etc.)), or the space of labels customized by the implementer, which is not specifically limited here.

[0080] S2. Obtain an image classification system 100 including a feature extraction module 110, a feature rotation module 120, and a classification module 130, and use the training set to train the image classification system 100 once or multiple times in the following manner to obtain a trained image classification system 100.

[0081] In step S1, when the training sets of image classification used by the embodiment of the present invention are sample images corresponding to CIFAR10, CIFAR100, and ImageNet, respectively, the present invention uses ResNet-32 and ResNeXt-50 as the basic models, and adopts the technical ideas of the above embodiment of the present invention to design the image classification system 100 to improve the ResNet-32 and ResNet-50 models. Specifically, Figure 2 As shown, it is the improved ResNet-32 model structure diagram, from Figure 2As can be seen from the figure, the features corresponding to the output categories in the existing ResNet-32 model are changed to the preset feature dimensions, and then the whole model serves as the feature extraction module 110, followed by the feature rotation module 120 and the classification module 130, wherein the classification module 130 further includes a fixed parameter linear layer 132 and a classification layer 134, and the classification layer 134 is obtained by replacing the adjusted classification layer; Figure 3 As shown, it is the improved ResNeXt-50 model structure diagram, from Figure 3 It can be seen that the features corresponding to the output categories in the existing ResNeXt-50 model are changed to preset feature dimensions, and then the whole model serves as a feature extraction module 110, followed by a feature rotation module 120 and a classification module 130, wherein the classification module 130 also includes a fixed parameter linear layer 132 and a classification layer 134, and the classification layer 134 is obtained by replacing the adjusted classification layer. Figure 2 There are four types of stacked blocks, namely: 3x means there are three blocks in the first type of stacking, 4x means there are four blocks in the second type of stacking, 6x means there are six blocks in the third type of stacking, and 3x means there are three blocks in the fourth type of stacking; Figure 3 There are four types of stacked blocks in the , namely: 4x means there are four blocks in the first type of stacking, 5x means there are five blocks in the second type of stacking, 10x means there are ten blocks in the third type of stacking, 5x means there are five blocks in the fourth type of stacking, and Figure 2 and Figure 3 In the example, 3x3 conv refers to a convolutional layer with a 3x3 kernel, channel refers to the number of output channels, Average Pool refers to the average pooling layer, and FC refers to the fully connected layer. The training process for the two improved models is as follows:

[0082] When the training set for image classification is CIFAR10, ResNet-32 is used as the basic model, and the technical concept of designing the image classification system 100 according to the above embodiment of the present invention is used to improve the ResNet-32 model (i.e. Figure 2 The model shown in FIG10 is a schematic diagram of a training process for image classification using the sample images in the Cifar10 dataset as the training set for image classification. The schematic process is as follows: the training set is input into the improved ResNet-32 model for one or more trainings to obtain a trained image classification system 100 (i.e., the improved ResNet-32 model).

[0083] When the training set for image classification is CIFAR100, ResNet-32 is used as the basic model, and the technical concept of designing the image classification system 100 according to the above embodiment of the present invention is used to improve the ResNet-32 model (i.e. Figure 2The model shown in FIG1 is a schematic diagram of a method for image classification system 100. When the sample images in the Cifar100 dataset are used as the training set for image classification, the schematic process is as follows: the training set is input into the improved ResNet-32 model for one or more trainings to obtain a trained image classification system 100 (i.e., the improved ResNet-32 model).

[0084] When the training set for image classification is ImageNet, ResNeXt-50 is used as the basic model, and the technical concept of designing the image classification system 100 according to the above embodiment of the present invention is used to improve the ResNeXt-50 model (i.e., Figure 3 The model shown in the figure) is used as the training set for image classification using sample images in the ImageNet dataset. The schematic process is as follows: the training set is input into the improved ResNeXt-50 model for one or more trainings to obtain a trained image classification system 100 (i.e., the improved ResNeXt-50 model).

[0085] It should be noted that, when selecting a deep neural network model as the basic model of the image classification system 100 designed in the embodiment of the present invention, ResNet-32 and ResNeXt-50 models are only two implementation methods of the embodiment of the present invention. In order to make the image classification effect better, the embodiment of the present invention can select other deep neural network models as the basis, and use the image classification system 100 designed in the above embodiment of the present invention to improve the basic model. No specific limitation is made here.

[0086] The functions of each module in the image classification system 100 designed in the embodiment of the present invention are as follows: the feature extraction module 110 performs feature extraction on the input sample image to obtain a feature vector corresponding to the sample image; the feature rotation module 120 rotates the feature vector based on the orthogonal projection matrix to obtain a rotated feature vector corresponding to the sample image, wherein the orthogonal projection matrix is ​​extracted from a learnable orthogonal matrix; the fixed parameter linear layer 132 of the classification module 130 equalizes the rotated feature vector corresponding to the sample image to obtain the logit value of the sample image belonging to each category, and the classification layer 134 of the classification module 130 determines the confidence that the sample image belongs to each category based on the logit value of the sample image belonging to each category; the classification loss value is determined according to the confidence and label of the corresponding sample image belonging to each category in the training set, and the parameters in the feature extraction module 110 and the learnable orthogonal matrix are updated according to the classification loss value. Logit can be expressed as:

[0087] logit=M ★ Rf(x;θ)

[0088] Where x is the image data, θ is the parameter of the feature extraction module 110, f(·;θ) represents the feature extraction module 110, R is the orthogonal projection matrix of the feature rotation module 120, M ★ is a fixed parameter linear layer 132, and logit represents the logit value of the image obtained after the image data x is processed by the feature extraction module 110 and the fixed parameter linear layer 132, belonging to each category. According to one embodiment of the present invention, the orthogonal projection matrix is ​​obtained based on a learnable orthogonal matrix R, wherein: the orthogonal matrix R is a matrix with both row and column dimensions of max(N,d), wherein N represents the total number of categories that can be identified by the classification, and d represents the dimension of the feature vector obtained by the feature extraction module. When N=d, the orthogonal projection matrix is ​​equal to the orthogonal matrix R itself; when N≠d, the orthogonal projection matrix extracts a block matrix that satisfies the orthogonal matrix characteristics from the orthogonal matrix R. By introducing a learnable orthogonal matrix R between the feature extraction layer 110 and the fixed parameter linear layer 132, the orthogonal matrix R can rotate the features obtained by the feature extraction module 110 (f(x;θ)) so that they are consistent with M ★ Registration. This is equivalent to enabling the direction of the feature to be learned. According to one embodiment of the present invention, the parameters of the fixed parameter linear layer 132 are set based on the preset equiangular tight frame to:

[0089]

[0090] Among them, the M * It is not updated when the system is trained for classification. I is the unit matrix, I∈(N×N), and E is a vector of all 1s, E∈(1,N). According to one embodiment of the present invention, in step S2, when the image classification system 100 is trained, the orthogonal matrix R of the feature rotation module 120 is converted into a skew-symmetric matrix based on a preset algebraic definition, and the loss value is calculated and the feature extraction module 110 and the feature rotation module 120 are updated in the following manner:

[0091]

[0092]

[0093]

[0094] Among them, θ is the parameter of the feature extraction module, α is the learning rate, The loss gradient under the parameter θ, R is an orthogonal matrix. It should be noted that the image classification system of the present invention is designed with the above-mentioned cross entropy loss function as an embodiment during training. The loss function in the image classification system of the present invention can be other types of loss functions, which are not specifically limited here. Since the orthogonal matrix R is a special square matrix, its column vectors (or row vectors) are orthogonal to each other, and the modulus is 1. The SGD optimizer cannot guarantee that the parameters can still satisfy the orthogonal constraints and modulus constraints after each parameter update. Therefore, in order to achieve the optimization (update) of the orthogonal matrix R, the inventor used an algebraic conclusion to encode an orthogonal matrix with a skew-symmetric matrix. For example: A is a skew-symmetric matrix (that is, it satisfies A+A T =0), an orthogonal matrix R can be obtained by the following mapping:

[0095] First, an orthogonal matrix group is:

[0096]

[0097] The corresponding Lie algebra is

[0098]

[0099] There is a substitution relationship between the above two algebraic systems, which is based on the following mapping:

[0100]

[0101]

[0102] Therefore, a constrained optimization problem on an orthogonal matrix can be transformed into:

[0103]

[0104] Where loss(·) is the loss function, R is an orthogonal matrix, L is the dimension of the orthogonal matrix R, and A is a skew-symmetric matrix. Through the above processing, an optimization problem constrained to an orthogonal matrix R is transformed into an optimization problem with a skew-symmetric constraint. Then, for a skew-symmetric matrix A, it can be represented by any square matrix B minus its own transpose:

[0105]

[0106] At this point, merging the two steps can eliminate the orthogonal matrix constraint:

[0107]

[0108] From the above, we can see that the orthogonal matrix R to be optimized is converted into an unconstrained square matrix B. Therefore, the general stochastic gradient descent optimization method can be used directly to optimize, and a B is used to parameterize the orthogonal matrix R (under this parameterization, each iteration can ensure that R maintains the orthogonal matrix constraint):

[0109] R←exp(BB T )

[0110]

[0111] Where B is the skew-symmetric matrix of the square matrix transformed by R through the preset algebraic definition. is the loss gradient corresponding to B. In summary, in order to address the problem in the unbalanced long-tail classification problem where setting the parameters of the fixed-parameter linear layer 132 based on ETF will impair the learning ability of the image classification system 100, an orthogonal matrix R is introduced to learn the direction of ETF in the feature space. This approach greatly improves the generalization ability of the image classification system 100 in long-tail scenarios.

[0112] After the image classification system 100 designed in the embodiment of the present invention is trained using the training method of the above embodiment, it is applied to image classification tasks. The following describes the process of the image classification system 100 in classifying images from a method perspective.

[0113] According to one embodiment of the present invention, the present invention proposes an image classification method, which includes: obtaining an image to be classified; inputting the image to be classified into the trained image classification system 100 obtained by the training method of the image classification system 100 in the above embodiment for classification.

[0114] Even if a classification module 130 has a good feature representation, it cannot have a satisfactory classification effect if the classification layer 134 is not appropriate. Therefore, in order to further improve the classification effect of the model for long-tail classification scenarios. According to one embodiment of the present invention, the present invention proposes another image classification method, which includes: obtaining an image to be classified; obtaining an adjusted image classification system 100, which is obtained by replacing the classification layer 134 in the trained image classification system 100 obtained by the training method of the image classification system 100 in the above embodiment with the adjusted classification layer 134; using the adjusted image classification system 100 to classify the image to be classified, wherein the adjusted classification layer 134 uses the Post-Hoc Logit Adjustment method, which introduces a set of decision threshold margins on the logit so that the reasoning of the model can meet Fisher consistency to better handle long-tail distribution data. Assume that the proportion of samples in each category in the training set is P1,…P N , and use the following method to get the category prediction value corresponding to the image:

[0115]

[0116] Among them, logit i is the logit value of the image belonging to the i-th category, P i is the proportion of samples belonging to the i-th category in the image. This process is equivalent to calculating the original logit value (i.e. logit) for each category. i ) and the correction factor (i.e. log(P i )) and obtain the corrected logit value corresponding to each category (i.e. [logit i +log(P i )]), take the category corresponding to the largest modified logit value as the output of the image classification system 100 for the image, that is, predict the category to which the image belongs.

[0117] From the above description, it can be seen that when reasoning about image data x, from the perspective of improving the decision-making ability of the classification module 130, it is further proposed to use the post-logit adjustment method during inference. This method can ensure that the image classification system 100 fully utilizes the ability of the learned balanced features, so that the image classification system 100 can have better results in long-tail classification scenarios.

[0118] In order to verify the effectiveness of the image classification system 100 and the training method of the system proposed in the above embodiment, the inventors conducted the following experiments:

[0119] The experiments used the CIFAR10, CIFAR100, and ImageNet datasets, and used the imbalance ratio to indicate the degree of imbalance in these datasets. The imbalance ratio is the ratio of the number of samples between the most common and the rarest categories in a dataset. The long-tail datasets, CIFAR10 and CIFAR100, both contain 60,000 images of size 32×32, of which 10,000 are used for testing and 50,000 are used for training. The labels of the original CIFAR10 and CIFAR100 datasets are uniformly distributed. The inventors generated long-tail versions of the original data. Specifically, the inventors used exponential decay to generate long-tail training sets with imbalance ratios of 50, 100, and 200, while the test sets of the two datasets remained unchanged. The imbalance ratio is the ratio of the number of samples in the most common category to the number of samples in the least common category, measuring the degree of imbalance in the dataset. The long-tail ImageNet-LT dataset is a long-tail version generated from the ImageNet-1K dataset. ImageNet-LT has 1K (1000) categories and 115.8K (115800) images for training, with an imbalance ratio of 256. In addition, the validation set (20000) and test set (50000) of ImageNet-LT have 20 and 50 images per category, respectively.

[0120] The experimental results of the present invention are compared with those of existing methods. The existing methods are divided into two technical routes: methods based on class balance, including class balanced loss (CB), calibration, decoupling-label smoothing (LWS), decoupling-class recalibration (cRT), see-saw, balanced softmax (BS), multi-label calibration (MARC), and label adaptation (LADE); methods based on contrastive learning, including triplet loss (TSC), hard contrast loss (HCL), and k-negative sample contrast loss (KCL).

[0121] On Cifar10, with ResNet-32 as the basic model, the technical concept of designing the image classification system 100 according to the above embodiment of the present invention is used to improve the ResNet-32 model (i.e. Figure 2The Cifar10 dataset includes sample images of 10 categories: airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. The training process involves using the sample images in the Cifar10 dataset as the model's training set, which also includes labels for each sample image. The training set is then fed into the improved ResNet-32 model for one or more training runs, resulting in a trained image classification system 100 (i.e., the improved ResNet-32 model). The classification performance of the improved ResNet-32 model according to the present invention was tested using a test set.

[0122] On Cifar100, with ResNet-32 as the basic model, the technical concept of the image classification system 100 designed by the above embodiment of the present invention is used to improve the ResNet-32 model (i.e. Figure 2 The model shown in Figure 1 is a Cifar10 dataset, which includes sample images from 100 categories, including various aquatic mammals, fish, food containers, household appliances, and so on. The training process involves using the sample images from the Cifar100 dataset as the model's training set, which also includes labels for each sample image. The training set is then fed into the improved ResNet-32 model for one or more training runs, resulting in a trained image classification system 100 (i.e., the improved ResNet-32 model). The classification performance of the improved ResNet-32 model according to the present invention was tested using a test set.

[0123] On ImageNet-LT, the ResNeXt-50 model is used as the basic model, and the technical concept of the image classification system 100 designed by the above embodiment of the present invention is used to improve the ResNeXt-50 model (i.e., Figure 3 The model shown in the figure) is a dataset of ImageNet-LT, which includes sample images of 1,000 categories, namely, animal categories (dogs, cats, birds, fish, horses, etc.), plant categories (flowers, trees, grass, etc.), appliance categories (vehicles, furniture, electrical appliances, etc.), food categories (fruits, vegetables, desserts, meat, etc.), and natural landscape categories (beaches, mountains, forests, rivers, etc.). The training process is as follows: the sample images in the ImageNet-LT dataset are used as the training set of the model, which also includes the label corresponding to each sample image; the training set is input into the improved ResNeXt-50 model for one or more trainings to obtain a trained image classification system 100 (i.e., the improved ResNeXt-50 model). The classification effect of the improved ResNeXt-50 model according to the embodiment of the present invention was tested using a test set.

[0124] The experimental results obtained through the above experiments are shown in Tables 1 and 2. Table 1 shows the comparison of the experimental results of image classification using the ResNet-32 model designed in accordance with the present invention on the Cifar10 and Cifar100 datasets with the results of the existing methods; Table 2 shows the comparison of the experimental results of image classification using the ResNeXt-50 model designed in accordance with the present invention on the ImageNet-LT dataset with the results of the existing methods. Among them, Fixed is a method based on the fixed parameter linear layer 132 of ETF fixed setting parameters, and RBL is the method proposed in accordance with the present invention (in the RBL model, the orthogonal matrix R is initialized to the unit matrix for the experiment). To achieve fair comparison, both methods introduce the post-logit adjustment (Post-Hoc Logit Adjustment) decision method. As can be seen from Table 1, the RBL framework (improved ResNet-32 model) designed by the present invention has a higher accuracy in image classification than the existing methods; as can be seen from Table 2, the present invention provides accuracy on three different subsets of the test set: a large number of samples (Many), a medium number of samples (Medium), a small number of samples (Few), and all samples (All). The Many subset only contains categories with more than 100 samples in the training set, the Medium subset and the Few subset contain categories with 20 to 100 samples and less than 20 samples in the training set, respectively, and All contains categories of all samples in the entire training set. The RBL framework (improved ResNeXt-50 model) designed by the present invention has a higher accuracy in image classification than the existing methods.

[0125] Table 1

[0126]

[0127] Table 2

[0128]

[0129] From the above experimental results, it can be seen that the method of the present invention is generally superior to the previous methods, which verifies the effectiveness of the method of the present invention.

[0130] In summary, the solution of the present invention introduces an orthogonal matrix to learn the direction of ETF in the feature space. This method greatly improves the generalization ability of the image classification system 100 in long-tail scenarios. When classifying image data x, from the perspective of improving the decision-making ability of the classification module 130, it is further proposed to use the post-logit adjustment method during classification. This method can ensure that the image classification system 100 can give full play to the ability of the learned balanced features, so that the image classification system 100 can have better results in long-tail classification scenarios. Therefore, the image classification system 100 designed based on the solution of the present invention has good generalization for long-tail image classification scenarios and the performance of the system itself is also good.

[0131] It should be noted that the image classification system 100 designed by the present invention is not limited to the application to the ResNeXt-50 model and the ResNet-32 model. In addition to the classification models verified by the experiment, various other models used for classification can use the method of the present invention to improve the classification effect of the classification model (image classification system). When training the image classification system 100 designed by the present invention, other forms of data sets can also be used for training, which is not specifically limited here. Initializing the orthogonal matrix R as a unit matrix in the image classification system 100 designed by the present invention is only one embodiment of the present invention. The orthogonal matrix R can also be initialized to other matrices that meet the characteristics of the orthogonal matrix, which is not specifically limited here.

[0132] It should be noted that although the above describes the various steps in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0133] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0134] Computer-readable storage media can be a tangible device that holds and stores the instructions used by an instruction execution device. Computer-readable storage media can, for example, include, but are not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, a punch card or a raised structure in a groove on which instructions are stored, for example, and any suitable combination thereof.

[0135] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image classification system for classifying an input image, characterized in that: The system comprises: A feature extraction module is used to extract features from an image to obtain a feature vector corresponding to the image; The feature rotation module is used to rotate the feature vector based on the orthogonal projection matrix to obtain the rotated feature vector, wherein the orthogonal projection matrix is ​​extracted from a preset orthogonal matrix, and the orthogonal projection matrix is ​​obtained according to a learnable orthogonal matrix, wherein: an orthogonal matrix is ​​a row and column dimension. The matrix, where Represents the total number of categories that can be identified by the classification, d represents the dimension of the feature vector obtained by the feature extraction module, when N d, the orthogonal projection matrix is ​​equal to the orthogonal matrix itself; when N d, the orthogonal projection matrix extracts a block matrix that satisfies the orthogonal matrix characteristics from the orthogonal matrix; The classification module includes a fixed parameter linear layer and a classification layer based on a preset equiangular tight framework, wherein: The fixed parameter linear layer is used to equalize the rotated feature vector to obtain the logit value of each category of the image. The parameters of the fixed parameter linear layer are set to: Among them, the It is not updated when the system is trained for classification. is the unit matrix, , is a vector of all 1s, ; The classification layer is used to determine the confidence that the image belongs to each category based on the logit value of the image belonging to each category; When the image classification system is trained, the orthogonal matrix of the feature rotation module is converted into a skew-symmetric matrix based on a preset algebraic definition, and the feature extraction module and the feature rotation module are updated in the following manner: in, are the parameters of the feature extraction module, is the learning rate, In the parameter The loss gradient under is an orthogonal matrix, B is the skew-symmetric matrix transformed from R by the preset algebraic definition, is the loss gradient corresponding to B.

2. A training method for an image classification system, characterized in that: The method comprises: S1. Obtain a training set for image classification, which includes multiple sample images and a label corresponding to each sample image, wherein the label indicates the true value of the category of the corresponding sample image; S2. Obtain an image classification system including a feature extraction module, a feature rotation module, and a classification module, and use the training set to train the image classification system according to claim 1 one or more times in the following manner to obtain a trained image classification system: The feature extraction module extracts features from the input sample image to obtain a feature vector corresponding to the sample image; The feature rotation module rotates the feature vector based on the orthogonal projection matrix to obtain a rotated feature vector corresponding to the sample image, wherein the orthogonal projection matrix is ​​extracted from a learnable orthogonal matrix; The fixed parameter linear layer of the classification module equalizes the rotated feature vector corresponding to the sample image to obtain the logit value of the sample image belonging to each category, and the classification layer of the classification module determines the confidence that the sample image belongs to each category based on the logit value of the sample image belonging to each category; The classification loss value is determined according to the confidence and label of the corresponding sample image in the training set belonging to each category, and the parameters in the feature extraction module and the learnable orthogonal matrix are updated according to the classification loss value.

3. An image classification method, characterized in that: The image classification method comprises: Get the image to be classified; The image to be classified is input into a trained image classification system obtained by the training method of the image classification system according to claim 2 for classification.

4. An image classification method, characterized in that: The image classification method comprises: Get the image to be classified; Obtaining an adjusted image classification system, which is obtained by replacing a classification layer in a trained image classification system obtained by using the image classification system training method according to claim 2 with the adjusted classification layer; The image to be classified is classified using the adjusted image classification system, wherein the adjusted classification layer obtains the category prediction value corresponding to the image in the following manner: in, is the logit value of the image belonging to the i-th category, is the proportion of samples belonging to the i-th category in the image.

5. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of any one of the methods of claims 2-4.

6. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the steps of the method according to any one of claims 2 to 4.

Images