Multi-category classification method and system based on AUC optimization
By designing and optimizing the loss function of multi-category AUC indicators in multi-category classification, combined with deep multi-layer perception machines, the problem of AUC optimization in multi-category classification is solved, and efficient multi-category classification performance and training acceleration are achieved.
Patent Information
- Application Number
- CN202210376289.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-11
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-04-11
AI Technical Summary
The prior art is difficult to effectively optimize AUC indicators in multi-category classification problems, especially in the case of uneven categories, and methods of directly optimizing accuracy cannot obtain ideal AUC performance.
A multi-category classification method was designed to construct the loss function of multi-category AUC indicators, and optimize it using square loss, exponential loss and hinge loss. In combination with deep multi-layer perception mechanism, the initial multi-classifier was built, and the final classification was obtained through iterative training. to optimize AUC indicators.
It significantly improves the AUC performance in multi-category classification, accelerates the model training process, reduces the computational complexity of the loss function and its gradient, and realizes efficient classification in multi-category scenarios.
Smart Images

Figure CN114861764B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of machine learning technology, and in particular to a learning method and system based on AUC optimization technology and for imbalanced data. Background Art
[0002] The area under the ROC curve (AUC) is widely adopted as a substitute for accuracy because it can directly reflect the performance of binary sorting and is insensitive to the distribution of class labels. The research on AUC can be traced back to the radar signal detection analysis in the 1970s, and then AUC was used in psychology and medical detection problems. In recent years, with the rapid development of machine learning and deep learning technology, the AUC evaluation indicator has gradually been adopted by the field of machine learning and has been successfully applied to specific learning tasks such as imbalanced learning, cost-sensitive learning, information retrieval, and personalized recommendation.
[0003] Although AUC has been adopted by many application problems, early machine learning models were not trained with reference to the AUC indicator, but mainly focused on maximizing accuracy. An early theoretical study on AUC pointed out that the overall expectation of AUC is proportional to the overall expectation of classification accuracy, but because the variance of AUC on class-imbalanced samples is usually larger than the accuracy, two classifiers with basically the same accuracy usually have completely different AUC values. This shows that early machine learning methods that directly optimize accuracy cannot achieve ideal AUC performance. This theoretical discovery has inspired a large number of machine learning-related research works focusing on AUC optimization.
[0004] Similar to most machine learning problems, the direct optimization problem of AUC is an NP-hard problem, and it is not feasible to directly construct the corresponding algorithm. At this time, it is often necessary to first construct a substitution loss function of the misplacement probability for optimization, and ensure that the function is continuous and differentiable and can approximate the misplacement loss. Based on the idea of substitution loss, a series of AUC optimization methods have emerged in the machine learning community, such as support vector machines based on hinge loss, ensemble learning boosting algorithms based on exponential loss, and least squares methods based on square loss.
[0005] In summary, AUC optimization has made significant breakthroughs in both algorithm and theory. However, mainstream AUC optimization research focuses only on single-task and binary classification machine learning problems. Practical application problems often need to consider multi-category classification problems, so how to design AUC optimization methods in such complex situations is still a scientific problem that needs to be solved urgently. Summary of the invention
[0006] In view of the above problems, the present invention uses the multi-classification AUC index M to measure the objective function of the design model training and accelerates the optimization process, and proposes a multi-classification classification method, which specifically includes:
[0007] Construct an initial multi-classifier for a multi-category scenario; construct a multi-classifier AUC indicator based on the AUC indicator of the multi-category scenario, and construct a loss function of the multi-classifier AUC indicator; optimize the loss function; train the initial multi-classifier through the optimized loss function to obtain a final classifier; and perform multi-category classification in the multi-category scenario with the final classifier.
[0008] The multi-class classification method of the present invention comprises the following steps: the initial multi-classifier is constructed by a deep multi-layer perceptron, and the last layer of the initial multi-classifier is subjected to N C Output, N C is a finite sample dataset for this multi-category scene The number of categories.
[0009] The multi-category classification method of the present invention obtains the binary classification AUC index AUC of the multi-category scene. i|j (f (i) ), the multi-classifier AUC index AUCM(f) is reduced to all pairwise binary classification AUC index AUC i|j (f (i) )’s average value:
[0010]
[0011] AUC i|j (f (i) ) represents the AUC index between the i-th class sample and the j-th class sample, f (i) The corresponding sub-classifier is the i-th output.
[0012] The multi-class classification method of the present invention, wherein the loss function comprises: square loss Exponential loss and hinge loss
[0013]
[0014]
[0015]
[0016] express The subset of samples of the i-th category in , express The subset of samples of the jth class in n i express The number of samples, n j express The number of samples, x1 is from For example, x2 comes from For an example of , and i≠j, is f parameterized by the model parameter space θ∈Θ (i) ; (x) + In the form of hinge loss, satisfying (x) + =x·I[x≥0], I[x≥0] represents the indicator function of the event x≥0, when x≥0 holds, I[x≥0]=1, otherwise I[x≥0]=0.
[0017] The multi-class classification method of the present invention, wherein the square loss Optimize to get:
[0018]
[0019] Index loss Optimize to get:
[0020]
[0021] Hinge loss Optimize to get:
[0022]
[0023] Among them, ⊙ is the Hadamard product, Representation Matrix The transposed matrix of represents an intermediate recursive variable for the i-th class sample, It represents the mth sample after arranging the i-th class samples in descending order of prediction scores. represents another intermediate recursive variable of the i-th sample, X represents the matrix formed by concatenating the features of all N samples, and Y (i) =[I[y1=i],…,I[y w =i]] T , For sample About the activation set of class i.
[0024] The multi-class classification method of the present invention uses square loss Exponential loss and hinge loss The initial multi-classifier is trained iteratively until the difference between two adjacent AUCM(f) is less than the preset threshold, and the final classifier is output.
[0025] The multi-category classification method of the present invention uses the final classifier Make predictions for the classified samples x Select f (i) (x) The largest category i is used as the class label prediction result for the sample x to be classified.
[0026] The present invention also proposes a multi-category classification system based on AUC optimization, comprising: a model construction module, used to construct an initial multi-classifier for a multi-category scenario; a model training module, used to train the initial multi-classifier to obtain a final classifier; wherein, a multi-classifier AUC index is constructed according to the AUC index of the multi-category scenario, and a loss function of the multi-classifier AUC index is constructed; the processing speed of the loss function is optimized; the initial multi-classifier is trained by the optimized loss function to obtain the final classifier; a multi-category classification module, used to perform multi-category classification in the multi-category scenario with the final classifier.
[0027] The present invention also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed, the multi-category classification method based on AUC optimization as described above is implemented.
[0028] The present invention also proposes a data processing device, comprising the computer-readable storage medium as described above. When the processor of the data processing device calls and executes the computer-executable instructions in the computer-readable storage medium, multi-category classification based on AUC optimization is implemented in a multi-category scenario. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is a flow chart of the multi-category classification method based on AUC optimization of the present invention.
[0030] Figure 2 It is a graph showing how the acceleration ratio of the present invention varies with sample size.
[0031] Figure 3 It is a schematic diagram of a data processing device of the present invention. DETAILED DESCRIPTION
[0032] The purpose of this invention is to achieve explicit AUC optimization in multi-class scenarios. First, based on the M-metric definition, an unbiased estimate of the alternative risk function on the training sample is constructed as the loss function to form a general framework for multi-classification AUC optimization. On this basis, an acceleration algorithm for loss and gradient is designed for square loss, exponential loss, and hinge loss to achieve efficient model training.
[0033] Based on the multi-classification AUC indicator M metric, the inventors designed the AUC loss function and optimization algorithm for multi-category scenarios, and achieved efficient model training by accelerating the calculation of the loss function and gradient.
[0034] The key points of the present invention are: 1. Multi-classification AUC optimization model; in terms of technical effect, an abstract framework for AUC optimization is formed, which is mainly divided into two parts: (a) designing multiple classifiers based on a one-to-many decomposition strategy (constructing classifiers based on deep neural networks), (b) constructing an unbiased estimate of the alternative risk function on training samples based on the M metric as the optimization objective function, and solving the classifier through an optimization algorithm. Subsequently, based on this abstract framework, a multi-classification AUC optimization method based on exponential loss, square loss, and hinge loss is specifically implemented; 2. Multi-classification AUC algorithm acceleration; in terms of technical effect, based on factor decomposition, matrix decomposition, dynamic programming and other technologies, an acceleration algorithm is implemented for AUC based on exponential loss, square loss, and hinge loss, respectively, which significantly reduces the computational complexity of the loss function and its gradient;
[0035] Figure 1 Flow chart of the multi-class classification method based on AUC optimization of the present invention. Figure 1 As shown, the present invention proposes a multi-classification method based on multi-classification AUC optimization, comprising the following steps:
[0036] S1, initial multi-classifier construction;
[0037] S2, construction of multi-classification AUC index;
[0038] S3, construction of multi-classification AUC loss function;
[0039] S4, multi-classification AUC loss and gradient calculation acceleration;
[0040] S5, model optimization, output of the final classifier;
[0041] S6. Use the obtained classifier to predict the scores of each category.
[0042] S0: First, two important events are introduced: ε ij Represents the event y1=i, y2=j∨y1=j, y2=i, ε i Indicates the event y1=i, y2≠i∨y1≠i, y2=i. Given the event I[A] represents an event The indicator function of If it holds, I[A] = 1, otherwise I[A] = 0. Given a finite data set N CIt represents the number of categories, and N is the total number of sample points in the data set.
[0043] S1: Multi-classifier construction
[0044] For a N C Classification problem, assuming that the sample is from the product space Obtained in. is the space of input features, where And d is the dimension of the input. For the label space [N C ]. Let Z i =(x i ,y i ) is the joint representation of the features and labels of the i-th sample, where y i Using one-hot vector Specifically, if the sample x i belongs to the kth category, then And for any l≠k, we have The present invention adopts one vs. all (OVA) decomposition to construct a classifier. In this case, N C Classifier of class (N C >2) refers to a set of scoring functions where f (i) : X→R, providing score prediction for the binary classification task of identifying the class label y=i.
[0045] In the specific implementation process, a deep multi-layer perceptron (MLP) is used to implement the classifier. C The i-th channel corresponds to f (i) .
[0046] S2: Multi-classification AUC indicator construction
[0047] First, the AUC index in the multi-class scenario is determined. Since the binary AUC index has been clearly defined, the present invention completes the AUC index design for multi-classification tasks by aggregating several binary AUC indexes.
[0048] S2-1: Pairwise AUC metric calculation
[0049] First, given a class pair (i, j), and the i-th subclassifier f (i) , calculate the corresponding binary AUC index AUC according to the following formula i|j (f (i) ):
[0050]
[0051] Related symbols I[·] and ε (ij) See S0, Δ(f (i) ) = f (i) (x1)-f (i) (x2).
[0052] S2-2: Pairwise AUC aggregation
[0053] Based on the calculation in S2-1, one vs. one (OVO) decomposition technique is used to give a classifier The multi-classification AUC index is reduced to the average of all pairwise binary classification AUC indicators, denoted as AUCM(f):
[0054]
[0055] It can be found that due to the use of OVO decomposition, CUCM(f) gives the same weight to all class pairs (i, j) and does not produce any bias in the class distribution. On the contrary, it can be proved that if other methods are used for decomposition, different weights will be generated for different class pairs (i, j), and the more samples there are in class i and class j, the larger the corresponding weight value will be. It can be seen that AUCM(f) avoids the class imbalance problem in a second-order form.
[0056] S3: Construction of multi-classification AUC loss function
[0057] The construction of the indicator AUCM(f) has been completed in S2. However, since AUCM(f) itself is a discrete function, it can only be directly optimized through a combinatorial optimization method, which is an NP-hard problem. In view of this, the present invention further designs an AUCM(f) proxy loss function framework to construct a continuously differentiable loss to replace the original indicator. The content in S3 lays the foundation for S4-S5 to further complete efficient model learning and optimization by optimizing the replacement loss.
[0058] S3-1: General framework
[0059] Ideally, we expect to directly maximize AUCM(f). Considering the risk minimization paradigm in machine learning, we can express this objective as a risk form R(f):
[0060]
[0061] Thus, the maximization problem is transformed into the minimization problem f∈argmin f R(f) is solved by:
[0062]
[0063] It is a 0-1 misalignment loss.
[0064] Due to the non-differentiability of the 0-1 loss function, the present invention further designs the R(f) proxy loss form. Specifically, using the differentiable loss function l φ Instead of l 0-1 Loss function, first get the proxy expected loss R φ (f):
[0065]
[0066] Further considering the unknown distribution of data and the limited number of actual training samples, it is impossible to directly obtain the expected risk. in is the training sample subset corresponding to category i, and for each category i, The corresponding number of samples is n i Based on the training samples, we use R φ The sample unbiased estimate of (f) As the final loss function. It is easy to get It can be expressed as the following formula:
[0067]
[0068] Where x1 and x2 are samples from two different categories i and j in the training samples.
[0069] Consider the learning process (i) It is parameterized by the model parameter space θ∈Θ, denoted by f (i) for Finally, the model is trained on the training set by solving the following optimization problem:
[0070]
[0071] S3-2: Implementing the framework based on square loss, exponential loss, and hinge loss
[0072] Based on the general framework, three specific implementations of the framework are given based on square loss, exponential loss, and hinge loss:
[0073] Squared loss:
[0074] Let l φ (t) = (1-t) 2 , with square loss
[0075]
[0076] Exponential loss:
[0077] Let l φ (t) = exp(-t), exponential loss
[0078]
[0079] Hinge loss:
[0080] Let l φ (t) = (1-t) + , where (x) + In the form of hinge loss, satisfying (x) + = x·I[x≥0], with exponential loss
[0081]
[0082] S4: Multi-classification AUC loss and gradient calculation acceleration
[0083] In S3, the loss function has been designed based on the practical point of view. However, due to the four-fold summation in the loss function, the computational complexity of the loss and gradient is and Where T loss , T grad The time complexity of calculating the loss and gradient of a single sample is respectively. In view of this, the present invention further constructs a loss and gradient calculation acceleration module, which accelerates the relevant calculations of square loss, exponential loss and hinge loss through matrix decomposition, factor decomposition, and dynamic programming. After optimization by this module, the loss (gradient) calculation complexity of square loss and exponential loss can be reduced to Where N is the total number of training samples, which reduces the computational complexity of hinge loss to in
[0084]
[0085] S4-1 Square Loss Acceleration
[0086] For category i, first construct the AUC comparison graph Its nodes are all the examples in the training sample, and its edges are represented by the following adjacency matrix Aff (i) Sure:
[0087] Aff (i) =D (i) (1-Y (i) Y (i)T +Y (i) (1-Y (i) ) T D (i) .
[0088] in Y (i) =[I[y1=i],…,I[y N =i]] T .
[0089] Further definition:
[0090]
[0091]
[0092]
[0093]
[0094]
[0095] Can This breaks down into:
[0096]
[0097] Among them, ⊙ is the Hadamard product, which means the operation of multiplying each sample by each other, and X is the matrix formed by concatenating the features of all N samples. Obviously, we first calculate κ (i) , Then calculate from the above formula , you can The calculation is completed in time. For gradient calculation, we further apply the chain rule in the above formula and perform calculations in a similar order to get time to complete a complete gradient calculation.
[0098] S4-2 Exponential Loss Acceleration
[0099] make:
[0100]
[0101] By factoring we can get:
[0102]
[0103] Obviously, first calculate a i , b i , and then calculate from the above formula The loss calculation can be accelerated. Based on the above formula, the corresponding gradient acceleration algorithm can be obtained through the composite derivation rule.
[0104] S4-3 Hinge loss acceleration
[0105] Given a category i, first All samples in f(i) (x) is sorted in descending order to obtain Then, right All samples in f (i) (x) is sorted in descending order to obtain
[0106] Given category i and positive examples Constructing an Activation Set
[0107]
[0108] Constructing recursive variables
[0109]
[0110]
[0111] The recursive equation is:
[0112]
[0113]
[0114] In sorting results Basically, only one sample traversal is needed to obtain all recursive variables.
[0115] Finally, it can be:
[0116]
[0117] Complete the calculation. Its time complexity depends on the sorting time complexity, that is,
[0118] For gradient calculation, additional recursive variables need to be constructed:
[0119]
[0120] in is the gradient of the classifier with respect to the parameter. There is a further recursive formula:
[0121]
[0122] Finally,
[0123]
[0124] The calculation is completed. Similarly, its computational complexity is
[0125] S5: Model optimization, output optimal classifier
[0126] According to the loss and gradient calculation module in S4, the optimizer is set according to the stochastic gradient descent algorithm, and the model parameters are optimized through continuous iteration until the AUC index on the validation set no longer increases significantly or begins to decrease. The iteration is terminated and the final classifier is output.
[0127] S6: Use the obtained classifier to predict the scores of each category
[0128] The classifier derived from S5 predicts the unknown sample x Select f (i) (x) The largest category i is taken as the final category label prediction result.
[0129] S7: Example
[0130] The proposed classification AUC optimization framework was experimented on some common classification datasets. The experiments compared methods based on different sampling strategies (BM, MM, IHT, NM, TL), methods based on different loss functions (FOCAL, CBCE, CBFOCAL, LDAM) and different AUC optimization methods (DeepAUC). The datasets used include:
[0131] CIFAR-100( https: / / www.cs.toronto.edu / ~kriz / cifar.html ) Image classification dataset, contains 100 classes, 600 images per class, and an unbalanced subset was sampled in the experiment.
[0132] User-Imb( https: / / www.kaggle.com / c / talkingdata-mobile-user- demographics / data ) dataset, which contains 12 target classes. The sub-tables describe the demographic characteristics of users (age, gender, etc.), and the original features include recorded events, application attributes, and device information. In the experiment, an unbalanced subset of the original dataset is sampled to generate a class-imbalanced dataset.
[0133] iNaturalist2017( https: / / www.kaggle.com / c / inaturalist-challenge-at- fgvc-2017 ) dataset is a large-scale image classification benchmark dataset with 675,170 images covering 5,089 different species of plants and animals. Since it is very time-consuming to directly train a deep model in such a large-scale dataset, the experiment uses a ResNet-50 model pre-trained on ImageNet for each image to generate 2048-dimensional features as the original features.
[0134]
[0135] Table 1
[0136] The comparison of experimental results on three imbalanced datasets is shown in Table 1. Table 1 shows the AUCM value. The higher the value, the better the performance.
[0137] The present invention compares three different comparison methods: sampling-based, loss function-based and AUC optimization-based. From the experimental results, it can be seen that the method proposed by the present invention to directly optimize the multi-classification AUC index is always better than all the comparison methods on all data sets.
[0138] In addition, the invention compares the efficiency of the proposed acceleration algorithm in the accelerated calculation of the loss function and the accelerated calculation of the gradient. Figure 2 The log of acceleration ratio of the proposed acceleration algorithm in loss calculation and gradient calculation for three different loss functions (square error loss, exponential loss and hinge loss) is recorded as the number of samples increases. As can be seen from the figure, as the sample size increases, the acceleration effect of the proposed acceleration algorithm gradually increases, thus verifying the high efficiency of the proposed algorithm.
[0139] Figure 3 Schematic diagram of the data processing device of the present invention. Figure 3 As shown, an embodiment of the present invention further provides a computer-readable storage medium, and a data processing device. The computer-readable storage medium of the present invention stores computer-executable instructions, and when the computer-executable instructions are executed by the processor of the data processing device, the above-mentioned multi-category classification method based on AUC optimization is implemented. A person of ordinary skill in the art can understand that all or part of the steps in the above method can be completed by instructing related hardware (such as a processor, FPGA, ASIC, etc.) through a program, and the program can be stored in a readable storage medium, such as a read-only memory, a disk or an optical disk. All or part of the steps of the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiment can be implemented in the form of hardware, such as implementing its corresponding functions through an integrated circuit, or in the form of a software function module, such as implementing its corresponding functions through a processor executing a program / instruction stored in a memory. The embodiment of the present invention is not limited to any specific form of combination of hardware and software.
[0140] The above implementation modes are only used to illustrate the present invention, but not to limit the present invention. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the present invention. The patent protection scope of the present invention should be defined by the claims.
Claims
1. A multi-class classification method based on AUC optimization, characterized in that: include: The initial multi-classifier for multi-category scenes of image classification is constructed with a deep multi-layer perceptron. The last layer of the initial multi-classifier is Output is a finite sample dataset for this multi-category scene The number of categories of this finite sample data set It is an image classification dataset; including: obtaining the binary classification AUC index of the multi-category scene , the multi-classifier AUC index Reduced to all pairwise binary classification AUC indicators The average value of: , Indicates Class samples and The AUC indicator between class samples, For the The output of each path corresponds to the sub-classifier; Construct a multi-classifier AUC indicator based on the AUC indicator of the multi-category scenario, and construct a loss function of the multi-classifier AUC indicator; the loss function includes: square loss , exponential loss and hinge loss ;in, ; ; ; express The subset of samples of the i-th category in , express The subset of samples of class j in , express The number of samples, express The number of samples, For For example, From , and , is the model parameter space After parameterization ; The form of hinge loss satisfies , Represents event x 0 indicator function, when x 0 Established =1, otherwise =0; Square loss Optimize to get ; For index loss Optimize to get ; For hinge loss Optimize to get ; The initial multi-classifier is trained through the optimized loss function to obtain the final classifier; where, Representation Matrix The transposed matrix of , , , , , , indicating the An intermediate recursive variable for class samples, Indicates that the The first samples, , indicating the Another intermediate recursive variable for class samples, Indicates all The matrix formed by concatenating the features of samples, , For sample About The activation set of the class; The final classifier is used to perform multi-category classification in the multi-category scenario.
2. The multi-category classification method according to claim 1, characterized in that: By square loss , exponential loss and hinge loss The initial multi-classifier is trained iteratively until two adjacent The difference is less than the preset threshold, and the final classifier is output .
3. The multi-category classification method according to claim 2, characterized in that: Through this final classifier Treat classification samples Making predictions , select The largest category As the sample to be classified The class label prediction results.
4. A multi-class classification system based on AUC optimization, characterized in that: include: The model building module is used to build an initial multi-classifier for multi-category scenes of image classification with a deep multi-layer perceptron. The last layer of the initial multi-classifier is Output is a finite sample dataset for this multi-category scene The number of categories of this finite sample data set It is an image classification dataset; including: obtaining the binary classification AUC index of the multi-category scene , the multi-classifier AUC index Reduced to all pairwise binary classification AUC indicators The average value of: , Indicates Class samples and The AUC indicator between class samples, For the The output of each path corresponds to the sub-classifier; A model training module is used to train the initial multi-classifier to obtain a final classifier; wherein, a multi-classifier AUC index is constructed according to the AUC index of the multi-category scene, and a loss function of the multi-classifier AUC index is constructed; the processing speed of the loss function is optimized; the initial multi-classifier is trained by the optimized loss function to obtain the final classifier; wherein the loss function includes: square loss , exponential loss and hinge loss ; ; ; ; Square loss Optimize to get ; For index loss Optimize to get ; For hinge loss Optimize to get ; express The subset of samples of the i-th category in , express The subset of samples of class j in , express The number of samples, express The number of samples, For For example, From , and , is the model parameter space After parameterization ; The form of hinge loss satisfies , Represents event x 0 indicator function, when x 0 Established =1, otherwise =0; Representation Matrix The transposed matrix of , , , , , , indicating the An intermediate recursive variable for class samples, Indicates that the The first samples, , indicating the Another intermediate recursive variable for class samples, Indicates all The matrix formed by concatenating the features of samples, , For sample About The activation set of the class; The multi-category classification module is used to perform multi-category classification in the multi-category scenario using the final classifier.
5. A computer-readable storage medium storing computer-executable instructions, which, when executed, implement the multi-category classification method based on AUC optimization as described in any one of claims 1 to 3.
6. A data processing device, comprising the computer-readable storage medium as described in claim 5, when the processor of the data processing device calls and executes the computer-executable instructions in the computer-readable storage medium, multi-category classification based on AUC optimization is implemented in a multi-category scenario of image classification.
