Method for optimizing learning performance of long-tail machine based on hierarchical KL divergence model

Abstract

The invention belongs to the technical field of machine learning algorithms, and particularly relates to a method for optimizing long-tail machine learning performance based on a hierarchical KL divergence model. The method comprises the following steps: designing a hierarchical KL-RS model, and carrying out equivalent dual conversion; a hierarchical KL-RS model is solved based on a search and gradient descent method; a composite loss function combining an Ldam loss function and a Focal loss function is adopted in the long-tail machine learning process, so that the average accuracy of the model and the accuracy of the model in the worst category are improved; experimental results based on data sets CIFAR10 (LT) and CIFAR100 (LT) show that the method of the invention can effectively improve the average accuracy and the worst accuracy of the neural network.

Description

Technical Field

[0001] The present invention belongs to the technical field of long-tail machine learning, and in particular relates to a method for optimizing long-tail machine learning performance based on a hierarchical KL divergence model. Background Art

[0002] Long-tail machine learning is a machine learning task in which the training data has a long-tail feature. The long-tail feature is a distribution characteristic commonly found in real-world data. It describes the phenomenon of imbalanced training data categories: training data categories usually include head categories and tail categories; each category in the head category contains a large number of samples, but the number of head categories is relatively small compared to the total number of categories; while each category in the tail category has only a small number of samples, but the majority of categories in the total number of categories are tail categories[1]. Traditional machine learning methods often tend to ignore the performance of the model on the tail categories, causing the trained model to be heavily biased towards the head type, and the accuracy and generalization performance on the tail categories are significantly reduced[2].

[0003] Class rebalancing is currently an important approach to addressing the challenge of long-tail distributions. Its core idea is to adjust the model performance bias caused by class imbalance by adjusting the sample distribution during training or changing the loss function. This method is simple and efficient and can be directly embedded into conventional machine learning optimization processes, making it easy to implement and deploy. There are two main class rebalancing methods: one is the resampling method, whose basic idea is to reduce the sampling frequency of the top classes and increase the sampling frequency of the tail classes, so that the tail samples can be fully learned by the model [3-5]. However, this method often leads to overfitting the model to the tail classes. The other is the class-sensitive learning method, whose basic idea is to design a class-sensitive loss function. Traditional methods use the same loss function for all classes, but class-sensitive loss functions differentiate the loss functions for different classes. Representative works include the label distribution aware margin loss function (Ldam) and the focal loss function [2,6].

[0004] Considering the following characteristics of long-tail distribution learning: on the one hand, the training set data of long-tail learning exhibits long-tail distribution characteristics, but the test set data exhibits uniform distribution characteristics. Solving the challenge of long-tail learning is actually solving the challenge of distribution shift; on the other hand, improving the performance of the model in the tail category is actually improving the performance of the model in the worst-performing category, which is the requirement of robust optimization.

[0005] To this end, the present invention designs a hierarchical KL divergence robustly satisfying model (KL Robust Satisficing, KL-RS) that can cope with the challenge of distribution shift, and proposes a method for solving the model based on search and gradient descent; further, by replacing the cross-entropy loss function of the hierarchical KL-RS mathematical optimization model with the loss functions Ldam and Focal, the KL-RS Ldam and KL-RS Focal mathematical optimization model methods are proposed. Summary of the Invention

[0006] The purpose of the present invention is to provide a method for optimizing long-tail machine learning performance based on a hierarchical KL divergence robustness model with high accuracy.

[0007] The present invention provides a method for optimizing the performance of long-tail machine learning based on a hierarchical KL divergence robustness model, comprising designing a hierarchical KL-RS model and performing an equivalent dual transformation; solving the hierarchical KL-RS model using a search-based and gradient descent method; and using a composite loss function combining an Ldam loss function and a Focal loss function in the long-tail machine learning process to improve the average accuracy of the model and the accuracy on the worst category. The specific steps are as follows:

[0008] (1) Design a hierarchical KL-RS model and perform equivalent dual transformation;

[0009] For a long-tail distribution learning task, it can be modeled into the following mathematical form:

[0010] Let Θ represent the set of model parameters, l:Θ×R n+1 →R is the cross entropy loss function; for any θ∈Θ and (x,y)∈R n+1 , l(θ,(x,y)) is the cross entropy loss of model θ for data sample (x,y). Treat the data as random variables and use Represent the joint random variable of features and labels, with Denotes a realization of the random variable. Consider N data samples: Use these samples to construct the empirical distribution Right now The usual empirical risk minimization (ERM) model has the following mathematical form:

[0011]

[0012] The objective function in this formula indicates that the data follows the distribution The expectation E of the loss function when .

[0013] When the machine learning task based on this model faces a long-tail distribution of training data, the model's performance will be biased towards the head category, and the model's prediction accuracy for the tail category is low and its generalization ability is weak. In order to address the shortcomings of the ERM model when facing long-tail distribution data, the present invention adopts the KL divergence robust satisfaction optimization model (KL Robust Satisficing, KL-RS) to solve the problem. The model has the following mathematical form:

[0014]

[0015] in, Is any relative to An absolutely smooth distribution, i.e. The support set of A subset of the support set of ; express Relative to KL divergence; τ is a hyperparameter set by the decision maker, reflecting the loss that the decision maker can accept; λ is the objective function of optimization, which reflects the robustness of the machine learning model to distribution shifts. The smaller λ is, the more robust the machine learning model is. This mathematical optimization model uses an affine function about the degree of distribution shift to control the performance of the model on different distributions, and controls the speed at which the performance of the trained model decreases as the distribution decreases by minimizing the slope λ of the affine function, thereby improving the robustness of the model. Taking into account the characteristic that data can be grouped according to labels in long-tail distribution learning, the present invention further proposes a hierarchical KL divergence robust satisfaction optimization model, recorded as a hierarchical KL-RS model. The hierarchical KL-RS model has a similar principle to the KL-RS model, but can allow decision makers to control the model's resistance to different distribution shifts according to demand, specifically the marginal probability distribution shift of the label and the conditional probability shift of the feature relative to the label. The hierarchical KL-RS model has the following mathematical form:

[0016]

[0017] The two variables λ1 and λ2 are used to measure the model's vulnerability to shifts in the marginal probability distribution and the model's vulnerability to shifts in the conditional probability distribution. Smaller values mean the model is more resistant to distribution shifts. w is a non-negative hyperparameter that acts as a weight. When the value of w increases, the hierarchical KL-RS model focuses more on the model's ability to resist shifts in the conditional probability distribution. In particular, when w = ∞, the hierarchical KL-RS model only focuses on the model's robustness to shifts in the conditional probability distribution. When the value of w decreases, the hierarchical KL-RS model focuses more on the model's ability to resist shifts in the marginal probability distribution. In particular, when w = 0, the hierarchical KL-RS model only focuses on the model's robustness to shifts in the marginal probability distribution.

[0018] The original mathematical form of the hierarchical KL-RS model requires that for any This is equivalent to P being an optimization variable. This will lead to the appearance of bilinear terms on the left side of the constraint, making the optimization problem non-convex, even if the loss function is a convex function about θ. In order to avoid the optimization difficulties caused by this non-convexity, the present invention converts the hierarchical KL-RS model into the following mathematical form based on the duality property:

[0019]

[0020] Among them, the distribution P is completely eliminated, when the empirical distribution When the model is determined, it is a completely deterministic model. The derivation of its dual equivalent form is based on the following conclusions [7-9]:

[0021]

[0022] Specifically, the first step is to shift the constraints of the hierarchical KL-RS model to obtain the following equivalent transformation:

[0023]

[0024] Using the dual transformation, this constraint is equivalent to:

[0025]

[0026] By moving the terms again, we get the following equivalent form:

[0027]

[0028] Apply the dual transformation again to get the final result.

[0029] (2) Solution of Hierarchical KL-RS Model Based on Search and Gradient Descent

[0030] When the loss function is a neural network and the data scale is relatively large, the solution of the equivalent dual model (4) requires the help of the gradient descent algorithm. However, since the expectation of the loss is in the constraint, there is no way to directly use the gradient descent algorithm. In order to solve this model, the present invention proposes to use the golden section search method to solve the hierarchical KL-RS model, referred to as Algorithm1. The algorithm is proposed based on a property. When the parameter θ is given, the mathematical model is a convex optimization problem about (λ1,λ2), and the optimal (λ1,λ2) can be easily found by searching. When (λ1,λ2) is given, the expression on the left side of the constraint can be used as the objective function to perform gradient descent on θ to find the optimal θ. * , and based on the optimization results, determine whether the current (λ1,λ2) is a feasible solution to the original problem. If it is, further search will make (λ1,λ2) smaller; if it is not, further search will make (λ1,λ2) larger. This process repeats until the algorithm reaches the termination condition.

[0031] In general, the optimization process of the model is that Algorithm1 performs a golden section search for λ1 and calls an algorithm based on bisection to solve the optimal λ2 (fixed λ1) (referred to as Algorithm2) to find the best λ2. For any pair (λ1, λ2), the algorithm will call the feasibility test algorithm for the parameters (λ1, λ2) (referred to as Algorithm3). The essence of this algorithm is to use the stochastic gradient descent algorithm (referred to as Algorithm4) for the conditional optimization problem of the parameters (λ1, λ2) to optimize and then determine whether there is θ that makes the problem solvable.

[0032] Specifically:

[0033] Algorithm 1, performs a golden section search on λ1. The algorithm first sets an upper bound λ on the search interval. r and the lower bound λ l , and calculate the current smaller left golden section point λ′ l and the right golden section point λ′ with a larger value r , where the latter is greater than the former. And find the corresponding optimal by calling Algorithm2 at two points and Calculate and compare the two target values. If a smaller value can be obtained at the left golden section point, set the search upper bound to the right golden section point λ r , otherwise the lower bound is set to the left golden section point λ l The search is then repeated within the new upper and lower bounds until the algorithm converges. See Appendix 1 for the pseudocode of Algorithm 1.

[0034] Algorithm2, performs a binary search on λ2 given λ1. The algorithm starts from an initial positive number λ0 and a lower bound λ = 0 to start searching, and then call Algorithm3 to check whether the current (λ1,λ0) is a feasible solution to the original problem. If it is a feasible solution, set the upper bound to Otherwise, the lower bound is set to λ0 and the upper bound is set to 2λ0. Calculate the intermediate value λ between the upper and lower bounds mid , calling Algorithm 3 to check whether the intermediate value is a feasible solution. If not, update the upper bound to the intermediate value; if so, update the lower bound to the intermediate value. Repeat this process until the algorithm terminates. See Appendix 2 for the pseudocode of Algorithm 2.

[0035] Algorithm 3 is used to determine whether the current solution (λ1,λ2) is a feasible solution to the original problem. Specifically, it calls Algorithm 4 to conditionally optimize the objective function. The pseudocode for Algorithm 3 is shown in Appendix 3.

[0036] Algorithm 4 performs conditional random optimization on θ given (λ1,λ2). This algorithm is a modification of the algorithm mentioned in papers [10-12]. In each iteration of the original algorithm, only one sample of the outer random variable is sampled; this algorithm changes it to M1 samples, where M1 is greater than or equal to 2. This is because, in experiments, it was found that although the original algorithm had convergence guarantees, it failed to converge in practice. However, after modifying it to sample several samples, convergence was guaranteed. This may be because the mean and variance of data from different categories vary greatly, and sampling only one category in each round is prone to non-convergence. The pseudocode of Algorithm 4 is shown in Appendix 4.

[0037] In summary, the above solution method is to use monotonicity and convexity to efficiently search (λ1,λ2) and use the gradient descent algorithm to determine whether (λ1,λ2) is a feasible solution.

[0038] The hierarchical KL-RS model proposed in this paper can be applied to machine learning tasks using any loss function. Conventional machine learning for classification tasks uses the cross-entropy loss function. To further improve the performance of long-tail learning tasks, this paper replaces the conventional cross-entropy loss function with the Ldam loss function and Focal loss function, which are more suitable for long-tail learning.

[0039] Ldam loss function definition: For a classification task with k categories, assuming that the number of training samples for each category is n j ,remember C is a constant; model f has k outputs, where the j-th output represents the probability that the model determines the sample to be of the j-th class. For sample (x, y), use z j =f(x) j Represents the j-th output of the model, then the Ldam loss function is:

[0040]

[0041] Among them, Δ y is Δ j The category in is the value of the category y of the sample itself;

[0042] Definition of Focal loss function: Assume that the probability of the jth class predicted by the model for sample (x, y) is 0≤p j ≤1, then the Focal loss function is:

[0043] FL(p j )=-(1-p j ) γ log(p j ),#(6)

[0044] Here, γ is a hyperparameter.

[0045] Experimental results based on the CIFAR10(LT) and CIFAR100(LT) datasets show that the method proposed in this paper can effectively improve the average accuracy and worst case accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is the amount of data collected for the CFIAR10(LT) dataset training data when ρ = 0.1.

[0047] Figure 2 is the amount of data collected for training data of the CFIAR10(LT) dataset when ρ = 0.01.

[0048] Figure 3 is the amount of data collected for training data of the CFIAR100(LT) dataset when ρ = 0.1.

[0049] Figure 4 is the amount of data collected for training data of the CFIAR100(LT) dataset when ρ = 0.01.

[0050] Figure 5 It is the t-SNE graph of the features extracted by the neural network trained by the ERM model in the worst category and other categories of the test set.

[0051] Figure 6It is the t-SNE graph of the features extracted by the neural network trained by the hierarchical KL-RS model in the worst category and other categories of the test set.

[0052] Figure 7 This is the t-SNE graph of features extracted by the neural network trained by the KL-RS Focal model in the worst category and other categories of the test set.

[0053] Figure 8 This is the t-SNE graph of features extracted by the neural network trained by the KL-RS Ldam model in the worst category and other categories of the test set. DETAILED DESCRIPTION

[0054] The present invention is further described below through experimental examples in conjunction with the accompanying drawings.

[0055] The experiment was conducted on the classic long-tail distribution learning datasets CIFAR10(LT) and CIFAR100(LT). The dataset was obtained by downsampling the CIFAR10 and CIFAR100 datasets. The downsampling process is controlled by the parameter ρ, where ρ represents the ratio of the number of samples of the smallest category to the number of samples of the largest category in the training data. The smaller ρ is, the more obvious the long-tail characteristics of the training data are. In the downsampling process, the number of samples from the head category to the tail category presents a geometric progression. The experiment of this invention selected two different downsampling coefficients of ρ = 0.1 and ρ = 0.01, and the number of training data samples for each category is referenced. Figure 1 , Figure 2 , Figure 3 and Figure 4 .

[0056] The test set data is evenly sampled from each category.

[0057] This paper uses the following evaluation metrics: average accuracy and worst-case accuracy. Average accuracy is the most commonly used metric for evaluating classification models and comprehensively reflects a model's classification capabilities. Worst-case accuracy refers to the accuracy of the category with the lowest classification accuracy. The accuracy of the model in the worst-case category reflects the model's ability to extract features and classify tail categories.

[0058] The experiments were conducted on Ubuntu Server 18.04.01, using an Intel(R) Xeon(R) W-2140B CPU @ 3.20 GHz and 32 GB of memory, with 8 cores and 16 threads. The GPU was a GeForce GTX1080Ti, and CuDA version 11.4 was used. The runtime environment was based on Python 3.9.7, and the network model used, ResNet18, and training were both implemented and performed using the PyTorch 1.12.1 framework.

[0059] The experimental steps of the present invention are as follows: for each algorithm, the random number seed is set to 0, 1, 2, 3, and 4 respectively, and five models are trained. The mean and standard deviation of the performance of the five models on the test set are recorded. The main impact of the random number seed on the training process is reflected in the following aspects: the random initialization parameters of the neural network, the sampling process of the mini-batch stochastic gradient descent algorithm, and the sampling of the test data set and the downsampling of the training data set when constructing the long-tail distribution data sets CIFAR10(LT) and CIFAR100(LT).

[0060] Model Training Settings: ERM, KL-RS, and CVaRDRO all use the cross-entropy loss function. In the experiments, the models were trained using the stochastic gradient descent (SGD) optimizer provided by the PyTorch library. All algorithms use SGD for optimization, continuously sampling mini-batches of training data to calculate gradients and using these gradients to update the network. The calculated gradients vary between algorithms due to different loss functions or aggregation methods. Overall, each algorithm was trained for 200 epochs. The optimization step size was 0.1 at the beginning of training and was reduced to 0.01 after 160 epochs. The momentum of the SGD optimizer was set to 0, indicating that the experiment used the most common SGD optimizer. The weight decay parameter (Weight Decay) was set to 0.0002, which adds an additional L2 regularization term to the objective loss function. The batch size was set to 1024.

[0061] Experimental Results Analysis: The experimental results are shown in Table 1, with the standard deviations of the various indicators in parentheses. First, a comparison is made between the three aggregation methods: ERM, KL-RS, and CVaRDRO. When ρ = 0.1, the three methods perform very similarly on the CIFAR10(LT) dataset, but KL-RS far surpasses the other two methods in terms of worst-case accuracy on CIFAR100(LT). When ρ = 0.01, KL-RS's average accuracy is 20% higher than the other two, and its worst-case accuracy is three times that of ERM and five times that of CVaRDRO, a very significant improvement.

[0062] Then we consider composite methods. All composite methods have improved performance compared to the original single replacement loss function methods. When ρ = 0.1, the most obvious improvement is the worst accuracy indicator on the CIFAR100 (LT) dataset, and other indicators have a small improvement. When ρ = 0.01, both indicators of the CIFAR10 (LT) dataset have improved significantly. KL-RS Ldam achieved the highest average accuracy and worst category accuracy on the CIFAR10 (LT) dataset. Similarly, in the CIFAR100 (LT) dataset, the best performing models are KL-RS Focal and KL-RS Ldam.

[0063] Furthermore, when the imbalance of the long-tail distribution increases (i.e., ρ becomes smaller), the KL-RS model's improvement over the baseline model becomes even more significant. Specifically, when the imbalance factor ρ = 0.01, we observe that the average accuracy of the KL-RSLdam model compared to other methods improves by approximately 14% on the CIFAR10(LT) dataset and by approximately 13% on the CIFAR100(LT) dataset. This result further validates the robustness and effectiveness of the KL-RS method in dealing with extreme long-tail distribution scenarios.

[0064] Furthermore, the experiment of this invention compares the ability of models trained by several methods to extract features on CIFAR10(LT) with ρ=0.01, and uses t-SNE technology to reduce the dimension of the features extracted by ERM, KL-RS, KL-RS Focal and KL-RS Ldam models and displays the results of the dimension reduction. Figure 5 , Figure 6 , Figure 7 as well as Figure 8 Since this is an analysis of the worst-performing class, all other classes are represented by the same color, and the worst-performing class 10 is represented by a red dot. Figure 5 This is the result of the ERM method. It can be seen that the worst-performing class is completely mixed in with the data of other classes, and the model hardly extracts any features that can distinguish the 10th class from other classes. This is also the reason why the worst accuracy of the ERM method is very low. Figure 6 and Figure 7 The results of the KL-RS method and the KL-RS Focal method show that a large number of points are still mixed with other types of data, but there is an area on the right side of the figure where there is almost no other type of data. This means that the two methods can extract some unique features of the 10th type of data, which is why the worst accuracy of the two methods has improved. Figure 8This is the result of the KL-RS Ldam method. Compared to the previous two methods, the area in the upper right corner of the image, which consists almost entirely of data points from category 10, has become larger. This change is more pronounced in the plots where samples from each category are plotted in different colors during the experiment. This means that KL-RS Ldam extracts more features that distinguish category 10 from other categories, further improving the model's ability to represent tail categories. This is why KL-RS Ldam has the highest worst-case accuracy of all methods.

[0065]

[0066] Table 1

[0067] appendix:

[0068] Appendix 1: Pseudo code for the algorithm (Algorithm 1) for solving the hierarchical KL-RS model based on the golden section search method.

[0069]

[0070] Appendix 2: Pseudo code of the algorithm (Algorithm2) for solving the optimal λ2 (fixed λ1) based on bisection.

[0071]

[0072] Appendix 3: is the pseudo code of the feasibility test algorithm (Algorithm3) for parameters (λ1,λ2).

[0073]

[0074] Appendix 4: Pseudocode for the stochastic gradient descent algorithm (Algorithm 4) for the conditional optimization problem with parameters (λ1,λ2)

[0075] References

[0076] [1]Yifan Zhang,Bingyi Kang,Bryan Hooi,Shuicheng Yan,and Jiashi Feng.Deep long-tailed learning:A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence,2023.

[0077] [2]Kaidi Cao,Colin Wei,Adrien Gaidon,Nikos Arechiga,and TengyuMa.Learning imbalanced datasets with label-distribution-aware margin loss.InAdvances in neural information processing systems,volume 32,2019.

[0078] [3]Nitesh V Chawla,Kevin W Bowyer,Lawrence O Hall,and W PhilipKegelmeyer.Smote:synthetic minority over-sampling technique.Journal ofartificial intelligence research,16:321–357,2002.

[0079] [4]Andrew Estabrooks,Taeho Jo,and Nathalie Japkowicz.A multipleresampling method for learning from imbalanced data sets.Computationalintelligence,20(1):18–36,2004.

[0080] [5]Xu-Ying Liu,Jianxin Wu,and Zhi-Hua Zhou.Exploratory undersamplingfor class-imbalance learning.IEEE Transactions on Systems,Man,andCybernetics,Part B(Cybernetics),39(2):539–550,2008.

[0081] [6]Tsung-Yi Lin,Priya Goyal,Ross Girshick,Kaiming He,and Piotr Dollár.Focal loss for dense object detection.In Proceedings of the IEEEinternational conference on computer vision,pages 2980–2988,2017.

[0082] [7]Hans and Alexander Schied.Convex measures of risk andtrading constraints.Finance and stochastics,6:429–447,2002.

[0083] [8]Alexander Shapiro,Darinka Dentcheva,and AndrzejRuszczynski.Lectures on stochastic programming:modeling and theory.SIAM,2021.

[0084] [9]Hans and Alexander Schied.Stochastic finance:an introductionin discrete time.Walter de Gruyter,2011.

[0085]

[10] Yifan Hu,Xin Chen,and Niao He.Sample complexity of sample averageapproximation for conditional stochastic optimization.SIAM Journal onOptimization,30(3):2103–2133,2020.

[0086]

[11] Yifan Hu,Siqi Zhang,Xin Chen,and Niao He.Biased stochastic first-order methods for conditional stochastic optimization and applications inmeta learning.In Advances in Neural Information Processing Systems,volume 33,pages 2759–2770,2020.

[0087]

[12] Yifan Hu,Xin Chen,and Niao He.On the bias-variance-cost tradeoffof stochastic optimization.In Advances in Neural Information ProcessingSystems,volume 34,pages22119–22131,2021。

Claims

1. A method for optimizing long-tail machine learning performance based on a hierarchical KL divergence model, characterized in that: This involves designing a hierarchical KL-RS model and performing an equivalent dual transformation. The hierarchical KL-RS model is solved using a search-based gradient descent method. A composite loss function combining the Ldam loss function and the Focal loss function is used in the long-tail machine learning process to improve the model's average accuracy and the accuracy in the worst category. The specific steps are as follows: (1) Design a hierarchical KL-RS model and establish an equivalent dual transformation; For a long-tail distribution learning task, it is modeled into the following mathematical form: Let Θ represent the set of model parameters, l:Θ×R n+1 →R is the cross entropy loss function; for any θ∈Θ and (x,y)∈R n+1 , l(θ,(x,y)) is the cross entropy loss of model θ for data sample (x,y); treat the data as random variables and use Represent the joint random variable of features and labels, with represents a realization of the random variable; consider N data samples: Use these samples to construct the empirical distribution Right now The model based on the principle of empirical risk minimization (ERM) has the following mathematical form: The objective function in the formula indicates that the data follows the distribution The expectation E of the loss function when When the training data for machine learning tasks based on this model presents a long-tail distribution, the model's performance will be biased towards the head category, and the model's prediction accuracy for the tail category will be low and its generalization ability will be weak. To address the shortcomings of the ERM model in dealing with long-tail distribution data, the KL divergence-based robust satisfactory optimization model (KL-RS) is used to solve the problem. The mathematical form of this model is: in, Is any relative to An absolutely smooth distribution, i.e. The support set of A subset of the support set of ; express Relative to KL divergence; τ is a hyperparameter set by the decision maker, reflecting the loss that the decision maker can accept; λ is the optimization objective function, reflecting the robustness of the machine learning model to distribution shifts. The smaller λ is, the more robust the machine learning model is. This mathematical optimization model uses an affine function about the degree of distribution shift to control the performance of the model on different distributions, and controls the speed at which the performance of the trained model decreases as the distribution decreases by minimizing the slope λ of the affine function, thereby improving the robustness of the model. Taking into account the characteristics of data grouped according to labels in long-tail distribution learning, a hierarchical KL divergence robust satisfaction optimization model is further proposed, denoted as the hierarchical KL-RS model. The hierarchical KL-RS model has a similar principle to the KL-RS model, but allows decision makers to control the model's resistance to different distribution shifts according to needs, specifically the marginal probability distribution shift of the label and the conditional probability shift of the feature relative to the label. The hierarchical KL-RS model has the following mathematical form: Among them, the two variables λ1 and λ2 are used to measure the vulnerability of the model to the shift of the marginal probability distribution and the vulnerability of the model to the shift of the conditional probability distribution. A smaller value means that the model has stronger resistance to distribution shifts. w is a non-negative hyperparameter used as a weight. When the value of w increases, the hierarchical KL-RS model pays more attention to the ability of the model to resist the shift of the conditional probability distribution. In particular, when w = ∞, the hierarchical KL-RS model only focuses on the robustness of the model to the shift of the conditional probability distribution. When the value of w decreases, the hierarchical KL-RS model pays more attention to the ability of the model to resist the shift of the marginal probability distribution. In particular, when w = 0, the hierarchical KL-RS model only focuses on the robustness of the model to the shift of the marginal probability distribution. The original mathematical form of the hierarchical KL-RS model requires that for any All of them hold true, which is equivalent to P being an optimization variable; this will lead to the appearance of bilinear terms on the left side of the constraint, making the optimization problem non-convex, even if the loss function is a convex function about θ; in order to avoid the optimization difficulties caused by this non-convexity, the hierarchical KL-RS model is converted into the following mathematical form according to the duality property: Among them, the distribution P is completely eliminated, when the empirical distribution When it is determined, the model is a completely deterministic model; (2) Search and gradient descent based solution of the hierarchical KL-RS model; To solve the equivalent dual model (4), we first use the golden section search method, referred to as Algorithm 1. This algorithm is based on a property: when the parameter θ is given, the mathematical model is a convex optimization problem about (λ1,λ2), and the optimal (λ1,λ2) can be found by searching. When (λ1,λ2) is given, the expression on the left side of the constraint is used as the objective function to perform gradient descent on θ to find the optimal θ. * , and based on the optimized results, determine whether the current (λ1,λ2) is a feasible solution to the original problem; if it is a feasible solution, further search will make (λ1,λ2) smaller; if it is not a feasible solution, further search will make (λ1,λ2) larger; this process is repeated until the algorithm termination condition is reached; The model solving process is that Algorithm1 performs a golden section search on λ1 and calls an algorithm based on bisection to solve the optimal λ2 (fixed λ1), denoted as Algorithm2, to find the best λ2 for any pair (λ1,λ2); the algorithm calls a feasibility test algorithm for the parameters (λ1,λ2), denoted as Algorithm3, which uses the stochastic gradient descent algorithm (Algorithm4) for the conditional optimization problem of the parameters (λ1,λ2) to optimize and then determines whether there is θ that makes the problem solvable.

2. The method for optimizing long-tail machine learning performance according to claim 1, characterized in that: In step (2): The algorithm 1 performs a golden section search on λ1; first, an upper bound λ is set for the search interval. r and the lower bound λ l , and calculate the current smaller left golden section point λ′ l and the right golden section point λ′ with a larger value r , where the latter is greater than the former; and the corresponding optimal one is found by calling Algorithm 2 at two points and Calculate and compare the two target values. If a smaller value can be obtained at the left golden section point, set the search upper bound to the right golden section point λ r , otherwise the lower bound is set to the left golden section point λ l ; Then repeat the search within the new upper and lower bounds until the algorithm converges; The algorithm 2 performs a binary search on λ2 given λ1; first, from an initial positive number λ0 and a lower bound λ =0 to start searching, and then call Algorithm 3 to detect whether the current (λ1,λ0) is a feasible solution to the original problem; If it is a feasible solution, set the upper bound to Otherwise, the lower bound is set to λ0 and the upper bound is set to 2λ0; Calculate the intermediate value of the upper and lower bounds λ mid , call Algorithm 3 to check whether the intermediate value is a feasible solution. If not, update the upper bound to the intermediate value; If yes, update the lower bound to the middle value; repeat the above process until the algorithm termination condition is reached; Algorithm 3 is used to determine whether the current (λ1,λ2) is a feasible solution to the original problem. Specifically, it calls Algorithm 4 to conditionally optimize the objective function. The algorithm 4 performs conditional random optimization on θ given (λ1,λ2). Algorithm 4 is a modification of the original algorithm. In each round of iteration, the original algorithm only samples one sample of the outer random variable. Algorithm 4 is modified to: sample M1 samples of the outer random variable, where M1 is greater than or equal to 2.

3. The method for optimizing long-tail machine learning performance according to claim 2, characterized in that: In step (2), in order to further improve the performance of long-tail learning tasks, a composite loss function combining the Ldam loss function and the Focal loss function is used; Ldam loss function definition: For a classification task with k categories, assuming that the number of training samples for each category is n j ,remember C is a constant; model f has k outputs, where the j-th output represents the probability that the model determines the sample to be of the j-th class; for sample (x, y), z is used j =f(x) j Represents the j-th output of the model, then the Ldam loss function is as follows: Among them, Δ y is Δ j The category in is the value of the category y of the sample itself; Definition of Focal loss function: Assume that the probability of the jth class predicted by the model for sample (x, y) is 0≤p j ≤1, then the Focal loss function is: FL(p j )=-(1-p j ) γ log(p j ),#(6) Among them, γ is a hyperparameter.

Images