A method and device for tuning machine learning models based on ensemble learning

Abstract

This invention relates to a method and apparatus for adjusting a machine learning model based on ensemble learning. The method includes the following steps: constructing an initial target classification model; the initial target classification model is a machine learning model; using the BF method to iteratively adjust the initial target classification model for k rounds to reduce the bias of the initial target classification model; using the BBF method to adjust the initial target classification model multiple times to reduce the variance of the initial target classification model; averaging the output of the multiple adjustments to finally obtain the adjusted target classification model; and using an apparatus to implement the method. The beneficial effects of this invention are: improved performance of the machine learning model, better classification of image targets on datasets with limited data, and resolution of the overfitting problem of single models on these datasets, thereby improving the accuracy of target classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine learning, and in particular to a machine learning model adjustment method and device based on ensemble learning. BACKGROUND

[0002] With the rapid development of key technologies such as the Internet, cloud computing, and big data, artificial intelligence technology is profoundly changing people's lives and driving changes in all aspects of society, such as finance, entertainment, education, healthcare, and social interaction. Machine learning is an important part of artificial intelligence, aiming to learn the potential relationships between data from a large amount of data, and has great business mining potential. Many of these algorithms have penetrated into various branches of artificial intelligence, including autonomous driving, recommendation systems, facial recognition, natural language processing, and others.

[0003] Ensemble learning plays an important role in machine learning and pattern recognition, and has achieved great success in the business practices of major Internet companies and machine learning-related competitions, earning the title of "evergreen tree" in the field of machine learning.

[0004] Traditional machine learning methods mostly use a single model to output prediction results, such as support vector machine (SVM), the basic idea of which is to solve a separating hyperplane that can correctly divide the training data set and has the maximum geometric interval; regularized logistic regression (RLR), the basic idea of which is to minimize the cross-entropy loss of the output probability and the label, and prevent overfitting by constraining the two norm of the linear coefficient of the model; ridge regression (RR), the basic idea of which is to minimize the squared error of the output and the label, and also prevent overfitting by constraining the two norm of the linear coefficient of the model. Ensemble learning constructs multiple different base learners, and then fuses them through certain strategies to obtain a learner with stronger generalization performance. The related theory of ensemble learning points out that when the correlation between each base learner is low and the performance of each base learner is relatively reliable, better performance can be obtained after ensemble learning. The two most representative methods of ensemble learning are Boosting and Bagging. Boosting adopts an incremental learning process, in which each base learner learns on the basis of the existing model, and then adds the newly learned learner to the existing model. The existing LogitBoost algorithm is a representative Boosting method, which decomposes the classification problem into multiple regression problems, and then iteratively optimizes them. The newly generated learner pays more attention to the samples that are misclassified by the previous learners. Boosting is good at integrating weak learners (learners with large bias) to reduce the bias of the model. The Bagging algorithm trains multiple different base learners in parallel, and obtains the output through voting or averaging. Bagging is good at integrating unstable learners (learners with large variance) to reduce the variance of the model.

[0005] The classical methods in the field of machine learning, such as support vector machine (SVM), regularized logistic regression (RLR), ridge regression (RR) and the like, use a single model to output a prediction result. They belong to a small-bias and large-variance learner, and are excellent in many scenes, but still have obvious limitations. On the one hand, they only use a single model to output a prediction result, and have problems of abnormal value sensitivity and large generalization error. Many cases show that the output of an integrated multiple learners is better than that of a single learner. On the other hand, since the fitting performance of the learners is better than that of a weak learner, the bias of a single model is already small, and simple use of a boosting algorithm for integration is easy to cause overfitting, so it is not suitable to use boosting to integrate them. However, if bagging is used to integrate them, since the single learners are relatively stable, the generalization performance of the integrated model cannot be well improved by introducing sample diversity through bagging.

[0006] The traditional model adjustment method is mainly applied to neural networks. Since the neural network updates the weights of each hidden layer and output layer through gradient descent, such a training method is very suitable for adjustment. However, for the non-differentiable model such as kernel SVM, gradient descent cannot be used for adjustment, and in the case of small amount of data, kernel SVM is a very strong benchmark, and most methods are difficult to surpass kernel SVM. However, using a single learner for prediction, the output is still easily affected by noise points, resulting in poor model generalization ability. For RLR and RR, although the gradient descent method can also be used for adjustment, the model after training is already globally optimal, and under the condition of not changing the loss function, it is impossible to obtain a better model by adjustment, and after adjustment, only one model is used for prediction, so the output result is also easily affected by noise points. When training a model on a medical disease classification data set, since the data amount of the data set is usually small and has noise, it is difficult to select appropriate model parameters, so using a single model is easy to cause overfitting, resulting in poor classification effect. SUMMARY

[0007] To solve the above technical problems, the present application first adjusts the classifier by a boosting fine-tuning (BF) algorithm to reduce the bias of the model, and then combines bagging and boosting fine-tuning (BBF)

[0008] The algorithm performs multiple adjustments to the classifier and fuses the final outputs to reduce model variance. It further improves model performance by first reducing model bias and then model variance. In classification tasks, for an initial model, such as kernel SVM, RLR, or RR, its predictions are integrated and improved using Boosting and Bagging. During BF adjustment, residuals are calculated based on the initial learner's predictions and the true labels. Then, the classification problem is decomposed into multiple regression problems, and regression models are trained to fit the residuals. In each Boosting iteration, pseudo-labels are recalculated based on the residuals between the current model's predictions and the true labels, and the regression model is trained using these pseudo-labels.

[0009] This invention provides a method for tuning a machine learning model based on ensemble learning, which specifically includes the following steps:

[0010] S101: Construct an initial target classification model; the initial target classification model is a machine learning model;

[0011] S102: The initial target classification model is adjusted through k iterations using the BF method to reduce the bias of the initial target classification model;

[0012] S103: The BBF method is used to adjust the initial target classification model multiple times to reduce the variance of the initial target classification model;

[0013] S104: Average the output of the multiple adjustments in step S103 to obtain the final adjusted target classification model.

[0014] Further, in step S102, the k-round iterative adjustment specifically includes:

[0015] S201: Randomly generate a matrix W1, randomly map the original features, and input the mapped features into the activation function δ to obtain H1=δ[z(X,W1)]; where z() represents the normalization operation; X represents the input original features;

[0016] S202: Input the training set D into the initial target classification model to obtain the sample weights B1 and pseudo-labels. ;

[0017] S203: Combine H1, sample weights B1, and pseudo-labels. The first base learner L1 of a BF learner is input to obtain sample weights B2 and pseudo-labels. The BF learner includes k base learners, namely L1-L... k ;

[0018] S204: Enter the second round of iteration, randomly generate a matrix W2, and input H2=delta[z(X, W2)], sample weight B2 and pseudo label to the second base learner L2 of the BF learner, obtain sample weight B3 and pseudo label ;

[0019] S205: Continue steps S203-S204 until k rounds of iteration, input H k =delta[z(X, W k )], sample weight B k and pseudo label to the kth base learner L k ;

[0020] S206: Sum the outputs of the initial model, the k base learners L1-L k , to obtain the output result after reducing the bias of the initial target classification model.

[0021] Further, in step S103, the multiple adjustments specifically refer to: in step S102, only one BF learner is trained, and when multiple adjustments are performed, k different BF learners are trained, and the average outputs of the k different BF learners are outputted.

[0022] Further, when training the k different BF learners, k different training subsets D1-D k are randomly generated from the training set D, and correspond to the training of the k different BF learners.

[0023] A machine learning model adjustment device based on ensemble learning, comprising a processor and a storage device; the processor loads and executes the instructions and data in the storage device to realize a machine learning model adjustment method based on ensemble learning.

[0024] The beneficial effects provided by the present application are: the performance of the machine learning model is improved, which can better classify the image target classification problem with small amount of data, solves the problem that a single model is prone to overfitting on these data sets, and thus improves the accuracy of target classification. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a flowchart of the method of the present application;

[0026] Figure 2 is a process diagram of k rounds of iteration adjustment of the initial target classification model using the BF method;

[0027] Figure 3 is a schematic diagram of integrated multiple adjustments of the initialization model using BBF;

[0028] Figure 4 is the experimental data result graph of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical scheme and advantages of the present application more clear, the embodiments of the present application will be further described below with reference to the drawings.

[0030] Please refer to Figure 1 , Figure 1 is a method flowchart of the present application; the present application provides a machine learning model adjustment method and device based on ensemble learning, including the following steps:

[0031] S101: constructing an initial target classification model; the initial target classification model is a machine learning model;

[0032] As an embodiment, the machine learning model can be a kernel SVM, LR and RC machine learning model, etc.

[0033] S102: using the BF method to perform k rounds of iterative adjustment on the initial target classification model to reduce the bias of the initial target classification model;

[0034] As an embodiment, please refer to Figure 2 , Figure 2 is a process diagram of using the BF method to perform k rounds of iterative adjustment on the initial target classification model; the diagram represents using Boosting to iterate k rounds for fine-tuning. In each round of boosting iteration, a matrix W is first randomly generated to randomly map the original features, then the features are input into the ReLU activation function, and finally a Boosting learner is trained according to the Boosting calculated pseudo label and sample weight using the ridge regression model. The role of introducing the ReLU activation function is to introduce nonlinear features to the model, and the random mapping of the original features is to improve the instability of the model to facilitate the next step of using Bagging for integration. The output of the final model is the sum of the output values of the initialized model and all Boosting learners.

[0035] Specifically, in step S102, the k rounds of iterative adjustment are as follows:

[0036] S201: randomly generating a matrix W1 to randomly map the original features, and inputting the mapped features into an activation function δ to obtain H1=δ[z(X,W1)]; wherein z() represents a normalization operation; X represents the input original features;

[0037] S202: inputting the training set D into the initial target classification model to obtain sample weight B1 and pseudo label ;

[0038] S203: inputting H1, sample weight B1 and pseudo label to a first base learner L1 of a BF learner to obtain sample weight B2 and pseudo label ; the BF learner comprises k base learners, respectively L1-L k ; the base learner refers to a base learner of the BF algorithm, which can be an SVR or a ridge regression learner;

[0039] S204: entering a second round of iteration, randomly generating a matrix W2, and inputting H2=δ[z(X,W2)], sample weight B2 and pseudo label to a second base learner L2 of the BF learner to obtain sample weight B3 and pseudo label ;

[0040] S205: continuing steps S203-S204 until k rounds of iteration, inputting H k =δ[z(X,W k )] to the k base learners L1-L k to obtain sample weight B k and pseudo label ;

[0041] S206: summing the outputs of the initial model and the k base learners L1-L k to obtain an output result after reducing the bias of the initial target classification model.

[0042] Next, the initial target classification model is further adjusted multiple times.

[0043] Please refer to Figure 3 , Figure 3 is a schematic diagram of using BBF to integrate and adjust the initial model multiple times; the diagram shows that Bagging is used to integrate k BF learners to fine-tune the initial model, wherein each BF learner comprises multiple Boosting learners. Since the features and labels used to train each Boosting learner are different, the BF learner is a very diverse model, which is very suitable for integration using Bagging. The output of the model is the mean of the output probabilities of all BF learners.

[0044] Specifically, refer to steps S103-S104;

[0045] S103: using the BBF method to adjust the initial target classification model multiple times to reduce the variance of the initial target classification model; in step S103, the multiple adjustments specifically refer to: in step S102, only one BF learner is trained, and when multiple adjustments are performed, k different BF learners are trained, and the average output of the k different BF learners is output. When training the k different BF learners, k different training subsets D1-D k , corresponding to the training of the k different BF learners.

[0046] S104: average output of the multiple adjustments in step S103 to obtain the adjusted target classification model.

[0047] A machine learning model adjustment device based on ensemble learning, comprising a processor and a storage device; the processor loads and executes instructions and data in the storage device to implement a machine learning model adjustment method based on ensemble learning.

[0048] As an example, please refer to Figure 4 , Figure 4 The mean and standard deviation of 10 repeated experiments on 42 UCI data sets are shown. The relevant parameter explanations are as follows: std: standard deviation; RBF-SVC: RBF kernel SVM classifier; RLR: regularized logistic regression; RR: ridge regression algorithm.

[0049] The present application uses BF and BBF to fine-tune RBF kernel SVM, LR and RC, respectively, the horizontal coordinate is different initialization model; the vertical coordinate is the accuracy according to the class proportion balance. The experimental results show that the BBF fine-tuning can significantly improve the classification performance of the initial learner on the small data image target classification problem, solving the problem of easy overfitting of single model on these data sets, thereby improving the accuracy of target classification.

[0050] The beneficial effects of the present application are: improving the performance of the machine learning model, which can better classify the small data image target classification problem, solving the problem of easy overfitting of single model on these data sets, thereby improving the accuracy of target classification.

[0051] The above only describes the preferred embodiments of the present application and does not limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An integrated learning-based machine learning model adjustment method, characterized in that: The method comprises the following steps: S101: constructing an initial target classification model; The initial target classification model is a machine learning model; The target of the target classification model comprises an image classification target and a medical classification target; S102: performing k rounds of iterative adjustment on the initial target classification model by using a BF method to reduce the bias of the initial target classification model; S103: performing multiple adjustments on the initial target classification model by using a BBF method to reduce the variance of the initial target classification model; S104: averaging the multiple adjustments in step S103 to obtain a final adjusted target classification model; In step S102, the k rounds of iterative adjustment are specifically as follows: S201: randomly generating a matrix W1, randomly mapping original features, inputting the mapped features into an activation function δ to obtain H1=δ[z(X,W1)]; wherein z() represents a normalization operation; X represents input original features; S202: input the training set D into the initial target classification model to obtain sample weight B1 and pseudo label ; the training set adopts a UCI data set; S203: input H1, sample weight B1 and pseudo label to a first base learner L1 of a BF learner, to obtain sample weight B2 and pseudo label ; the BF learner comprises k base learners, respectively L1-L k ; S204: Enter the second round of iteration, randomly generate a matrix W2, and input H2= δ[z(X, W2)], sample weight B2 and pseudo label to the second base learner L2 of the BF learner, and get sample weight B3 and pseudo label ; S205: Continue steps S203~204 until iteration k rounds, put H k = δ[z(X, W k )], sample weight B k and pseudo label into the k-th base learner L k ; S206: summing the outputs of the initial model, the k base learners L1-L k to obtain an output result after reducing the bias of the initial target classification model; In step S103, the multiple adjustments are specifically as follows: in step S102, only one BF learner is trained, and when multiple adjustments are performed, k different BF learners are trained, and the average output of the k different BF learners is outputted; When training k different BF learners, k different training subsets D1-Dk are randomly generated from the training set D k corresponding to the training of k different BF learners.

2. A machine learning model adjustment device based on ensemble learning, characterized by: The device comprises a processor and a storage device; the processor loads and executes instructions and data in the storage device to realize the method for adjusting a machine learning model based on ensemble learning in claim 1.

Images