A general ensemble method for multi-classification with adaptive weights

Abstract

The present invention provides a general ensemble method for multi-classification with adaptive weights. Different from the usual ensemble classification method that only targets one classification task, the present invention faces different classification problems. First, the weight allocation coefficient is determined. Then, the exponential function distribution characteristics are used to automatically calculate the weight of a model fusion according to the model evaluation index and the weight allocation coefficient. Finally, the fusion weight is adaptively adjusted through a continuous iteration method to achieve model fusion under different classification tasks. Experiments show that the method of the present invention can achieve model fusion on 9 data sets with different numbers of classifications, different fields, and different scales, and its fusion effect is better than the baseline method on most tasks. Compared with a single model, the F1 value of the result after fusion using the method of the present invention has steadily increased by 3‑25 percentage points; compared with other ensemble methods, the F1 value has steadily increased by 1‑2 percentage points, which proves the versatility and effectiveness of the method of the present invention.

Description

Technical Field

[0001] The present invention relates to the technical field of data mining and data classification processing, and in particular to a multi-classification universal integration method with adaptive weights. Background Art

[0002] The main idea of ​​ensemble learning is group decision-making, which combines the results of multiple weak learners through various combination mechanisms to obtain better results than any individual component algorithm, thereby improving accuracy and stability.

[0003] In current research, most methods focus on adjusting the weights in the integration process to obtain good classification results. In general, the existing research can be divided into methods in the training process and methods after training. Among them, the research on implementing fusion in the training process can be traced back to 1990. Freund and Shapire proposed the Boosting algorithm, which trains multiple weak classifiers, assigns different weights to each weak classifier, and adjusts its weights and votes to achieve classification. However, the Boosting algorithm requires knowing the classification error rate of the learner in the worst case in advance, which is difficult to do; AdaBoost integration method, Stacking method, etc. were proposed subsequently; the above methods need to be integrated in the training process, so each integration requires one training, which will take a lot of time. In order to save a lot of training time, some researchers have proposed some integration methods after the model training is completed. This type of method is usually divided into two stages. The first stage selects the base classifier, and the second stage uses the output results of the classifier training to improve the accuracy of the results through appropriate fusion rules. Compared with the integration method in the training process, the integration method after training can flexibly adjust the fusion strategy, which is convenient for finding a better fusion method. However, the current integration methods are all full integration methods, that is, all classifiers are integrated in the process of implementing the integration. However, as the number of classifiers continues to increase, the computing and storage overheads will continue to rise, while the integration effect will not improve.

[0004] In addition, existing ensemble methods generally only attempt to improve or adjust classification models to solve data classification problems in a specific field. There is currently no good general classification method that can be applied to multiple data types in different fields (such as news sentiment analysis data, Twitter sentiment analysis data, vegetation cover type data, Mushrooms dataset, Spambase dataset, Nursery dataset, Breast Cancer dataset, Wine dataset, and Iris dataset). Summary of the invention

[0005] In view of the problem that the existing integration methods cannot adaptively select integration weights when facing multiple classification problems and are not universal, the present invention provides a multi-classification universal integration method with adaptive weights.

[0006] The present invention provides a multi-classification universal integration method with adaptive weights, comprising:

[0007] Step 1: Build n base models and divide the given data set into training set and test set;

[0008] Step 2: Use the training set to train each base model, use each trained base model to predict the test set, and obtain the classification label and label probability corresponding to each base model; and calculate the F1 value of each base model according to the prediction results;

[0009] Step 3: Calculate the weight of each base model according to the F1 value of each base model and the current weight allocation coefficient;

[0010] Step 4: Perform weighted fusion according to the label probability and weight of each base model to obtain the fused label probability and the corresponding classification label, and calculate the corresponding F1 value;

[0011] Step 5: Update the weight adjustment coefficient according to the given step size, and iterate step 3 to step 4;

[0012] Step 6: Sort all fused label probabilities and corresponding classification labels according to the size of all F1 values ​​obtained in step 5, select n label probabilities and corresponding classification labels with larger F1 values ​​as the label probabilities and corresponding classification labels corresponding to n base models, and save the maximum F1 value;

[0013] Step 7: Iterate steps 3 to 6 until the current maximum F1 value is less than the maximum F1 value in the previous iteration, and take the classification label corresponding to the maximum F1 value in the previous iteration as the final classification result.

[0014] Furthermore, the type of the data set is one of News Sentiment, Twitter Sentiment, Covertype, Mushrooms, Spambase, Nursery, Breast Cancer, Wine and Iris.

[0015] Furthermore, each base model is trained using the training set by means of K-fold cross validation.

[0016] Furthermore, step 3 specifically includes:

[0017] According to the F1 value of each base model, the normalized F1 value is calculated according to formula (1):

[0018]

[0019] Among them, f i represents the F1 value of the i-th base model, f i * represents the normalized F1 value of the i-th base model, f min Indicates the smallest F1 value among all base models, f max Refers to the maximum F1 value among all base models;

[0020] According to the normalized F1 value, the weight of each base model is calculated according to formula (2):

[0021]

[0022] in, w i represents the weight of the i-th base model, and α represents the weight adjustment coefficient.

[0023] Furthermore, the value range of the weight adjustment coefficient is specifically α∈[t,2t,3t,…,5(n-1)], where t is a given step length.

[0024] Furthermore, in step 4, weighted fusion is performed according to formula (3):

[0025]

[0026] Among them, w i represents the weight of the i-th base model, p i Represents the label probability of the i-th base model.

[0027] Beneficial effects of the present invention:

[0028] (1) Different from the common method of distributing weights evenly or proportionally, the present invention adjusts the distribution of weights by changing the value of the weight allocation coefficient α, so that more weight combinations can be selected during model fusion, increasing the possibility of obtaining better weights. At the same time, since the method of the present invention uses the characteristics of the exponential function, the weight calculation can converge faster during the process of changing the α value.

[0029] (2) Due to the key role of the weight allocation coefficient α in the method of the present invention, the value of α becomes very important. The present invention studies the influence of the change of the value of α on the fusion effect by introducing the step size t. In Section (II), the present invention proves through experiments that the smaller the t value (that is, the smaller the initial value and the change step size of α), the wider the weight distribution range obtained, and the better the fusion effect. However, when t is reduced to a certain value, the improvement of the fusion effect will become very slow. This is because when the t value is very small, the change of α is very small. From the weight calculation formula (2), it can be seen that the smaller the change of α, the smaller the change of weight, and the fusion results of similar weight combinations are almost the same. Therefore, taking a suitable t value can not only reduce the amount of calculation, but also ensure that it does not affect the effect of model fusion.

[0030] (3) During the integration preparation process, the present invention uses K The fold cross validation is used to increase the stability of the prediction results of the base model, which increases the possibility of better results from model fusion. It is understandable that since ensemble preparation and model fusion are two relatively independent processes, it is not used. K The fusion method proposed in the present invention can also be used to perform model fusion on the base model trained by the fold cross validation. Therefore, the model fusion method proposed in the present invention can be flexibly applied to other scenarios as long as the user provides the classification labels of the classification model on the test set and the corresponding label probability values.

[0031] (4) The present invention selected three base models to conduct unit experiments and comparative experiments on nine data sets with different numbers of classifications, different fields, and different scales. The experimental results show that the method of the present invention can achieve relatively good fusion effects on each data set, proving that the method of the present invention is effective and can be applied to the integration of multiple classification models, which can provide a reference for the optimization of other classification problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 One of the flow charts of a multi-classification universal integration method with adaptive weights provided by an embodiment of the present invention;

[0033] Figure 2 A second flow chart of a multi-classification universal integration method with adaptive weights provided by an embodiment of the present invention;

[0034] Figure 3 is a schematic diagram of the fitting curve;

[0035] Figure 4 A schematic diagram of an integration preparation process provided by an embodiment of the present invention;

[0036] Figure 5The first round of fusion results on D1, D2, and D3 when t=0.02, 0.04, 0.1, 0.2, 0.5, and 1 are provided in the embodiment of the present invention, where the ellipse marks the part F1 The interval with larger value;

[0037] Figure 6 This is a comparison chart of the F1 values ​​of the base model and UMEAW on the D1-D9 datasets provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] The embodiment of the present invention provides a multi-classification universal integration method with adaptive weights, the framework is as follows Figure 1 As shown in the figure, the framework consists of two parts: ensemble preparation and model fusion; ensemble preparation refers to the process of obtaining predicted classification labels and label probabilities from a single classifier, and model fusion refers to the process of obtaining fusion results from individual classifier results through fusion strategies. Figure 1 and Figure 2 As shown, the embodiment of the present invention specifically includes the following steps:

[0040] S101: Build n base models and divide the given data set into a training set and a test set;

[0041] Specifically, a single model participating in model fusion is called a base model, denoted as B-model. The type of the data set is one of News Sentiment, Twitter Sentiment, Covertype, Mushrooms, Spambase, Nursery, Breast Cancer, Wine and Iris.

[0042] S102: using the training set to train each base model, using each trained base model to predict the test set, to obtain the classification label and label probability corresponding to each base model; and calculating the F1 value of each base model according to the prediction result;

[0043] Specifically, the way the base classifier training is handled will affect the model fusion effect. In the application of deep learning, overfitting occurs in many cases, which is manifested as the learner parameters overfitting the training data set during the training process, thereby affecting the generalization performance of the model on the test data set. The fundamental reason is that the amount of training data is not enough to support complex models, which leads to noise in the data set learned by the model. However, if the results are averaged, the overfitting phenomenon can be reduced to a certain extent. Figure 3 As shown in the figure, due to overfitting, the decision boundary of a single model is gray, but in fact, the black decision boundary has better generalization ability, so it works better. If multiple models are fitted and the results are averaged, the consideration of these noise points will be reduced due to the average of the results, and the decision boundary will gradually move closer to the black line.

[0044] Therefore, in order to obtain a stable fusion result, as an implementation method, each base model is trained using the training set by means of K-fold cross validation. Figure 4 As shown, first, the training set is evenly divided into K groups, and create a validation set for each subset of data, and then use the remaining K -1 subset of data is used for training. In this way, we will get K A set of prediction results. K The probability of the group prediction label is averaged to obtain the prediction result of the base model, that is, the averaged probability and the corresponding label. K The value is determined according to the number of categories. For example, when facing a 3-category problem, it is generally K It is taken as 5. This can avoid the extreme situation where each data is divided into its own category, and also prevent the fold from being too large to waste computing resources.

[0045] In classification problems, in order to conveniently represent different categories, each category is usually given a label, which is called a classification label and is represented by label in the present invention. Classification problems with more than or equal to 2 classification labels are collectively referred to as multi-classification problems. In the process of obtaining the prediction result, the classification model generates a probability value corresponding to each classification label, and then determines the final result by the size of the probability value corresponding to the label. The probability value obtained in this process is called label probability in the present invention.

[0046] F1 is an indicator for evaluating the quality of the model. This value combines the evaluation advantages of precision and recall, and can better evaluate the experimental results. The calculation formula is in, TP is True Positive, FP is False Positive, and FN is False Negative.

[0047] S103: Calculate the weight of each base model according to the F1 value of each base model and the current weight allocation coefficient;

[0048] Specifically, the weight of model fusion is very important, and the weight distribution directly affects the model fusion effect. Therefore, how to quickly and effectively find a better weight combination is very critical.

[0049] As an implementable method, this step includes the following sub-steps:

[0050] Step S1031: Calculate the normalized F1 value of each base model according to the F1 value of each base model according to formula (1):

[0051]

[0052] Among them, f i represents the F1 value of the i-th base model, f i * represents the normalized F1 value of the i-th base model, f min Indicates the smallest F1 value among all base models, f max Refers to the maximum F1 value among all base models; it can be understood that but

[0053] Step S1032: Calculate the weight of each base model according to the normalized F1 value according to formula (2):

[0054]

[0055] in, w i represents the weight of the i-th base model, and α represents the weight adjustment coefficient.

[0056] Specifically, in the embodiment of the present invention, on the one hand, an exponential function g(x)=e is introduced. x ,Will As the numerator, the weight can be positively correlated with the size of the model F1 value. At the same time, with the rapid growth of the exponential function, the weight calculation can converge quickly, thus solving the problem of slow convergence of weight calculation. On the other hand, the weight allocation coefficient α is also introduced, that is, the numerator of the nth model needs to be multiplied by the coefficient α, that is Its purpose is to adjust the weight of the nth model (the model with the largest F1 value). When the α value increases, the model will also obtain a larger weight, and the weights of other models will decrease accordingly. Due to the existence of the weight allocation coefficient α, the weight distribution of the model can be adjusted by changing the value of this coefficient, so that the model can obtain as many different weights as possible, thereby solving the problem of easily ignoring potential better weights when calculating weights.

[0057] And, add the numerators of all models as the denominator, that is This approach ensures that the sum of all scores is equal to 1, which is consistent with the original setting.

[0058] The basic principle followed by the weight calculation method given by formula (2) is to assign corresponding weights to the models according to the size of the F1 value of the models, that is, the models with larger F1 values ​​get larger weights, and the models with smaller F1 values ​​get smaller weights. At the same time, the case of assigning larger weights to the models with smaller F1 values ​​is also considered.

[0059] S104: Perform weighted fusion according to the label probability and weight of each base model to obtain the fused label probability and the corresponding classification label, and calculate the corresponding F1 value;

[0060] Specifically, weighted fusion is performed according to formula (3):

[0061]

[0062] Among them, w i represents the weight of the i-th base model, p i Represents the label probability of the i-th base model.

[0063] For the x classification problem, each sample corresponds to x label probabilities. Assuming there are y samples, these y samples have x×y label probabilities, which can be expressed in the form of a matrix, as shown in formula (4).

[0064]

[0065] Then the label probability matrix after the weighted fusion of n models can be expressed as:

[0066]

[0067] The final classification label result is the label corresponding to the maximum probability value of each sample, as shown in formula (6):

[0068] label k =argmax(p′ k1 ,p′ k2 ,…,p′ kx )k=1,2,…,y (6)

[0069] In order to facilitate the subsequent description, the labels of all samples are represented by vectors as L = [label1, label2, ..., label k ,…,label y ] T .

[0070] S105: Update the weight allocation coefficient according to the given step size, and iteratively execute steps S103 to S104;

[0071] Specifically, as can be seen from formula (2), the value of α directly determines the calculation result of the weight. By changing the value of this coefficient, the weight distribution of the model can be adjusted so that the model can obtain as many different weights as possible, thus solving the problem that it is easy to ignore potential better weights when calculating weights. Therefore, in order to obtain a better model fusion effect, it is necessary to further determine the value range of α, and then update its value within this value range.

[0072] As an implementable manner, the value range of the weight allocation coefficient is specifically α ∈ [t, 2t, 3t,..., 5(n - 1)], where t is the given step size.

[0073] Specifically, since f i * is the value after normalization, its value range is [0, 1]. Therefore the value range of is [1, e], then the value range of is [n - 1, (n - 1)e]. To simplify the calculation amount, take the approximate value of as Then formula (2) can be simplified to formula (7).

[0074]

[0075] Next, mainly determine its value range according to the influence of the value of α on w n (the maximum weight). First, let the value of α be a multiple of n - 1. At this time, it is very easy to reduce formula (7)-② to a constant. When α = 5(n - 1), w n = 5e / (5e + 3 / 2) ≈ 0.9. At this time, the weights of other models are already very small. If the value of α is continued to increase, the role of other models in the fusion process will be negligible. Therefore, take α = 5(n - 1) as the maximum value of α. When α = 1, w n is equal to the value calculated proportionally according to the F1 value of the model. When α < 1, w n can obtain smaller values, and w i , (i = 1, 2,..., n - 1) can obtain larger values. Therefore, in order to expand the value range of the weight and assign larger weights to worse models, this method adds the parameter t (0 < t < 1) as the minimum value of α. At the same time, for the convenience of calculating the intermediate value of α, take t as the change step size of α.

[0076] At this point, the value range and number of α can be determined. The value range is: [t, 2t, 3t, …, 5(n-1)], and the number of values ​​is 5(n-1) / t, where n is the number of models.

[0077] It should be noted that formula (2) is the final weight calculation formula, and formula (7) is a simplified formula for explaining the value of α.

[0078] S106: sort all fused label probabilities and corresponding classification labels according to the sizes of all F1 values ​​obtained in step S105, select n label probabilities and corresponding classification labels with larger F1 values ​​as label probabilities and corresponding classification labels corresponding to n base models respectively, and save the maximum F1 value;

[0079] S107: Iterate steps S103 to S106 until the current maximum F1 value is less than the maximum F1 value in the previous iteration, and take the classification label corresponding to the maximum F1 value in the previous iteration as the final classification result.

[0080] In the embodiment of the present invention, the adaptive weight is obtained by formula (2). On this basis, in order to obtain a better model fusion result, the result obtained by the integration preparation is used as input data, and multiple weighted fusions are performed by continuously iterating steps 3 to 6 to achieve an improvement in the fusion effect.

[0081] In order to verify the effectiveness of the multi-classification integration method provided by the present invention, the present invention also provides the following experiments.

[0082] 1. Experimental Setup

[0083] (1) Dataset

[0084] The experimental data comes from 9 different data sets, among which the data set for news sentiment analysis (News Sentiment) comes from the 2019CCF BDCI news sentiment analysis track, and the other data sets used to test the effectiveness of the method are all common public data sets in the field of machine learning. Table 1 shows the name of each data set, its ID in the present invention, the number of classifications, the total number of samples, and the number of data attributes. Since data sets D2-D9 are well-known data sets in the field of machine learning, the present invention will no longer conduct a detailed analysis of these data sets, but only introduce the situation of the News Sentiment data set. The attributes contained in this data set are: news ID number (id), news title content (title), news text content (content), and news sentiment label (label). The positive emotion of the label corresponds to 0, the neutral emotion corresponds to 1, and the negative emotion corresponds to 2.

[0085] For each data set, the training set and the test set are divided into a ratio of 7:3. Then the training set is divided into K parts. If K is less than or equal to the number of categories, there may be an extreme situation where each divided data is a separate category. In order to avoid this situation, the data sets D1, D2, D4, D5, D7, D8, and D9, whose number of categories is less than 5, are randomly divided into 5 parts, and D3 and D6 are randomly divided into 10 parts for the integration preparation process.

[0086] Table 1 Datasets used in the present invention

[0087] Dataset ID Classification Books Total number of samples Number of data attributes News Sentiment D1 3 7357 3 Twitter Sentiment D2 2 31,962 3 Covertype D3 7 581,012 55 Mushrooms D4 2 8124 21 Spambase D5 2 4601 58 Nursery D6 5 12,960 8 Breast Cancer D7 2 699 32 Wine D8 3 178 13 Iris D9 3 150 4

[0088] (2) Evaluation indicators

[0089] The F1 value is used as the evaluation index. This value combines the evaluation advantages of precision and recall, and can better evaluate the experimental results. The calculation method has been introduced in the above content, so it will not be repeated here.

[0090] It is worth noting that the multi-classification task calculates the Macro-F1 value, specifically: first calculate the F1 value of each category of labels, and then average the F1 values ​​of each category of labels to get the Macro-F1 value.

[0091] (3) Baseline method

[0092] (3.1) Selecting the base model

[0093] A base model with higher accuracy and diversity can help ensemble learning achieve better generalization accuracy. Therefore, when selecting a base model, it is necessary to consider both the accuracy and diversity of the model.

[0094] Since D1 involves multiple fields such as literature, politics, and news reporting, the traditional deep learning model (such as RNN / CNN) is abandoned as the base model for the analysis task of the D1 dataset, and the Transformer architecture model is selected. This experiment selects BERT (Reference 1: DEVLIN J, CHANG M, LEE K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprintarXiv:1810.04805, 2018), RoBERTa (Reference 2: CUI Y, CHE W, LIU T, et al. Pre-training with whole word masking for chinesebert[J]. arXiv preprint arXiv:1906.08101, 2019), and XLNet (Reference 3: YANG Z, DAI Z, YANG Y, et al. XLNet: Generalized Autoregressive Pretraining for Language Understanding[J]. Advances in Neural Information Processing Systems, 2019, 32: 5753-5763) and softmax as the base model. The model weight file of this experiment comes from the project published on GitHub. Since the D2-D9 datasets are more mature in the application of traditional machine learning models, and the training time of traditional machine learning models is relatively short, traditional machine learning models such as logistic regression (LR), support vector classifier (SVC) and random forest (RF) are selected as base models. This experiment implements the above three models by calling the sklearn package.

[0095] (3.2) Selecting a baseline method

[0096] We selected several classic full integration methods: Weigting average (Wavg), Majority voting (Mv), Weighted majority voting (Wmv) (Reference 4: LITTLESTONE N, WARMUTH M K. The Weighted Majority Algorithm [J]. Proceedings of the Second Annual Workshop on Computational Learning Theory, 1994, 108 (2): 388), Stacking (Stk) (Reference 5: WOLPERT DH. Stacked generalization [J]. Neural Networks, 2017, 5 (2): 241-259.) and the classic selective integration method GAS-EN (Reference 6: NGUYEN TT, LIEW AWC, TRAN MT, et al. Combining multiclassifiers based on a genetic algorithm–a gaussian mixture model framework [C] / / International Conference on Intelligent Computing. Cham: Springer, 2014: 56-67), and the recently proposed selective ensemble method AE (Reference 7: YANG J, WANG F. Auto-Ensemble: An Adaptive Learning Rate Scheduling Based Deep Learning Model Ensembling [J]. IEEE Access, 2020, 8: 217499–217509) are used as baseline methods for comparison with the method of the present invention (abbreviated as UMEAW). Among them, Weigting average, Majority voting, Weighted majority voting, and Stacking are implemented using the library provided by sklearn, GAS-EN is implemented by the open source code published by Harry Mills, and AE is implemented according to the algorithm proposed in the paper.

[0097] (4) Parameter settings

[0098] For the Transformer architecture models such as BERT, RoBERTa, and XLNet, the main parameters in the training process are shown in Table 2. The training parameters for the logistic regression, support vector classifier, and random forest models use the default parameters provided by the sklearn library.

[0099] Table 2 Main parameter settings of Transformer architecture model

[0100] parameter value max_seq_length 256 split_num 3 eval_steps 200 per_gpu_eval_batch_size 32 num_train_epochs 3 learning_rate 5e-6 lstm_hidden_size 512 lstm_layers 1 lstm_dropout 0.1 train_steps 5000

[0101] 2. Analysis of the influence of step length t on fusion effect

[0102] The weight allocation coefficient α directly affects the calculation result of the weight. Since α∈[t,2t,3t,…,5(n-1)], and the number of base models selected in this experiment is 3, α∈[t,2t,3t,…,10], so the value of α is only determined by t. Therefore, it is very necessary to conduct an in-depth analysis of the value of t.

[0103] In order to facilitate analysis and drawing, this experiment selected data sets D1, D2, and D3 with large differences in F1 values ​​to test the impact of the value of t on the fusion results.

[0104] First, we need to select the value of t to be tested in the experiment. As mentioned above, when t is 0.02, the minimum value of α is 0.02. At this time, according to the weight calculation formula (2), the weight of the model corresponding to the maximum F1 value is about 0.03, which accounts for a very small proportion in model fusion. When the value of t is gradually increased, the weight of the model corresponding to the maximum F1 value also increases. Therefore, t = 0.02 is used as the minimum value in the experiment of this section, and then the value of t is gradually increased to carry out the experiment. Take t = 0.02, 0.04, 0.1, 0.2, 0.5, 1.

[0105] In order to intuitively see the impact of the t value on model fusion, the experiments in this section only perform the first round of fusion in UMEAW and do not iterate multiple times. The experimental results on each dataset are shown in Figure 2. Figure 5 shown.

[0106] from Figure 5 It can be seen that no matter how large t is, the range of α values ​​when the model fusion effect is better is relatively fixed (the position circled by the ellipse in the figure), indicating that AW can better capture the better weights of weighted fusion.

[0107] At the same time, it can be seen from the figure that the smaller the t value is, the smaller the change in the value of α is, the more model fusion results are, and the better the optimal result is. For example, when t = 0.02 and t = 0.04, there are more prominent fusion results in the results of D3, but when t>0.1, these better results are lost. This shows that if you want to get a good fusion result, t should take a smaller value.

[0108] However, the value of t should not be too small. A too small t value not only increases the amount of calculation, but also slows down the process of obtaining a good weight combination. By observing the two graphs of t = 0.02 and t = 0.04, it can be found that the outlines of the two graphs are basically the same, and the important nodes in the t = 0.02 graph are not lost in the t = 0.04 graph. This shows that when t = 0.04, a good weight combination can be obtained, and it is not meaningful to continue to reduce the t value. Based on the above analysis, this experiment will select t = 0.04 for subsequent experiments.

[0109] 3. Analysis of the Impact of Iteration on Model Fusion

[0110] Based on the first round of fusion in Section (II), this section completes the entire UMEAW process and analyzes the impact of iteration on the fusion results. In Section (II), when t=0.04, after the first round of fusion, the maximum F1 values ​​of D1, D2, and D3 are 0.822, 0.888, and 0.773, respectively. After UMEAW fusion, the maximum F1 values ​​on the three datasets have all improved. The F1 values ​​of D1, D2, and D3 stopped iteration after 10, 13, and 8 iterations, respectively, and obtained the final F1 values ​​of 0.8287, 0.783, and 0.8972. The F1 values ​​obtained for each iteration of the three datasets are shown in Table 3.

[0111] As can be seen from the table, the F1 value of each iteration has a small improvement, indicating that the iteration process is effective for model fusion. However, compared with the fusion effect of the first round, the improvement of the F1 value of each subsequent iteration is relatively small. The possible reason is that the first round directly uses the base model for weighted fusion, and each base model has a large difference, so the F1 value improves greatly after fusion. The iterative operation starting from the second round is based on the results of the fusion of three base models, and the difference becomes smaller, resulting in a small improvement in the fusion effect.

[0112] Table 3 F1 values ​​obtained for each iteration of D1, D2 and D3

[0113]

[0114] 4. Analysis of the influence of base model on fusion effect

[0115] This section uses ablation experiments to explore the contribution of each base model to the model fusion results. Table 4 shows the F1 values ​​obtained by using UMEAW to fuse the remaining base models and the F1 values ​​obtained by using all three base models after removing different base models on the D1-D9 datasets. Among them, the base models of D1 are BERT, RoBERTa and XLNet, and the base models of D2-D9 are logistic regression, support vector classifier and random forest. In the table, BM1, BM2 and BM3 are used to represent the results of the three base models, No-BM1, No-BM2 and No-BM3 are used to represent the results of removing the corresponding base models, and All represents the results of fusing all base models.

[0116] Table 4 F1 value of ablation experiment

[0117] Dataset ID BM1 / No-BM1 BM2 / No-BM2 BM3 / No-BM3 All D1 0.803 / 0.818 0.812 / 0.813 0.802 / 0.819 0.829 D2 0.753 / 0.759 0.749 / 0.764 0.665 / 0.778 0.783 D3 0.697 / 0.876 0.773 / 0.887 0.869 / 0.853 0.897 D4 0.914 / 0.951 0.934 / 0.943 0.931 / 0.946 0.957 D5 0.904 / 0.947 0.697 / 0.951 0.945 / 0.948 0.953 D6 0.931 / 0.961 0.944 / 0.966 0.964 / 0.946 0.983 D7 0.891 / 0.969 0.964 / 0.959 0.942 / 0.971 0.979 D8 0.972 / 0.989 0.98 / 0.986 0.982 / 0.985 0.994 D9 0.976 / 0.976 0.947 / 0.981 0.976 / 0.980 0.989

[0118] As can be seen from Table 4, for D1, RoBERTa (BM2) has the largest F1 value, and No-BM2 is smaller than No-BM1 and No-BM3, indicating that RoBERTa has the largest contribution in the fusion of the three models, which is proportional to its own F1 value. For D2-D9, the performance of LR, SVC and RF on each data set is inconsistent. For example, LR performs best on D2, but performs worst on data sets such as D3, D4, and D6, indicating that each base model has its own strengths in processing different data sets. For D5, the F1 value of SVC (BM2) is only 0.697, but when it is fused with the other two base models using UMEAW, it can still achieve relatively good results, indicating that when a poorly performing model participates in the fusion of UMEAW, it can also make a positive contribution to the model fusion, proving that the method of the present invention has incomparable advantages over other methods based on the principle of collective decision-making.

[0119] V. Comparison of UMEAW with Baseline Methods

[0120] This section compares the fusion effects of Weigting average (Wavg), Majority voting (Mv), Weighted majority voting (Wmv), Stascking (Stk), GAS-EN, Auto-Ensemble (AE) and UMEAW on datasets D1-D9.

[0121] Table 5 F1 value comparison between UMEAW and baseline methods

[0122] Dataset ID Wavg Mv Wmv Stk GAS-EN AE UMEAW D1 0.817 0.816 0.817 0.813 0.823 0.817 0.829 D2 0.779 0.769 0.77 0.771 0.779 0.772 0.783 D3 0.862 0.868 0.875 0.834 0.892 0.897 0.897 D4 0.952 0.953 0.945 0.945 0.959 0.955 0.957 D5 0.941 0.946 0.945 0.923 0.945 0.931 0.953 D6 0.986 0.977 0.978 0.980 0.981 0.986 0.983 D7 0.966 0.974 0.971 0.946 0.979 0.971 0.979 D8 0.988 0.985 0.994 0.987 0.989 0.989 0.994 D9 0.979 0.989 0.985 0.989 0.986 0.985 0.989

[0123] For each data set, each method was executed three times, and the best result was taken for statistics. The results are shown in Table 5. The bold numbers in the table represent the best fusion results among all the results. It can be seen that UMEAW achieved the best results on the six data sets, and compared with other fusion methods, it has been stably improved by 1 to 2 percentage points, indicating that the method of the present invention has certain advantages over other fusion methods.

[0124] Figure 6 The F1 values ​​of D1 on the three base models BERT, RoBERTa, and XLNet and D2-D9 on the three base models LR, SVC, and RF are compared, as well as the results of the three base models fused with UMEAW. It can be seen that on 9 data sets with different numbers of classifications, the F1 value of the fused results is stably improved by 3% to 25% compared with the base models, which proves the effectiveness and versatility of the method of the present invention.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-classification universal ensemble method with adaptive weights, characterized in that: include: Step 1: Build n base models and divide the given data set into training set and test set; Step 2: Use the training set to train each base model, use each trained base model to predict the test set, and obtain the classification label and label probability corresponding to each base model; and calculate the F1 value of each base model according to the prediction results; Step 3: Calculate the weight of each base model according to the F1 value of each base model and the current weight allocation coefficient; Step 4: Perform weighted fusion according to the label probability and weight of each base model to obtain the fused label probability and the corresponding classification label, and calculate the corresponding F1 value; Step 5: Update the weight adjustment coefficient according to the given step size, and iterate step 3 to step 4; Step 6: Sort all fused label probabilities and corresponding classification labels according to the size of all F1 values ​​obtained in step 5, select n label probabilities and corresponding classification labels with larger F1 values ​​as the label probabilities and corresponding classification labels corresponding to n base models, and save the maximum F1 value; Step 7: Iterate steps 3 to 6 until the current maximum F1 value is less than the maximum F1 value in the previous iteration, and take the classification label corresponding to the maximum F1 value in the previous iteration as the final classification result.

2. The multi-classification universal integration method with adaptive weights according to claim 1, characterized in that: The type of the dataset is one of News Sentiment, Twitter Sentiment, Covertype, Mushrooms, Spambase, Nursery, Breast Cancer, Wine and Iris.

3. The multi-classification universal integration method with adaptive weights according to claim 1, characterized in that: Each base model is trained using the training set by means of K-fold cross validation.

4. The multi-classification universal integration method with adaptive weights according to claim 1, characterized in that: Step 3 specifically includes: According to the F1 value of each base model, the normalized F1 value is calculated according to formula (1): Among them, f i represents the F1 value of the i-th base model, f i * represents the normalized F1 value of the i-th base model, f min Indicates the smallest F1 value among all base models, f max Refers to the maximum F1 value among all base models; According to the normalized F1 value, the weight of each base model is calculated according to formula (2): Among them, w i ≥0, w i represents the weight of the i-th base model, and α represents the weight adjustment coefficient.

5. The multi-classification universal integration method with adaptive weights according to claim 4, characterized in that: The value range of the weight adjustment coefficient is specifically α∈[t,2t,3t,…,5(n-1)], where t is a given step length.

6. The multi-classification universal integration method with adaptive weights according to claim 1, characterized in that: In step 4, weighted fusion is performed according to formula (3): Among them, w i represents the weight of the i-th base model, p i Represents the label probability of the i-th base model.

Images