Classification method and apparatus considering ensemble error and ensemble diversity

Abstract

The application provides a classification method and device considering integrated error and integrated diversity, which comprises the following steps: first, acquiring a training set comprising M historical sample data x l , a feature vector X l and corresponding classification labels Y l ; then, obtaining a multi-classifier system c={c1,…,c U} by using an integrated learning algorithm; then, calculating the average similarity AS l between each sample x l in the training set and a sample to be classified; then, selecting K historical samples with the highest similarity based on the value of AS l , and forming a near neighbor region; then, optimizing a set of optimal weights based on the prediction performance of each base classifier in the multi-classifier system c in the near neighbor region; finally, determining the final classification result of the sample to be classified according to the optimal weights. The integrated performance of a set of classifiers and the diversity between different classifiers are considered, and a set of optimal weights can be generated according to the characteristics of each sample to be classified, which is used for weighted combination of the prediction results of different classifiers, so that the accuracy and stability of the final classification are ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ensemble learning, and particularly relates to a classification method and device considering ensemble error and ensemble diversity. BACKGROUND

[0002] Machine learning, as the first artificial intelligence technology, is currently widely used in data classification. Among them, the multiple classifier system (MCS), also known as ensemble learning, as the mainstream machine learning model, has also been widely concerned. The core idea of MCS is to use multiple base classifiers to classify sample data, and then based on the classification results of all base classifiers, a certain strategy is designed to synthesize all results to determine the final classification result, so as to ensure the final sample data classification accuracy. Common classifier weighted synthesis strategies mainly include Sum rule method, Product rule method and average synthesis method.

[0003] The individual accuracy of the base classifier and the diversity between different classifiers have always been considered as two key elements affecting the performance of MCS. However, due to the trade-off between the two, when using the above three strategies to synthesize the output results of different base classifiers, the final ensemble performance often cannot meet the expected situation, and the above synthesis strategies also have poor generalization effect due to the lack of considering the characteristics of each sample. SUMMARY

[0004] (I) Technical problems solved

[0005] In view of the shortcomings of the prior art, the present application provides a classification method and device considering ensemble error and ensemble diversity, which solves the problem that the existing weighted ensemble method lacks consideration of sample characteristics, performance differences of individual classifiers and diversity between different classifiers when aggregating the output results of base classifiers.

[0006] (II) Technical solutions

[0007] To achieve the above object, the present application is realized by the following technical solutions:

[0008] In a first aspect, a classification method considering ensemble error and ensemble diversity is provided, which comprises:

[0009] obtaining a training set containing M historical samples and corresponding feature vectors and classification labels;

[0010] training a multiple classifier system containing U base classifiers using the training set;

[0011] According to the feature vector of the sample to be classified, similarity between each historical sample and the sample to be classified is calculated;

[0012] The most similar historical sample of the K samples to be classified is screened out Each x k is obtained Corresponding feature vector and classification label;

[0013] The classification result of each base classifier on each x k is obtained; the classification result is the probability of the sample x k belonging to each classification;

[0014] The classification label of x k is converted into a label category vector;

[0015] A convex optimization model with the minimum integrated error as the target is obtained, and the convex optimization model is solved based on the classification result of each base classifier on each x k and the label category vector, to obtain the initial weight assigned to each base classifier when performing weighted integrated prediction on each x k by using the multi-classifier system;

[0016] The classification result of each base classifier on the sample to be classified is obtained;

[0017] A nonlinear optimization model with the maximum integrated diversity as the target is obtained, and the nonlinear optimization model is solved based on the global minimum value of the convex optimization model and the classification result of each base classifier on x k , to obtain the optimal weight of each base classifier;

[0018] The final classification result of the sample to be classified is obtained based on the optimal weight.

[0019] Further, the data types of the historical samples and the sample to be classified are both in text form and contain a plurality of feature value vectors.

[0020] Further, the classification label of x k is converted into a label category vector, including:

[0021]

[0022] Wherein, Y k represents the classification label of the kth historical sample x k ;

[0023] The category set of the classification label is θ h represents the hth classification.

[0024] Further, the convex optimization model is:

[0025]

[0026]

[0027] 0≤α u ≤1

[0028] wherein, α u is a decision variable, representing an initial weight assigned to base classifier c k when using a weighted ensemble prediction of x u using a multi-classifier system c; X k represents a feature vector of x k ; Y k represents a classification label of x k .

[0029] Further, the nonlinear optimization model is:

[0030]

[0031]

[0032]

[0033] 0≤α u ≤1

[0034] wherein, En represents a global minimum value of the convex optimization model.

[0035] In a second aspect, a classification device considering ensemble error and ensemble diversity is provided, which comprises:

[0036] a training set obtaining module, configured to obtain a training set containing M historical samples, and corresponding feature vectors and classification labels;

[0037] a multi-classifier system training module, configured to train a multi-classifier system containing U base classifiers using the training set;

[0038] a similar historical sample screening module, configured to calculate the similarity between each historical sample and the sample to be classified according to the feature vector of the sample to be classified, and screen out K most similar historical samples of the sample to be classified to obtain the corresponding feature vector and classification label of each x k .

[0039] a label category vector conversion module, configured to convert the classification label of x k into a label category vector;

[0040] an initial weight generation module, configured to obtain a convex optimization model taking the minimum ensemble error as an objective, and generate an initial weight of each base classifier for each x ksolving a convex optimization model by using the classification results of x k the initial weight assigned to each base classifier when performing weighted ensemble prediction;

[0041] a base classifier classification result acquisition module, configured to acquire a classification result of each base classifier for the sample to be classified;

[0042] an optimal weight generation module, configured to acquire a nonlinear optimization model with a maximum ensemble diversity as a target, and solve the nonlinear optimization model by using the classification result of x k and the global minimum value of the convex optimization model, to obtain an optimal weight of each base classifier;

[0043] a final classification result generation module, configured to obtain a final classification result of the sample to be classified based on the optimal weight.

[0044] Further, the data types of the historical samples and the sample to be classified are both in a text form and contain a plurality of feature value vectors.

[0045] Further, the conversion of the classification label of x k into a label category vector includes:

[0046]

[0047] wherein Y k represents a classification label of the kth historical sample x k .

[0048] a category set of the classification label is θ h represents the hth category.

[0049] Further, the convex optimization model is:

[0050]

[0051]

[0052] 0≤α u ≤1

[0053] wherein α u is a decision variable, representing the initial weight assigned to the base classifier c k when performing weighted ensemble prediction by using the multi-classifier system c on x u ; X k represents a feature vector of x k ; and Y k represents a classification label of x k .

[0054] Further, the nonlinear optimization model is:

[0055]

[0056]

[0057]

[0058] 0≤α u ≤1

[0059] Wherein, En represents the global minimum value of the convex optimization model.

[0060] (Three) beneficial effects

[0061] The application provides a classification method and device considering integrated error and integrated diversity. Compared with the prior art, the following beneficial effects are achieved:

[0062] 1) The application first acquires a training set including M historical sample data x l characteristic vectors X l and corresponding classification labels Y l , then obtains a multi-classifier system c = {c1,..., c U} based on the training set by using an ensemble learning algorithm, and then calculates the average similarity AS l between each sample x l in the training set and the sample to be classified ; based on the value of AS l , the K historical samples most similar to are selected to form a near neighbor region based on the prediction performance of each base classifier in the multi-classifier system c on the near neighbor region , a set of optimal weights is optimized and obtained; finally, the final classification result of the sample to be classified is determined according to the optimal weights. Not only the integrated performance of a set of classifiers and the diversity between different classifiers are considered, but also a set of optimal weights can be generated according to the characteristics of each sample to be classified, which is used to weight and combine the prediction results of different classifiers, so as to ensure the accuracy and stability of the final classification. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0064] Figure 1Flowchart of an embodiment of the present invention. DETAILED DESCRIPTION

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0066] The embodiments of the present application provide a classification method and device that take ensemble error and ensemble diversity into consideration, thereby solving the problem that existing weighted ensemble methods fail to consider sample characteristics, performance differences of individual classifiers, and diversity between different classifiers when aggregating the output results of base classifiers.

[0067] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0068] Example 1:

[0069] like Figure 1 As shown, the present invention provides a classification method considering ensemble error and ensemble diversity, which is executed by a computer and includes:

[0070] Obtain a training set containing M historical samples and corresponding feature vectors and classification labels;

[0071] Using the training set to train a multi-classifier system comprising U base classifiers;

[0072] According to the feature vector of the sample to be classified, the similarity between each historical sample and the sample to be classified is calculated;

[0073] Filter out the most similar historical samples of K samples to be classified Get each x k Corresponding feature vectors and classification labels;

[0074] Get each base classifier for each x k The classification result of the sample x k The probability of belonging to each category;

[0075] x k The classification labels are converted into label category vectors;

[0076] Obtain a convex optimization model with the goal of minimizing the ensemble error, and perform the optimization on each x based on each base classifier. kThe classification results and the label category vector are solved by the convex optimization model to obtain the classification results of each x using the multi-classifier system. k The initial weights assigned to each base classifier when making weighted ensemble predictions;

[0077] Obtain the classification results of each base classifier for the samples to be classified;

[0078] Obtain a nonlinear optimization model with the goal of maximizing ensemble diversity, and based on the global minimum of the convex optimization model and each base classifier on x k The classification results are used to solve the nonlinear optimization model and obtain the optimal weight of each base classifier;

[0079] The final classification result of the sample to be classified is obtained based on the optimal weight.

[0080] The beneficial effects of this embodiment are:

[0081] 1) The present invention not only considers the integrated performance of a group of classifiers and the diversity among different classifiers, but also generates a set of optimal weights according to the characteristics of each sample to be classified, which are used to weightedly synthesize the prediction results of different classifiers to ensure the accuracy and stability of the final classification.

[0082] The implementation process of the embodiment of the present invention is described in detail below:

[0083] S1. Get the training set The training set includes M historical samples x l The eigenvector X l and the corresponding classification label Y l , where l = 1,…,M; the category set of classification labels is Indicates that the current classification problem is a ω (ω ≥ 2) class classification problem.

[0084] Among them, the sample is a vector in text form. A sample consists of multiple features, each feature has a value, and the values ​​of multiple features form a vector.

[0085] S2, based on the training set, an ensemble learning algorithm is used to obtain a multi-classifier system c={c1,…,c U};c u represents the u-th base classifier, u=1,…,U;

[0086] S3. Obtain samples to be classified The eigenvector of Using the training set And Euclidean distance measure, calculate each historical sample x l and samples to be classified The similarity of l, l=1,…,M;

[0087]

[0088] here and X l,i Represent new samples and historical sample x l The i-th eigenvalue of ;

[0089] S4. According to AS l The size of K is selected The most similar historical sample is recorded as and get x k The eigenvector X k and the classification label Y k ;

[0090] S5. Each x k The eigenvector X k Input to each base classifier c of the multi-classifier system u , get each base classifier c u For each x k The eigenvector X k The classification result c u (X k )={c u,1 (X k ),…,c u,h (X k ),…,c u,ω (X k )},c u,h (X k ) represents the base classifier c u Consider historical sample x k The probability of belonging to the hth class, h = [1, ω];

[0091] S6. Classification labels Y of K historical samples k Convert to label category vector The specific formula is as follows:

[0092]

[0093] S7, to ensure the minimum integration error, based on c u (X k ) and Y k Solve the convex optimization model; and the convex optimization model is:

[0094]

[0095]

[0096] 0≤α u ≤1

[0097] Among them, α u Represents the decision variable, which means using the multi-classification system c to classify x k Assigned to the base classifier c when making weighted ensemble predictions u The initial weight of

[0098] The importance of each classifier in the ensemble, that is, the weight of each classifier, is determined based on the ensemble error of the classifier. Because the constructed convex optimization model is not a non-convex model in the strict sense, it is impossible for only one set of weights to achieve the global minimum. There will be many sets of weights that meet the global minimum of the convex optimization model, and the optimal one needs to be further selected from them.

[0099] S8, will The eigenvector of Input to each base classifier c in the multi-classifier system c u , get each base classifier c u Treat classification samples The eigenvector of Classification results

[0100] S9. Let the global minimum of the convex optimization model be En, then continue to solve the nonlinear optimization model with the goal of maximizing the integrated diversity to obtain the optimal weight; and the nonlinear optimization model is:

[0101]

[0102]

[0103]

[0104] 0≤α u ≤1

[0105] The second optimization is to design a diversity measure based on the first optimization to ensure the maximum ensemble diversity. That is, to find a set of solutions that can maximize the ensemble diversity among all the solutions that can be obtained in the first optimization.

[0106] S10, the optimal weight obtained by the nonlinear optimization model is recorded as use synthesis The final classification result of the sample to be classified is θ η ,

[0107]

[0108] To verify the superiority of the proposed method, 30 standard classification data sets were collected from two machine learning public data repositories KEEL and UCI, and the basic information of the data sets is shown in Table 1. 90% of the data in each data set was randomly extracted as a training set, and the remaining 10% was used as a test set.

[0109] Table 1

[0110]

[0111]

[0112] Ten base classifiers were generated from the training set using the Bagging algorithm to form a multi-classifier system c, and a decision tree was selected as the base classifier. For each data set, the above process was repeated ten times. According to the average performance on ten repeated experiments, the performance difference between the proposed classifier dynamic weighting ensemble classification method and the existing three combination strategies (sum rule, product rule, and average combination rule) and six common ensemble algorithms (Bagging algorithm, random forest algorithm, AdaBoosting algorithm, gradient ensemble tree algorithm, and XGBoost algorithm) was compared.

[0113] The relevant experimental results are shown in Table 2. In Table 2:

[0114] BAG represents the Bagging algorithm

[0115] RF represents the random forest algorithm

[0116] ABT represents the AdaBoosting algorithm

[0117] GB represents the gradient ensemble tree algorithm

[0118] XGB represents the XGBoost algorithm

[0119] Sum represents the sum rule

[0120] Product represents the product rule

[0121] SAC represents the average combination rule

[0122] PWE represents the proposed method of the present application.

[0123] Table 2 experimental results

[0124]

[0125]

[0126] As can be seen from Table 2, the method of the embodiment is superior to the existing synthesis strategy and integrated method in most cases, and the method can obtain a set of optimal weights for synthesizing the output results of different classifiers according to the characteristics of each sample.

[0127] Embodiment 2:

[0128] A classification device considering integrated error and integrated diversity, comprising:

[0129] a training set acquisition module configured to acquire a training set containing M historical samples, corresponding feature vectors and classification labels;

[0130] a multi-classifier system training module configured to train a multi-classifier system containing U base classifiers by using the training set;

[0131] a similar historical sample screening module configured to calculate the similarity between each historical sample and the sample to be classified according to the feature vector of the sample to be classified, and screen out K most similar historical samples of the sample to be classified obtain each x k corresponding feature vector and classification label;

[0132] a label category vector conversion module configured to convert the classification label of x k into a label category vector;

[0133] an initial weight generation module configured to acquire a convex optimization model taking the minimum integrated error as a target, and solve the convex optimization model based on the classification result of each base classifier for each x k and the label category vector to obtain the initial weight assigned to each base classifier when performing weighted integrated prediction of each x k by using the multi-classifier system;

[0134] a base classifier classification result acquisition module configured to acquire the classification result of each base classifier for the sample to be classified;

[0135] an optimal weight generation module configured to acquire a nonlinear optimization model taking the maximum integrated diversity as a target, and solve the nonlinear optimization model based on the global minimum value of the convex optimization model and the classification result of each base classifier for x k to obtain the optimal weight of each base classifier;

[0136] a final classification result generation module configured to obtain the final classification result of the sample to be classified based on the optimal weight.

[0137] It can be understood that the classification device considering integration error and integration diversity provided in the embodiment of the present invention corresponds to the above-mentioned classification method considering integration error and integration diversity. The explanation, examples, beneficial effects, etc. of the relevant contents can refer to the corresponding contents in the classification method considering integration error and integration diversity, and will not be repeated here.

[0138] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0139] 1) The present invention not only considers the integration performance of a group of classifiers and the diversity between different classifiers, but also generates a set of optimal weights according to the characteristics of each sample to be classified, which is used to weight the prediction results of different classifiers to ensure the accuracy and stability of the final classification. First, obtain M historical sample data x l The eigenvector X l and the corresponding classification label Y l Based on the training set, the ensemble learning algorithm is used to obtain the multi-classifier system c={c1,…,c U}, and then calculate each sample x in the training set l and samples to be classified The average similarity between AS l ; Based on AS l The value of The most similar K historical samples constitute the neighboring area Then based on the base classifiers in the multi-classifier system c in the neighboring area Optimize the prediction performance on the Finally, according to Determine the samples to be classified The final classification result.

[0140] It should be noted that, through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software plus necessary universal hardware platforms. Based on such an understanding, the above technical solutions can be embodied in the form of a software product in essence or in the form of a part of the prior art that makes a contribution. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments. In this article, relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0141] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A classification method considering ensemble error and ensemble diversity, characterized in that: The method includes: Obtain a training set containing M historical samples and corresponding feature vectors and classification labels; Using the training set to train a multi-classifier system comprising U base classifiers; According to the feature vector of the sample to be classified, the similarity between each historical sample and the sample to be classified is calculated using the training set and Euclidean distance measurement; Filter out the most similar historical samples of K samples to be classified Get each x k Corresponding feature vectors and classification labels; Get each base classifier for each x k The classification result of the sample x k The probability of belonging to each category; x k The classification labels are converted into label category vectors; Obtain a convex optimization model with the goal of minimizing the ensemble error, and perform the optimization on each x based on each base classifier. k The classification results and the label category vector are solved by the convex optimization model to obtain the classification results of each x using the multi-classifier system. k The initial weights assigned to each base classifier when making weighted ensemble predictions; Obtain the classification results of each base classifier for the samples to be classified; Obtain a nonlinear optimization model with the goal of maximizing ensemble diversity, and based on the global minimum of the convex optimization model and each base classifier on x k The classification results are used to solve the nonlinear optimization model and obtain the optimal weight of each base classifier; The final classification result of the sample to be classified is obtained based on the optimal weight; The data types of the historical samples and the samples to be classified are both in text form and vectors containing multiple eigenvalues; The method also includes: collecting 30 standard classification datasets from two open machine learning data repositories, KEEL and UCI. 90% of the data from each dataset was randomly sampled as a training set, and the remaining 10% was used as a test set. Ten base classifiers were generated from the training set using the bagging algorithm to form a multi-classifier system c, with decision trees selected as the base classifiers. The 30 standard classification datasets include faults and parkinson. The technical solution is embodied in the form of a software product, which can be stored in a computer-readable storage medium to enable a computer device to execute the method.

2. A classification method considering ensemble error and ensemble diversity as claimed in claim 1, characterized in that: The x k The classification labels are converted into label category vectors, including: Among them, Y k Represents the kth historical sample x k Classification label of The category set of classification labels is θ h represents the hth category.

3. A classification method considering ensemble error and ensemble diversity as claimed in claim 1, characterized in that: The convex optimization model is: 0≤α u ≤1 Among them, α u Is the decision variable, indicating the use of multi-classifier system c for x k Assigned to the base classifier c when making weighted ensemble predictions u The initial weight of X k Represents x k The eigenvector of k Represents x k The classification label of .

4. A classification method considering ensemble error and ensemble diversity as claimed in claim 1, characterized in that: The nonlinear optimization model is: 0≤α u ≤1 Where En represents the global minimum of the convex optimization model.

5. A classification device considering ensemble error and ensemble diversity, characterized in that: The device includes: The training set acquisition module is used to obtain a training set containing M historical samples and corresponding feature vectors and classification labels; A multi-classifier system training module, configured to use the training set to train a multi-classifier system comprising U base classifiers; The similar historical sample screening module is used to calculate the similarity between each historical sample and the sample to be classified based on the feature vector of the sample to be classified, using the training set and Euclidean distance measurement; and screen out the K most similar historical samples to the sample to be classified. Get each x k Corresponding feature vectors and classification labels; Label category vector conversion module, used to convert x k The classification labels are converted into label category vectors; The initial weight generation module obtains a convex optimization model with the goal of minimizing the integrated error, and performs a weighted calculation on each x based on each base classifier. k The classification results and the label category vector are solved by the convex optimization model to obtain the classification results of each x using the multi-classifier system. k The initial weights assigned to each base classifier when making weighted ensemble predictions; The base classifier classification result acquisition module is used to obtain the classification results of each base classifier for the sample to be classified; The optimal weight generation module is used to obtain a nonlinear optimization model with the goal of maximizing the ensemble diversity, and is based on the global minimum of the convex optimization model and the optimal weight generation module for each base classifier. k The classification results are used to solve the nonlinear optimization model and obtain the optimal weight of each base classifier; The final classification result generation module is used to obtain the final classification result of the sample to be classified based on the optimal weight; The data types of the historical samples and samples to be classified are all in text format and vectors containing multiple eigenvalues. 30 standard classification datasets were collected from two open machine learning data repositories, KEEL and UCI. 90% of the data from each dataset was randomly sampled as a training set, and the remaining 10% was used as a test set. Ten base classifiers were generated from the training set using the bagging algorithm to form a multi-classifier system c, with a decision tree selected as the base classifier. The 30 standard classification datasets include faults and parkinson. The technical solution is embodied in the form of a software product, which can be stored in a computer-readable storage medium to enable a computer device to execute the method.

6. The classification device considering ensemble error and ensemble diversity according to claim 5, characterized in that: The x k The classification labels are converted into label category vectors, including: Among them, Y k Represents the kth historical sample x k Classification label of The category set of classification labels is θ h represents the hth category.

7. The classification device considering ensemble error and ensemble diversity according to claim 5, characterized in that: The convex optimization model is: 0≤α u ≤1 Among them, α u Is the decision variable, indicating the use of multi-classifier system c for x k Assigned to the base classifier c when making weighted ensemble predictions u The initial weight of X k Represents x k The eigenvector of k Represents x k The classification label of .

8. The classification device considering ensemble error and ensemble diversity according to claim 5, characterized in that: The nonlinear optimization model is: 0≤α u ≤1 Where En represents the global minimum of the convex optimization model.

Images