Adaptive ensemble learning classification method applied to unbalanced data
Through the adaptive ensemble learning method of sliding window feature extraction, clustering undersampling and cross-entropy weight optimization, the problem of difficulty in identifying a few categories of samples in the imbalanced dataset is solved, and the robustness and computational efficiency of the model are improved.
Patent Information
- Application Number
- CN202411551532.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-01
- Publication Date
- 2024-12-27
AI Technical Summary
In the case of unbalanced data sets, traditional machine learning models are difficult to effectively identify a few categories of samples, and the parameter adjustment dependence is large, resulting in high computing resource consumption, and existing methods are prone to information loss or model overfitting.
Adaptive ensemble learning classification method is adopted, combining sliding window feature extraction, clustering undersampling and cross-entropy weight optimization, and the number of samples in a few categories is improved by generating a subset of diversity features, balancing data sets, training base classifiers and weighted voting. Identification ability.
It significantly improves the identification performance of a few categories of samples, optimizes classification performance, reduces computing resource consumption, and improves the robustness of the model without sacrificing overall classification accuracy.
Smart Images
Figure CN119202949A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of machine learning, and in particular to an adaptive integrated learning classification method applied to imbalanced data. Background Art
[0002] An imbalanced dataset refers to a dataset in which the number of samples of different categories is significantly different, that is, the number of samples of some categories is much larger than that of other categories. This phenomenon is common in machine learning and data mining, especially in practical applications. For example, in medical diagnosis, the number of samples of normal patients may be significantly larger than that of patients with rare diseases; in fraud detection, the number of samples of normal transactions may be much larger than that of fraud transactions. In an imbalanced situation, traditional evaluation indicators (such as accuracy) may be misleading. For example, in a dataset containing 95% normal samples and 5% fraud samples, even if the model predicts all samples as normal, its accuracy can still reach 95%. In this case, the model may ignore the importance of minority classes (such as fraud) and fail to effectively identify these minority class samples. The scarcity of minority class samples makes it difficult for the model to learn and capture the characteristics of these classes, resulting in a low recognition rate for minority classes. In this regard, most existing solutions solve the problem by reducing the majority class samples or increasing the minority class samples, which is prone to information loss or model overfitting. They perform generally when dealing with complex feature spaces and decision boundaries between classes.
[0003] In addition, the current model relies heavily on parameter adjustments during training, especially in terms of the size and step size of the sliding window. This means that each time these parameters are adjusted, multiple base classifiers need to be retrained and their performance indicators on the validation set, such as F1-score, AUC, etc., need to be evaluated. This repetitive training process significantly increases the consumption of computing resources, especially when using an integrated learning model. Specifically, when adjusting the size and step size of the sliding window, you are actually changing the way the model extracts features. The width of the sliding window determines the range of features extracted from the data each time, while the step size determines the speed at which the window moves on the data and the degree of overlap covered. Different combinations of window width and step size may result in different feature subsets, which will affect the training of each base classifier and the final integration effect. Therefore, each adjustment of these parameters requires retraining all base classifiers to ensure that they can adapt to the new feature subsets.
[0004] In order to cope with these challenges, targeted methods need to be adopted, such as data resampling, adjusting classification thresholds, using weighted loss functions, etc., to improve the performance of the model on minority classes and avoid the deviation of evaluation indicators. In view of this, the present invention proposes an adaptive ensemble learning classification method for imbalanced data. Summary of the invention
[0005] The purpose of the present invention is to propose an adaptive ensemble learning classification method for imbalanced data; it combines the diversity extraction of feature subsets and clustering undersampling technology, so as to improve the recognition ability of minority class samples and optimize the classification performance when facing the classification problem of imbalanced data sets.
[0006] To achieve the above object, the technical solution of the present invention is: an adaptive ensemble learning classification method applied to unbalanced data, specifically comprising the following steps:
[0007] Step 1: Collect raw data from various databases, pre-process the raw data, and extract negative data and positive data; divide the negative data and positive data into training set, validation set, and test set according to the preset ratio;
[0008] Step 2: Use sliding windows to extract features from the positive and negative training sets;
[0009] Step 3: Cluster the negative classes in step 2;
[0010] Step 4: Randomly undersample the samples based on step 3;
[0011] Step 5: Combine the undersampled samples with the positive training set samples under the same feature subset;
[0012] Step 6: Select random forest as the base classifier to train each sample combination in step 5;
[0013] Step 7: Use the combination of positive and negative validation sets to calculate the cross entropy loss of the base classifier after training in step 5, and assign weights to each base classifier by the inverse of the cross entropy loss; use the combination of positive and negative test sets to perform weighted voting to obtain the predicted label, and perform indicator calculation with the true label of the test set to determine the performance of the final model.
[0014] Preferably, the step S1 specifically comprises:
[0015] Collect raw data from various databases; the raw data is stored in .csv format, and the last column of the raw data is the label column, and the remaining columns are feature columns;
[0016] Normalize the original data as a whole except for the label column, and then extract two categories of normalized data according to the label, one is the majority class data, that is, negative class data, and the other is the minority class data, that is, positive class data;
[0017] The negative data and positive data are divided into training set, validation set and test set respectively in the ratio of 7:2:1.
[0018] Preferably, the step 2 is specifically:
[0019] Step 2.1: Pre-mark the column index for each column feature of each type of training set, and set the width W and step size S of the sliding window;
[0020] Step 2.2: The sliding window moves from lower index to higher index, gradually extracting feature subsets. When the sliding window moves to the last column, the remaining columns are filled in a cyclic manner to obtain M different feature subsets:
[0021]
[0022] Where F represents the total number of features in the data.
[0023] Preferably, the negative class clustering specifically divides the negative class data into K clusters by K-means clustering.
[0024] Preferably, the step S4 specifically includes randomly under-sampling each of the K clusters after the negative class clustering.
[0025] Preferably, the calculation of the cross entropy loss is specifically as follows:
[0026]
[0027] Where CE_Loss represents the cross entropy loss, N val is the number of samples in the validation set, y i is the true label of the i-th validation sample, p i is the predicted probability of the i-th sample.
[0028] Preferably, the step of assigning a weight to each base classifier by the inverse of the cross entropy loss is as follows: normalizing the inverse of the cross entropy loss of each base classifier on the validation set to obtain the weight of each base classifier:
[0029]
[0030] In the formula, Weight m,l Represents the weight of the lth base classifier under the mth feature subset, CE_Loss m,l represents the cross entropy loss of the lth base classifier under the mth feature subset on the validation set, M×L is the total number of base classifiers, M represents the number of feature subsets, and L represents the number of base classifiers under each feature subset.
[0031] Preferably, the weighted voting is performed using the combination of positive and negative test sets to obtain the final prediction result, which is specifically calculated as follows:
[0032]
[0033] In the formula, Prediction represents the prediction result, P m,l (X test ) represents the predicted probability of the combination of positive and negative test sets, c = {0,1} represents the predicted label, 0 represents the predicted label for the negative class, 1 represents the predicted label for the positive class, X test is the combination of positive and negative test sets.
[0034] Preferably, the width W and step length S of the sliding window are determined by a two-stage optimization strategy. In the first stage, the step length S is fixed to optimize the width W; in the second stage, the width W is fixed to optimize the step length S, as follows:
[0035] In the first stage, the fixed step size S is equal to the total number of features F. The width W starts from 2, and the performance indicators of multiple feature subsets are evaluated on the validation set, and the average performance corresponding to each width W is recorded; the width W is gradually increased by 1, and the above evaluation and recording process is repeated until the width W increases to F-1; by recording the performance indicators under each width W, a set of performance data can be obtained, and the Pareto optimization algorithm is used to comprehensively consider these performance indicators to determine whether the performance of each window width is dominated by other window widths, so as to identify the window width W that performs best on multiple indicators. * ;
[0036] In the second stage, the optimal window width W found in the previous stage is used * Based on, gradually increase the step length S by 1, repeat the evaluation and recording process in the first stage until the step length S increases to F, use the Pareto optimization algorithm, comprehensively consider multiple performance indicators, and determine the optimal step length S * .
[0037] Preferably, the performance indicators include accuracy, F1 score, AUC and cross entropy loss.
[0038] Compared with the prior art, the present invention has the following beneficial effects:
[0039] The present invention first generates multiple data representations with different feature subsets through feature extraction technology (sliding window) to capture more features of minority classes. Then, a clustering algorithm is applied to each feature subset to undersample the data and select representative samples to balance the class distribution in the data set. Then, multiple base classifiers are trained on these balanced data subsets, and the prediction results of these base classifiers are integrated using ensemble learning technology (weighted voting) to improve the robustness of the overall model and the minority class recognition performance. Finally, by performing a detailed performance evaluation of the model (such as calculating F1-score and AUC), the parameters in each step are adjusted to ensure that the recognition ability of minority class samples is significantly improved without sacrificing the overall classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 Visualize imbalanced data graph for binary classification;
[0041] Figure 2 It is a diagram of the feature extraction and filling process in the present invention;
[0042] Figure 3 This is a graph showing the clustering results of multiple feature subsets using K-means in the present invention;
[0043] Figure 4 This is a graph of random undersampling results for each cluster in the present invention;
[0044] Figure 5 This is a diagram of the integrated learning framework of the feature and sampling mixture of the present invention;
[0045] Figure 6 It is the overall flow chart of the present invention;
[0046] Figure 7 This is a process diagram of the present invention using a combination of positive and negative test sets to perform weighted voting to obtain a predicted label. DETAILED DESCRIPTION
[0047] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.
[0048] Reference Figures 1 to 6 As shown, the present invention aims to solve the significant challenge of existing imbalanced dataset classification methods in identifying minority class samples. Figure 1 It shows the class imbalance problem in the dataset, which causes the majority class samples to dominate the model training, thus significantly reducing the recognition performance of the minority class samples. Figure 2 Shows the diversity of features extracted by sliding windows. Figure 3 and Figure 4 The complexity of existing clustering undersampling techniques in terms of data balance and information preservation is plotted. Figure 5 The overall framework of the present invention is described. Figure 6 It is a technical flow chart.
[0049] The present invention proposes an adaptive ensemble learning classification method for imbalanced data, which combines sliding window feature extraction, cluster undersampling, and cross entropy weight optimization ensemble learning methods. The implementation of the present invention specifically includes two major steps: (1) first building a small parameter model according to the proposed technology; (2) using real imbalanced data for testing. The specific steps of the present invention are as follows:
[0050] Step 1: First, collect the original data from major databases. These data are stored in ".csv" format; the last column is the label column, and the remaining columns are feature columns; then normalize the original data except the label column; according to the label, extract two categories from the normalized data, one is the majority class (negative class) and the other is the minority class (positive class); divide the negative class into training set, validation set, and test set according to the ratio of 7:2:1, and similarly divide the positive class according to the same ratio.
[0051] Step 2: Use sliding windows to extract features from the training sets of positive and negative classes.
[0052] Step 3: Cluster only the negative classes in step 2.
[0053] Step 4: Randomly undersample the samples based on step 3.
[0054] Step 5: Combine the undersampled samples with the positive training set samples under the same feature subset.
[0055] Step 6: Select random forest as the base classifier and train each sample combination in step 5.
[0056] Step 7: Use the combination of positive and negative validation sets to calculate the cross entropy loss of the base classifier after training in step 5, and assign weights to each base classifier by the inverse of the cross entropy loss. Use the combination of positive and negative test sets to do weighted voting to obtain the predicted labels, and calculate indicators with the true labels of the test set to determine the performance of the final model; that is, use the trained classifier to predict the combination of positive and negative test sets to obtain the predicted probabilities of the positive and negative classes. Then multiply the predicted probabilities of the positive and negative classes with the assigned base classifier weights, and then accumulate all weighted positive and negative prediction probabilities to obtain the final voting results (predicted labels) of all base classifiers. Calculate the accuracy, F1-score, AUC and other indicators with the true labels of the test set, and use these indicators to determine the performance of the final model.
[0057] In this embodiment, the process of sliding window extracting features is as follows: Figure 2 As shown in the figure. Each column of features is pre-labeled with the column index, and the width W and step size S of the sliding window are set. As the sliding window moves from lower index to higher index, feature subsets are gradually extracted. When the sliding window moves to the end, the remaining columns are filled in a cyclic manner. Assuming that the total number of features in the data is F, M different feature subsets can be obtained according to formula (1). This method moves in the feature space by sliding the window, which is similar to the convolution kernel in a convolutional neural network. In the convolution operation, the convolution kernel slides on the input data, covering a local area each time, and the dot product sum of the convolution kernel and the local area is calculated to obtain a single value. The convolution kernel moves on the input features according to the set step size until all features are traversed.
[0058]
[0059] In this embodiment, the negative class clustering specifically divides the negative class data into K clusters through K-means clustering.
[0060] K-means clustering is a simple and commonly used clustering algorithm that aims to discover the potential structure in the data by dividing the data into K clusters. Its goal is to make the data points in the same cluster as close to each other as possible, and the data points in different clusters as far away from each other as possible. The advantages of K-means are easy implementation and high computational efficiency, and it is suitable for large-scale data sets. It is often used in image processing, customer segmentation, pattern recognition and other fields. Figure 3 Set the cluster parameter to 3.
[0061] In this embodiment, step S4 specifically includes randomly under-sampling each of the K clusters after negative class clustering.
[0062] After clustering, each cluster is randomly undersampled. By randomly deleting some samples from the majority class, the number of samples in different classes is balanced. However, simple random undersampling may lead to the loss of valuable information, especially when the majority class samples contain important features, which may affect the overall performance of the model. Clustering-based undersampling reduces the information loss that may be caused by random undersampling by selecting representative samples from the majority class. It retains the diversity within the class through clustering and deletes redundant samples, thereby effectively balancing the class imbalance while retaining key features and improving the performance of the model when dealing with minority classes. Figure 4 The red markers are the results of random undersampling.
[0063] In this embodiment, after completing the first two steps, different types of base classifiers can be trained using the preprocessed samples.
[0064] The present invention uses random forest as the base classifier mainly because random forest has stronger robustness and generalization ability, can effectively process large-scale data sets and a large number of features, and has high anti-interference ability to noise and outliers. In addition, the implementation and parameter adjustment of random forest are relatively simple, and there is no need to scale the features, so that it usually performs better than SVM or a single decision tree in practical applications.
[0065] In this embodiment, the specific content of setting weights using cross entropy loss is as follows.
[0066] Cross entropy loss is commonly used in the optimization of deep learning networks to measure the difference between two probability distributions. In classification problems, it evaluates the difference between the predicted probability distribution and the true label distribution. In the present invention, cross entropy is used to evaluate the performance of each base classifier and serves as the weight of the base classifier vote. The main reasons are as follows: (1) The cross entropy weight assigns higher weights to correctly classified categories and lower probabilities to incorrectly classified categories. This allows the model to better focus on correctly classified samples during training. (2) Since cross entropy takes probability distribution into account, it is robust to noisy data to a certain extent, which helps to reduce the impact of noise on model performance. (3) Unlike indicators such as accuracy, cross entropy not only considers whether the classification is correct, but also the confidence of the classification. Cross entropy rewards base classifiers with high confidence and penalizes classifiers with low confidence, which is more in line with actual scenarios and improves the practicality of the model;
[0067] The calculation of the cross entropy loss is specifically as follows:
[0068]
[0069] Where CE_Loss represents the cross entropy loss, N val is the number of samples in the validation set, y i is the true label of the i-th validation sample, p i is the predicted probability of the i-th sample.
[0070] The weight of each base classifier is determined by the inverse of the cross entropy loss of its prediction on the validation set, and the final weight is normalized according to formula (3). The L base learners vote to determine the final prediction result determined by formula (4).
[0071]
[0072] In the formula, Weight j Represents the weight of the j-th base classifier, CE_Loss j represents the cross entropy loss of the jth base classifier on the validation set, M×L is the total number of base classifiers; Prediction represents the prediction result, h j (x) represents the predicted probability of the combination of positive and negative test sets, c = {0,1} represents the predicted label, 0 represents the predicted label for the negative class, and 1 represents the predicted label for the positive class.
[0073] In this embodiment, the window parameters are optimized based on the Pareto frontier.
[0074] The entire model is greatly affected by the combination of different window widths and step sizes. If we use global optimization to traverse all possible parameter combinations to find the optimal solution, this method is very time-consuming in high-dimensional space because the dimension and size of the parameter space will cause the computational complexity to grow exponentially, resulting in excessive search time and resource consumption. Therefore, the present invention adopts a two-stage optimization strategy to reduce the computational complexity. In the first stage, S is fixed to optimize M; in the second stage, W is fixed to optimize S. The staged optimization reduces the search space in each stage. In the first stage, S is fixed to optimize M; in the second stage, W is fixed to optimize S. Starting from W 2, evaluate the performance indicators of multiple feature subsets on the validation set, including accuracy, F1 score, AUC, and cross entropy loss. Record the average performance corresponding to each W. Increase W gradually in increments of 1, and repeat the above evaluation and recording process until it increases to F-1. By recording the performance indicators under each, a set of performance data can be obtained. The Pareto optimization algorithm is used to comprehensively consider these performance indicators to determine whether the performance of each window width is dominated by other window widths, so as to identify the window width W that performs best in multiple indicators. * In the second stage, the W found in the previous stage * Based on this, further optimize S. Increase S gradually by 1, repeat the above evaluation and recording process until S increases to F. Also use the Pareto optimization algorithm, comprehensively consider multiple performance indicators, and determine the optimal step size S * .
[0075] The present invention constructs a complete integrated learning framework by integrating sliding window feature extraction, cluster undersampling, cross entropy weight optimization, Pareto optimization and weighted voting mechanism. By combining the diversity selection of feature subsets with cluster undersampling, it can not only effectively process unbalanced data, but also retain the diversity and decision boundary information of majority class samples in the feature space, thereby greatly improving the minority class recognition performance. At the same time, the use of sliding window and Pareto optimization further improves the model's ability to process high-dimensional data.
[0076] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.
Claims
1. An adaptive ensemble learning classification method for imbalanced data, characterized in that: The specific steps include: Step 1: Collect raw data from major databases, pre-process the raw data, and extract negative and positive data; Divide the negative data and positive data into training set, validation set and test set respectively according to the preset ratio; Step 2: Use sliding windows to extract features from the positive and negative training sets; Step 3: Cluster the negative classes in step 2; Step 4: Randomly undersample the samples based on step 3; Step 5: Combine the undersampled samples with the positive training set samples under the same feature subset; Step 6: Select random forest as the base classifier to train each sample combination in step 5; Step 7: Use the combination of positive and negative validation sets to calculate the cross entropy loss of the base classifier after training in step 5, and assign weights to each base classifier by the inverse of the cross entropy loss; use the combination of positive and negative test sets to perform weighted voting to obtain the predicted label, and perform indicator calculation with the true label of the test set to determine the performance of the final model.
2. The adaptive ensemble learning classification method for imbalanced data according to claim 1, characterized in that: The step S1 is specifically as follows: Collect raw data from various databases; the raw data is stored in .csv format, and the last column of the raw data is the label column, and the remaining columns are feature columns; Normalize the original data as a whole except for the label column, and then extract two categories of normalized data according to the label, one is the majority class data, that is, negative class data, and the other is the minority class data, that is, positive class data; The negative data and positive data are divided into training set, validation set and test set respectively in the ratio of 7:2:
1.
3. The adaptive ensemble learning classification method for imbalanced data according to claim 1, characterized in that: The step 2 is specifically as follows: Step 2.1: Pre-mark the column index for each column feature of each type of training set, and set the width W and step size S of the sliding window; Step 2.2: The sliding window moves from lower index to higher index, gradually extracting feature subsets. When the sliding window moves to the last column, the remaining columns are filled in a cyclic manner to obtain M different feature subsets: Where F represents the total number of features in the data.
4. The adaptive ensemble learning classification method for imbalanced data according to claim 1, characterized in that: The negative class clustering specifically divides the negative class data into K clusters through K-means clustering.
5. The adaptive ensemble learning classification method for imbalanced data according to claim 4, characterized in that: The step S4 specifically includes randomly under-sampling each of the K clusters after the negative class clustering.
6. The adaptive ensemble learning classification method for imbalanced data according to claim 1, characterized in that: The calculation of the cross entropy loss is specifically as follows: Where CE_Loss represents the cross entropy loss, N val is the number of samples in the validation set, y i is the true label of the i-th validation sample, p i is the predicted probability of the i-th sample.
7. The adaptive ensemble learning classification method for imbalanced data according to claim 1, characterized in that: The method of assigning a weight to each base classifier by the inverse of the cross entropy loss is specifically to normalize the inverse of the cross entropy loss of each base classifier on the validation set to obtain the weight of each base classifier: In the formula, Weight m,l Represents the weight of the lth base classifier under the mth feature subset, CE_Loss m,l represents the cross entropy loss of the lth base classifier under the mth feature subset on the validation set, M×L is the total number of base classifiers, M represents the number of feature subsets, and L represents the number of base classifiers under each feature subset.
8. The adaptive ensemble learning classification method for imbalanced data according to claim 1, characterized in that: The weighted voting is performed by combining the positive and negative test sets to obtain the final prediction result. The specific calculation is: In the formula, Prediction represents the prediction result, P m,l (X test ) represents the predicted probability of the combination of positive and negative test sets, c = {0,1} represents the predicted label, 0 represents the predicted label for the negative class, 1 represents the predicted label for the positive class, X test is the combination of positive and negative test sets.
9. The adaptive ensemble learning classification method for imbalanced data according to claim 3, characterized in that: A two-stage optimization strategy is used to determine the width W and step size S of the sliding window in step 2. In the first stage, the step size S is fixed to optimize the width W; in the second stage, the width W is fixed to optimize the step size S, as follows: In the first stage, the fixed step size S is equal to the total number of features F. The width W starts from 2, and the performance indicators of multiple feature subsets are evaluated on the validation set, and the average performance corresponding to each width W is recorded; the width W is gradually increased by 1, and the above evaluation and recording process is repeated until the width W increases to F-1; by recording the performance indicators under each width W, a set of performance data can be obtained, and the Pareto optimization algorithm is used to comprehensively consider these performance indicators to determine whether the performance of each window width is dominated by other window widths, so as to identify the window width W that performs best on multiple indicators. * ; In the second stage, the optimal window width W found in the previous stage is used * Based on, gradually increase the step length S by 1, repeat the evaluation and recording process in the first stage until the step length S increases to F, use the Pareto optimization algorithm, comprehensively consider multiple performance indicators, and determine the optimal step length S * .
10. The adaptive ensemble learning classification method for imbalanced data according to claim 9, characterized in that: The performance indicators include accuracy, F1 score, AUC and cross entropy loss.