A feature selection method based on improved boruta algorithm

Abstract

The application provides a feature selection method based on an improved Boruta algorithm, and the steps are as follows: inputting a music emotion recognition real feature set without missing values and music emotion category labels, initializing an optimal feature subset and a highest classification accuracy; generating a shadow feature set and a mixed feature set; constructing a random forest classification model based on the mixed feature set; calculating the importance score of all features; marking whether the real features are important; saving the important features in the optimal feature subset based on the classification accuracy; updating the real feature set, repeating the above steps until all real features are marked as important or unimportant; and finally outputting the features in the optimal feature subset. The application aims at the problem that the Boruta algorithm does not consider the average reduction Gini coefficient and the prediction accuracy of features, and the features are comprehensively selected based on the average reduction accuracy, the average reduction Gini coefficient and the accuracy. Experiments prove that the method has better dimension reduction effect and higher classification accuracy.

Description

TECHNICAL FIELD

[0001] The application relates to a feature selection method based on an improved Boruta algorithm and belongs to the field of data feature engineering in machine learning. BACKGROUND

[0002] Machine learning has developed rapidly in recent years and is applied in many fields. In machine learning, features (such as the height and weight of a student) are inputs of training data and are very important for model training. In practical applications, there are usually many features in a training data set. Too many features will lead to a long time required for analyzing features and training a model and will also cause a 'dimension disaster', affecting the generalization of the model.

[0003] Feature selection can eliminate irrelevant or redundant features and finally select a feature subset. Training a model using the feature subset can obtain the best result. Therefore, feature selection plays an important role in reducing feature dimensions and improving model performance.

[0004] At present, common feature selection algorithms include filter method, wrapper method and embedded method. The Boruta algorithm in the wrapper method is a commonly used feature selection algorithm in machine learning. The algorithm selects features by comparing the average reduction accuracy of real features and shadow features, but the algorithm does not consider the average reduction Gini coefficient and prediction accuracy of features, thereby leading to poor model prediction effect. Therefore, the application considers three indexes of average reduction accuracy, average reduction Gini coefficient and prediction accuracy to screen features and improve the classification accuracy of the model. SUMMARY

[0005] The application provides a feature selection method based on an improved Boruta algorithm, aims to solve the problems of single feature importance evaluation standard and not considering classification accuracy in the Boruta algorithm, and proposes a feature selection method using average reduction accuracy, average reduction Gini coefficient and prediction accuracy as the feature importance measurement standard, which reduces the feature dimension and improves the classification accuracy.

[0006] In order to achieve the above purpose, the application adopts the following technical scheme:

[0007] Step 1: processing the original music emotion recognition data set, deleting missing values, and obtaining a music emotion recognition real feature set and a music emotion category label used for feature selection.

[0008] Step 2: taking the features in the music emotion recognition real feature set in step 1 as the input of a feature selection method based on an improved Boruta algorithm, and outputting the selected important features.

[0009] In step 2, the features in the music emotion recognition real feature set of step 1 are taken as the input of a feature selection method based on an improved Boruta algorithm, and the output is the selected important features, wherein the feature selection method based on the improved Boruta algorithm comprises the following steps:

[0010] Step 2-1: The music emotion recognition real feature set obtained in step 1 is specified as X, it is assumed that X has n columns of real features, the music emotion category label is y, the optimal feature subset B is initialized as an empty set, and the highest classification accuracy MaxAcc is 0.

[0011] Step 2-2: Generate a shadow feature set and a mixed feature set, copy the music emotion recognition real feature set X to generate an X' feature set, perform random row transformation on each column of feature data in X', generate a shadow feature set S, and the shadow feature set S also has n columns of features; finally, merge X and S to obtain a mixed feature set M containing 2*n columns of features.

[0012] Step 2-3: Use the mixed feature set M and the category label y to construct a random forest classifier, the number of base classifiers in the random forest classifier uses the default value 500, and the number of variables contained in each decision tree uses the default value log(N), N is the number of input features, that is, N=2*n. After the random forest classifier is trained, the average reduction accuracy MeanDecreaseAccuracy (MDA) and the average reduction Gini coefficient MeanDecreaseGini (MDG) of each feature are output.

[0013] Step 2-4: Calculate the importance score ImportanceScore (IS) of each feature based on the MDA and MDG obtained in step 2-3, and the calculation formula is:

[0014]

[0015] MDA max represents the maximum value of the average reduction accuracy MDA of the 2*n mixed features, MDA min represents the minimum value of the average reduction accuracy MDA of the 2*n mixed features, MDG max represents the maximum value of the average reduction Gini coefficient MDG of the 2*n mixed features, MDG min represents the minimum value of the average reduction Gini coefficient MDG of the 2*n mixed features.

[0016] Step 2-5: Label the importance of the real features, and it is assumed that {RIS i{1≤i≤n} are the IS results of n real features calculated by step 2-4, and the decision criterion for whether a real feature is important is:

[0017]

[0018] In formula (2), MaxSIS is the maximum value of the IS of n shadow features, and MeanSIS is the average value of the IS of n shadow features. That is, a real feature greater than the maximum value of the importance score of all shadow features is marked as an important feature, and a real feature less than the average value of the importance score of all shadow features is marked as an unimportant feature.

[0019] Step 2-6: Update the optimal feature subset based on the classification accuracy, select the real features with Label = "important" as the current important feature set T, and randomly divide the samples in T into 10 parts, each of which is tested once, and the remaining 9 parts are used as the training set, and the cross-validation is repeated 10 times. Each cross-validation uses a random forest classifier for model training, and the number of random forest classifiers uses the default value of 500, and the number of variables contained in each decision tree uses the default value of log(N), N is the number of input features. The classification accuracy calculation formula of the test set in each cross-validation is:

[0020]

[0021] Where j represents the number of cross-validation, TP represents the number of samples in the test set that are actually positive and predicted to be positive, TN represents the number of samples in the test set that are actually negative and predicted to be negative, FP represents the number of samples in the test set that are actually negative but predicted to be positive, and FN represents the number of samples in the test set that are actually positive but predicted to be negative.

[0022] Finally, the average of the classification accuracies of the 10 validation sets is taken as the current classification accuracy. If the current classification accuracy is greater than the maximum classification accuracy MaxAcc, the value of MaxAcc is updated to the current classification accuracy, and the "important" real features are saved to the optimal feature subset B; otherwise, they are not saved.

[0023] Step 2-7: Update the real feature set X, delete the real features with Label = "unimportant" from the real feature set X, and repeat steps 2-2 to 2-7 until all features are marked as important or unimportant.

[0024] Step 2-8: After the loop ends, output the optimal feature subset B, and the features in the optimal feature subset B are the feature screening results of the feature selection method based on the improved Boruta algorithm.

[0025] Compared with the prior art, the advantages of the present application are:

[0026] 1. The present application is a new method for feature selection of multi-dimensional data, which is suitable for most real-world scenarios, suitable for binary and multi-classification tasks, and has strong universality and versatility.

[0027] 2. The improved Boruta algorithm uses the sum of the average reduction accuracy and the average reduction Gini coefficient after normalization as the feature importance score, which is beneficial to selecting features more relevant to classification.

[0028] 3. The present application uses classification accuracy to determine the optimal feature subset, which improves the classification accuracy of the model compared to the improved Boruta algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 The flowchart of the present application.

[0030] Figure 2 The flowchart of the feature selection method based on the improved Boruta algorithm of the present application. DETAILED DESCRIPTION

[0031] In order to clearly and completely describe the technical solutions of the present application, the present application will be further described in detail in combination with the drawings.

[0032] As shown in Figure 1 The present application provides a feature selection method based on an improved Boruta algorithm, and the specific implementation process includes:

[0033] Step 1: Process the original music emotion recognition dataset, delete missing values, and obtain the music emotion recognition real feature set and music emotion category label for feature selection.

[0034] The music emotion recognition dataset is derived from the database for machine learning proposed by the University of California, Irvine (UCI database), and after deleting the missing values, a total of 400 data are obtained. The music emotion categories are divided into 4 categories, namely relaxation, happiness, sadness and anger. Each category has 100 samples. The features in the dataset are mainly acoustic features, including mel-frequency cepstral coefficients, rhythm, chroma, spectrum, harmonic features, etc., a total of 50, as shown in the second column "Features" in Table 1. The final input dataset contains 400 data and 50 features.

[0035] Step 2: The features in the music emotion recognition real feature set of step 1 are used as the input of the feature selection method based on the improved Boruta algorithm, and the output result is the selected important features.

[0036] Referring to Figure 2 The flowchart of the feature selection method based on the improved Boruta algorithm is shown in the figure, and the specific steps of feature selection are as follows:

[0037] Step 2-1: The music emotion recognition real feature set obtained in step 1 is designated as X, X has n columns of real features, the music emotion category label is y, the optimal feature subset B is initialized as an empty set, and the highest classification accuracy MaxAcc is 0, which is:

[0038] In this example, the music emotion recognition real feature set X has 50 columns of features. For the first iteration, the music emotion recognition real feature set X has 50 columns of features, the music emotion label y is divided into relaxation, happiness, sadness, and anger, and the sample number is 400.

[0039] Step 2-2: Generate shadow feature set and mixed feature set, copy the music emotion recognition real feature set X to generate X' feature set, and perform random row transformation on each column of feature data in X' to generate shadow feature set S. The shadow feature set S is also a feature set with n columns of features. Finally, X and S are merged to obtain a mixed feature set M containing 2*n columns of features, which is:

[0040] For the first iteration, 50 real features are generated into corresponding 50 shadow features through random row transformation, and the real features and shadow features are merged to obtain the mixed feature set M, which contains 400 sample rows and 100 column features.

[0041] Step 2-3: Use the mixed feature set M and the category label y to construct a random forest classifier. The number of base classifiers in the random forest classifier uses the default value 500, and the number of variables contained in each decision tree uses the default value log(N), N is the number of input features, that is, N=2*n. After the random forest classifier is trained, the average reduction accuracy MeanDecreaseAccuracy (MDA) and the average reduction Gini coefficient MeanDecreaseGini (MDG) of each feature are output, which is:

[0042] In the first iteration process, the mixed feature set M containing 400 sample rows and 100 column features is used to construct a random forest classifier. The number of base classifiers in the random forest classifier uses the default value 500, and the number of variables contained in each decision tree uses the default value log(100). After the random forest classifier is trained, the MDA and MDG of each feature are output. The MDA and MDG of the 50 real features and their corresponding 50 shadow features in the first iteration are recorded in columns 3 to 6 of Table 1.

[0043] Step 2-4: Calculate the importance score of each feature based on the MDA and MDG obtained in step 2-3, which is:

[0044] In the first iteration process, the importance scores of the 50 real features and their shadow features are calculated according to formula (1), and the results are shown in Table 1, columns 7 "Real Feature Importance Score" and column 8 "Shadow Feature Importance Score".

[0045] Step 2-5: Label the importance of real features, label the real features with importance scores greater than the maximum value of all shadow feature importance scores as important features, and label the real features with importance scores less than the average value of all shadow feature importance scores as unimportant features, which are as follows:

[0046] According to formula (3), the importance of real features is labeled. In the first iteration process, as shown in Table 1, column 8 "Shadow Feature Importance Score", the maximum value of the importance scores of the 50 shadow features is 0.0406, and the average value of the importance scores of the 50 shadow features is 0.022, so the real features with importance scores greater than 0.0406 are labeled as important, the real features with importance scores less than 0.022 are labeled as unimportant, and the remaining features are labeled as unknown. For example, for the feature numbered 2 "Lowenergy_Mean" feature, the real feature importance score of this feature is 0.0844, since 0.0844 is greater than 0.0406, it is labeled as "important"; for the feature numbered 10 "MFCC_Mean_6" feature, the real feature importance score of this feature is 0.0088, since 0.0088 is less than 0.022, it is labeled as "unimportant". The remaining features are labeled as "unknown", and the final feature labeling result after the first round of iteration is shown in the last column of Table 1 "Real Feature Importance Label".

[0047] Step 2-6: Update the optimal feature subset based on the classification accuracy, select the real features with Label = "important" as the current important feature set T, randomly divide the samples in T into 10 parts, each part is tested once, and the remaining 9 parts are used as the training set, cross-validation is repeated 10 times. Finally, the average of the classification accuracies of the 10 validation sets is taken as the current classification accuracy. If the current classification accuracy is greater than the maximum classification accuracy MaxAcc, update MaxAcc as the current classification accuracy, and save the real features labeled as "important" to the optimal feature subset B; otherwise, do not save. Specifically:

[0048] In the first iteration process, the real features marked as "important" are extracted, and the current feature set containing 400 rows of data and "important" real features is divided into 10 subsets, each containing 150 rows of data. Each subset is tested once, and the remaining 9 subsets are used as the training set. The random forest classifier is used for model training in each cross-validation, and the number of random forest classifiers is set to the default value of 500. Each decision tree contains log(33) variables, where 33 is the number of "important" real features. The classification accuracy of the test set is calculated using formula (4) after each training.

[0049] The classification accuracy of the 10 cross-validations is calculated to be 0.850, 0.950, 0.775, 0.700, 0.925, 0.875, 0.850, 0.825, 0.725, and 0.800, respectively. The average of these 10 numbers is 0.8275, and since 0.8275 is greater than MaxAcc = 0, the 33 "important" real features are saved to the optimal feature subset B, and the value of MaxAcc is updated to 0.8275.

[0050] Step 2-7: Update the real feature set X, remove the real features with Label = "not important" from the real feature set X, and repeat steps 2-2 to 2-7 until all features are marked as important or not important, specifically:

[0051] As shown in the last column "real feature importance label" in Table 1, after the first iteration, 2 unimportant real features are obtained, which are removed from the real feature set X. After deletion, X remains 48 real features, which are used as input for the next iteration, and steps 2-2 to 2-7 are repeated until all real features are marked as "important" or "not important".

[0052] Step 2-8: After the loop ends, the optimal feature subset is output, and the features in the optimal feature subset B are the feature selection results of the feature selection method based on the improved Boruta algorithm.

[0053] The last column "Invention" in Table 2 records the features selected by the algorithm of the present application, and "√" indicates the selected features.

[0054] The algorithm of the present application has the following advantages:

[0055] 1. As shown in Table 2, the algorithm of the present application selects 33 features, while the Boruta algorithm selects 47 features. The dimensionality reduction effect of the present algorithm is better.

[0056] 2. As shown in Table 3, using 33 features obtained by the application and 47 features obtained by the Boruta algorithm, respectively using ten-fold cross-validation method using random forest, XgBoost and SVM to build classification model, calculate the classification accuracy, compared with the Boruta algorithm, the application has higher accuracy in the case of less features.

[0057] 3. The application can not only be used for music emotion recognition, but also be used in other fields, as shown in Table 4, using the application algorithm and the Boruta algorithm to extract features from the Zoo, Credit and Sonar data sets in the UCI database, and using the random forest algorithm to build a classifier using ten-fold cross-validation. It is found that compared with the Boruta algorithm, the application has better dimension reduction effect and higher prediction accuracy on other field data sets.

[0058] In summary, the feature selection method based on the improved Boruta algorithm of the application has certain advantages in reducing feature dimension and improving model classification accuracy compared with the improved Boruta algorithm.

[0059] Those skilled in the art can make several changes and modifications to the embodiments under the premise of the application, and the above changes and modifications are within the protection scope determined by the claims.

[0060] Table 1: Importance score and importance label of the features in the first iteration of the application

[0061]

[0062] Table 2: Feature selection of the application algorithm and the Boruta algorithm

[0063] Feature Number Feature Boruta The Invention 1 RMSenergy_Mean √ 2 Lowenergy_Mean √ √ 3 Fluctuation_Mean √ √ 4 Tempo_Mean √ √ 5 MFCC_Mean_1 √ √ 6 MFCC_Mean_2 √ √ 7 MFCC_Mean_3 √ √ 8 MFCC_Mean_4 √ √ 9 MFCC_Mean_5 √ 10 MFCC_Mean_6 √ 11 MFCC_Mean_7 √ √ 12 MFCC_Mean_8 13 MFCC_Mean_9 √ √ 14 MFCC_Mean_10 √ √ 15 MFCC_Mean_11 √ √ 16 MFCC_Mean_12 17 MFCC_Mean_13 √ √ 18 Roughness_Mean √ √ 19 Roughness_Slope √ √ 20 Zero-crossingrate_Mean √ √ 21 AttackTime_Mean √ √ 22 AttackTime_Slope √ 23 Rolloff_Mean √ √ 24 Eventdensity_Mean √ √ 25 Pulseclarity_Mean √ √ 26 Brightness_Mean √ √ 27 Spectralcentroid_Mean √ √ 28 Spectralspread_Mean √ √ 29 Spectralskewness_Mean √ √ 30 Spectralkurtosis_Mean √ √ 31 Spectralflatness_Mean 32 EntropyofSpectrum_Mean √ √ 33 Chromagram_Mean_1 √ 34 Chromagram_Mean_2 √ 35 Chromagram_Mean_3 √ √ 36 Chromagram_Mean_4 √ √ 37 Chromagram_Mean_5 √ 38 Chromagram_Mean_6 √ 39 Chromagram_Mean_7 √ √ 40 Chromagram_Mean_8 √ 41 Chromagram_Mean_9 √ √ 42 Chromagram_Mean_10 √ 43 Chromagram_Mean_11 √ √ 44 Chromagram_Mean_12 √ 45 HarmonicChangeDetectionFunction_Mean √ √ 46 HarmonicChangeDetectionFunction_Std √ √ 47 HarmonicChangeDetectionFunction_Slope 48 HarmonicChangeDetectionFunction_PeriodFreq √ 49 HarmonicChangeDetectionFunction_PeriodAmp √ √ 50 HarmonicChangeDetectionFunction_PeriodEntropy √

[0064] Note: "√" represents the selected feature.

[0065] Table 3: Accuracy of the application and the Boruta algorithm applied to different classifiers

[0066] Classifier Boruta The Invention Random Forest 0.793 0.828 XgBoost 0.783 0.813 SVM 0.778 0.790

[0067] Table 4: Performance of the application and the Boruta algorithm applied to other field data sets

[0068]

Claims

1. A feature selection method based on an improved Boruta algorithm, characterized in that, The steps are as follows: Step 1: processing the original music emotion recognition dataset, deleting missing values, obtaining a music emotion recognition real feature set and a music emotion category label for feature selection; Step 2: taking the features in the music emotion recognition real feature set in step 1 as the input of a feature selection method based on an improved Boruta algorithm, and outputting the selected important features; The feature selection method based on the improved Boruta algorithm in step 2 has the following steps: Step 2-1: specifying the music emotion recognition real feature set obtained in step 1 as X, setting X to have n columns of real features, the music emotion category label as y, initializing the optimal feature subset B as an empty set, and the highest classification accuracy MaxAcc as 0; Step 2-2: generating a shadow feature set and a mixed feature set, copying the music emotion recognition real feature set X to generate an X' feature set, performing random row transformation on each column of feature data in X', generating a shadow feature set S, which is also a feature set with n columns of features; finally, merging X and S to obtain a mixed feature set M containing 2*n columns of features; Step 2-3: using the mixed feature set M and the category label y to construct a random forest classifier, the number of base classifiers in the random forest classifier using the default value 500, and the number of variables contained in each decision tree using the default value log(N), N being the number of input features, i.e. N = 2*n; the random forest classifier outputs the average reduction precision MeanDecreaseAccuracy (MDA) and the average reduction Gini coefficient MeanDecreaseGini (MDG) of each feature after training; Step 2-4: calculating the importance score ImportanceScore (IS) of each feature based on the MDA and MDG obtained in step 2-3, the calculation formula being: MDA max denotes the maximum value of the average reduction precision MD A of the 2*n mixed features, MD A min denotes the minimum value of the average reduction precision MD A of the 2*n mixed features, MDG max denotes the maximum value of the average reduction Gini coefficient MDG of the 2*n mixed features, MDG min denotes the minimum value of the average reduction Gini coefficient MDG of the 2*n mixed features; Step 2-5: Set {RIS i |1≤i≤n} are the IS results of the n real features calculated by step 2-4, and the decision criterion for whether a real feature is important or not is: In formula (3), MaxSIS is the maximum value of the IS of the n shadow features, and MeanSIS is the average value of the IS of the n shadow features; Step 2-6: updating the optimal feature subset based on the classification accuracy, selecting the real features with Label = "important" as the current important feature set T, and randomly dividing the samples in T into 10 parts, each of which is tested once, and the remaining 9 parts are used as the training set, with 10 times of cross-validation; each cross-validation uses a random forest classifier for model training, the number of random forest classifiers using the default value 500, and the number of variables contained in each decision tree using the default value log(N), N being the number of input features; the classification accuracy calculation formula for each cross-validation test set is: Where j represents the cross-validation number, TP represents the number of samples in the test set that are actually positive and predicted to be positive, TN represents the number of samples in the test set that are actually negative and predicted to be negative, FP represents the number of samples in the test set that are actually negative but predicted to be positive, and FN represents the number of samples in the test set that are actually positive but predicted to be negative; Finally, the average of the classification accuracies of the 10 validation sets is taken as the current classification accuracy; if the current classification accuracy is greater than the maximum classification accuracy MaxAcc, the "important" real features are saved to the optimal feature subset; otherwise, they are not saved. Step 2-7: Update the real feature set X, delete the real features with Label = "not important" from the real feature set X, and repeat steps 2-2 to 2-7 until all features are marked as important or not important. Step 2-8: After the loop ends, output the optimal feature subset, and the features in the optimal feature subset B are the feature screening results.

Images