Random forest improvement method based on feature importance

Abstract

The invention discloses a random forest improvement method based on feature importance, which performs classification and regression tasks with high accuracy, has high robustness and robustness, and comprises the following steps of: (1) calculating a feature importance weight; the method comprises the following steps: firstly performing minimum and maximum normalization, then rejecting features lower than a threshold value, and then performing normalization processing on the sum of importance values of the remaining features to obtain importance weights of the remaining features; (2) selecting a random subspace; and (3) dynamically adjusting the importance weight theta. And (4) determining a new random subspace by using the adjusted theta. And each tree uses respective random subspace to complete the construction of a decision tree and add the decision tree into a random forest. And finally, performing classification or regression tasks by using an improved random forest. Experimental results show that the method is superior to a conventional random forest method in the aspects of accuracy and stability, is particularly suitable for data of which the importance difference between the features is too large, and has practical and promotional values.

Description

Technical Field

[0001] The invention relates to a random forest improvement method based on feature importance, and belongs to the technical fields of artificial intelligence, machine learning and data mining. Background Art

[0002] Random Forests, an ensemble algorithm based on decision trees, has made significant progress in the fields of machine learning and data mining in recent years due to its concise algorithm design, easy implementation, and superior performance in processing high-dimensional data and complex data structures. Random Forests effectively improves the stability and prediction accuracy of the model by constructing multiple decision trees and performing ensemble voting, while reducing the risk of overfitting that may be caused by a single decision tree.

[0003] However, when the features with more information in the data set, that is, the proportion of important features is low, the performance of traditional random forests will drop significantly. The main reason for this performance drop is the feature selection bias in the design of traditional algorithms, that is, the algorithm tends to select features with less information but showing a certain degree of distinguishing ability in the current data distribution, resulting in the failure to fully utilize important features. In addition, in the process of selecting random subspaces, random forests are prone to introduce a large number of uninformative or low-information features. The presence of these irrelevant features not only increases the complexity of the model, but may also cause the decision tree to make unnecessary splits during the training process, thereby weakening the generalization ability and prediction accuracy of the model.

[0004] Furthermore, features with low information content may produce misleading split points during the splitting process of the decision tree, leading to irrational decision paths and thus affecting the overall prediction effect. At the same time, the lack of diversity in the random subspace also limits the advantage of random forests in improving model performance by integrating multiple decision trees. When the proportion of high-information features is low, the random subspace contains more uninformative features, resulting in a reduction in the differences between decision trees, which in turn weakens the effect of the integration method.

[0005] In summary, when dealing with data sets containing a low proportion of important features, the performance of traditional random forests is significantly reduced due to the bias of feature selection, the introduction of uninformative features, and insufficient diversity of random subspaces. Therefore, it is necessary to improve the random forest algorithm to enhance its adaptability and prediction performance in complex data environments, ensure that the model can fully utilize high-information features, and improve the overall prediction accuracy and generalization ability. Summary of the invention

[0006] The purpose of the present invention is to address the defects and shortcomings of the above-mentioned prior art and propose an improved random forest method based on feature importance. The method innovatively optimizes the feature importance and the selection of random subspaces, uses a permutation-based importance calculation method to evaluate feature importance, and combines a specific feature selection mechanism to play a key role in performance improvement.

[0007] The technical solution adopted by the present invention to solve the technical problem is a random forest improvement method based on feature importance, which specifically includes the following steps:

[0008] Step 1: Calculation of feature importance weights;

[0009] Based on the importance of the arrangement, the importance value β of each feature is calculated. After normalizing β to the minimum and maximum, β' is obtained, and the features with β' lower than the threshold α are discarded. The β' of the remaining features is normalized and summed to obtain the importance weight θ.

[0010] Step 2: Selection of random subspace;

[0011] Based on the importance weight θ, a certain number of features are selected from the remaining features without replacement and added to the random subspace of each tree.

[0012] Step 3: Dynamic adjustment of importance weight θ;

[0013] Use a sliding window with a step size of 1 to monitor the feature selection frequency of all trees in the window and adjust the feature importance weight θ in real time.

[0014] Step 4: Use the adjusted θ to determine the new random subspace;

[0015] Combined with the feature selection without replacement based on the importance weights θ of the remaining features in step 2 and the feature selection frequency monitored by the sliding window in step 3, the importance weights θ of the remaining features are dynamically adjusted. When the total number of trees is less than or equal to the size of the sliding window, the initial θ value calculated previously is directly used to select features in the random subspace. When the number of trees exceeds the size of the sliding window, the random subspace is reselected based on the dynamically adjusted and summed normalized θ value.

[0016] A further improvement of the present invention is that, in step 1, the specific steps of calculating the feature importance weights are as follows.

[0017] Step 1.1: Use the conventional random forest model to predict the data and obtain the conventional model error value.

[0018] Step 1.2: Based on the importance of permutation, shuffle the values ​​of each feature of the data, and use the conventional random forest model for prediction again to obtain the error value of the model under each feature.

[0019] Step 1.3: Subtract the error value of the general model from the error value of the model under each feature to obtain the importance value β of the feature, normalize β to the minimum and maximum value to obtain β', and discard the features whose β' is lower than the threshold α. Normalize the β' of the remaining features by summing up to obtain the importance weight θ of the remaining features, where θ represents the weight coefficient of feature selection. The weight θ obtained by summing up and normalizing here is called the initial θ value of the feature, which is used in the subsequent feature selection process.

[0020] A further improvement of the present invention is that, in step 2, the specific steps of selecting the random subspace are as follows.

[0021] Step 2.1: Determine the number of probabilistically selected features. In the present invention, random subspace refers to randomly selecting a feature subset from the entire remaining feature space when constructing each tree. This subset is fixed for use in the training process of the entire tree, and all nodes are split based on this feature subset. The features contained in the random subspace are obtained in a probabilistic selection manner according to the size of the weight coefficient θ, and are called probabilistically selected features. Comprehensively analyze the θ values ​​of all remaining features to determine the appropriate number of feature selections. By evaluating the overall distribution of θ values ​​and the relative importance of each feature, ensure that the number of selected features can effectively reflect the key features of the data while avoiding over-selection or under-selection.

[0022] Step 2.2: According to the θ value of each remaining feature, a certain number of features are randomly selected from the remaining features without replacement. Features with higher θ values ​​have a higher probability of being selected. This means that each feature can only be selected once in a selection process, avoiding repeated selection of features.

[0023] A further improvement of the present invention is that, in step 3, the specific steps of dynamically adjusting the importance weight θ are as follows.

[0024] Step 3.1: Construct a sliding window with a step size of 1, which is used to monitor the feature selection frequency of all features of each tree during the continuous tree construction process. The size of the sliding window is still considered comprehensively based on the total number of remaining features and the distribution of θ values. Within the sliding window, the feature selection frequency of each tree is continuously recorded to form a statistical data set of feature selection frequency.

[0025] Step 3.2: The system monitors features that appear repeatedly in the window. If the feature selection frequency of a feature in the sliding window is equal to the window size, the system will determine that the feature is overused, which may lead to overfitting of the model. To this end, the system will reduce the θ value of the repeated feature to reduce its probability of being selected in the subsequent feature selection process. The reduction in the θ value is dynamically adjusted according to the importance of different features in different data sets, but it can only be reduced to a preset minimum value.

[0026] Step 3.3: The system monitors the features that are not selected in the sliding window. If the feature selection frequency of a feature in the sliding window is equal to zero, it indicates that the feature may not become a probabilistic selection feature due to a low θ value. In order to increase the probability of selection of these unselected features, the system will increase their θ value. The increase in θ value is dynamically adjusted according to the importance of different features in different data sets, but it can only be increased to the preset maximum value.

[0027] Step 3.4: When the feature selection frequency of a feature is equal to half of the window size, the system will restore its θ value to the initial θ value, that is, the initial weight determined in the feature importance weight calculation step. This measure ensures that the dynamic adjustment of feature weights can be restored in time, maintaining the stability and adaptability of the model during the feature selection process.

[0028] Step 3.5: After all the θ values ​​are adjusted, the system will perform a sum normalization on the adjusted θ values. The purpose of sum normalization is to adjust the θ values ​​of all features to a uniform scale to ensure the comparability and fairness of weights in the subsequent feature selection process.

[0029] A further improvement of the present invention is that, in step 4, the specific steps of using the adjusted θ to determine a new random subspace are as follows.

[0030] Step 4.1: When constructing the random subspace in the initial stage, that is, when the total number of trees is less than or equal to the preset sliding window size, directly use the initial θ value calculated previously to select features in the random subspace. At this stage, there is no dynamic adjustment of the θ value.

[0031] Step 4.2: When the number of trees exceeds the size of the sliding window, the sliding window starts to slide and the system enters the dynamic adjustment stage. After that, the random subspace used by each newly generated tree will be determined based on the adjusted and normalized θ value of the previous sliding window. Specifically, after counting the selection frequencies of all features in the window and adjusting according to step 3, the new tree will use the adjusted θ and the normalized value to determine its own random subspace. When all probability selection features are selected, the window continues to slide.

[0032] Step 4.3: When the number of trees reaches the preset number, the sliding window stops and the value of θ no longer changes. All the trees selected according to the value of θ constitute the random forest based on the permutation feature importance and random subspace of the present invention.

[0033] Beneficial effects:

[0034] 1. The present invention calculates the importance weights of features and adopts a permutation-based feature importance method to ensure the use priority of high-information features in the model. By dynamically adjusting the feature selection process, features that contribute significantly to model performance can be more accurately identified and utilized, avoiding the interference of low-information features, thereby improving the prediction accuracy and efficiency of the model.

[0035] 2. The present invention introduces a sliding window mechanism to dynamically adjust the importance weights of features. Repeatedly selected features are monitored within the window to reduce their frequency of use, avoid excessive reliance on a high-weight feature, and prevent overfitting problems. In this way, the model can maintain the diversity of feature selection, thereby improving its generalization ability and adapting to changes in different data sets.

[0036] 3. Since the construction of random subspace is based on dynamically adjusted feature importance weights, the model can select different feature subsets in each tree, which greatly enhances the diversity of feature selection of the model. In this way, a feature is avoided from appearing repeatedly in all trees, thereby effectively reducing the correlation between features and improving the stability and reliability of the model.

[0037] 4. The mechanism of dynamically adjusting the feature importance weights enables the model to flexibly adapt to different data characteristics. During the training process, the weights of unselected features will be increased, thereby increasing their probability of being selected; while the weights of frequently selected features will be reduced. This mechanism improves the model's adaptability to various features, enhances the model's robustness, and avoids excessive reliance on specific features. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a system model flow chart of the present invention.

[0039] Figure 2 is the average accuracy of different thresholds α on different data sets.

[0040] Figure 3 Average accuracy on different datasets for different number of probability selection features. DETAILED DESCRIPTION

[0041] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0042] like Figure 1 As shown, the present invention provides a random forest improvement method based on feature importance, which includes the following steps:

[0043] Step 1: Calculation steps of feature importance weights:

[0044] Step 101: Use the conventional random forest model to predict the data and obtain the conventional model error value E

[0045] Step 102: Based on the importance of permutation, the values ​​of each feature of the data are shuffled, and the conventional random forest model is used for prediction again to obtain the error value of the model under each feature. The error value obtained by shuffling the i-th feature is set as E i .

[0046] Step 103, set β i =E i -E. β i is the importance value of each feature. i After minimum and maximum normalization, β' is obtained i , the formula is as follows:

[0047]

[0048] β' i Features smaller than the threshold α are discarded, and the sum of the remaining features is normalized to β' i , get the importance weights θ of the remaining features i , θ represents the weight coefficient of feature selection. The formula is as follows:

[0049]

[0050] Where F represents the set of remaining features, and the weight θ obtained by summing and normalizing is called the initial θ value of the feature, which is set to Used in the subsequent feature selection process.

[0051] Step 2, random subspace selection step:

[0052] Step 201, determine the number of probabilistically selected features. In the present invention, random subspace refers to randomly selecting a feature subset from the entire remaining feature space when constructing each tree. This subset is fixed for use in the training process of the entire tree, and all nodes are split based on this feature subset. The features contained in the random subspace are obtained in a probabilistic selection manner according to the size of the weight coefficient θ, which is called probabilistically selected features. The number of probabilistically selected features should be comprehensively considered based on the number of remaining features, the distribution of the initial θ of the remaining features, and other conditions.

[0053] Step 202: randomly select features from the remaining features in a no-replacement manner according to the θ values ​​of the remaining features.

[0054] Step 3, dynamic adjustment step of importance weight θ:

[0055] Step 301: construct a sliding window with a step size of 1. The window is used to monitor the feature selection frequency of all features of each tree during the continuous tree construction process. The feature selection frequency is set to f i The size of the sliding window is still considered comprehensively based on the total number of remaining features and the distribution of θ values, and is set to W.

[0056] In step 302, the system monitors the features that appear repeatedly in the window. If the feature selection frequency of a feature in the sliding window is equal to the window size, the system will determine that the feature is overused, which may lead to overfitting of the model. To this end, the system will reduce the θ value of the repeated feature to reduce its probability of being selected in the subsequent feature selection process. The reduction of the θ value is set to λ dec , dynamically adjusted according to the importance of different features in different data sets, but can only be reduced to a preset minimum value, set as λ min The formula is as follows:

[0057]

[0058] λ dec Dynamically adjust according to the importance of different features of different data sets.

[0059] Step 303, the system monitors the features that are not selected in the sliding window. If the feature selection frequency of a feature in the sliding window is equal to zero, it indicates that the feature may not become a probability selection feature due to a low θ value. In order to increase the probability of selection of these unselected features, the system will increase their θ value. The increase in θ value is set to λ inc , dynamically adjusted according to the importance of different features in different data sets, but can only be raised to a preset maximum value, set to λ max The formula is as follows:

[0060]

[0061] λ inc Dynamically adjust according to the importance of different features of different data sets.

[0062] Step 304: When the feature selection frequency of a feature is equal to half of the window size, the system will restore its θ value to the initial θ value, i.e. The formula is as follows:

[0063]

[0064] The formulas for steps 302, 303, and 304 are as follows:

[0065]

[0066] Step 305: After all θ values ​​have been adjusted, the system will perform a sum normalization process on the adjusted θ values, and the sum normalized value is set to θ i′ The formula is as follows:

[0067]

[0068] Step 4, use the adjusted θ to determine the new random subspace step:

[0069] Step 401, when constructing the random subspace in the initial stage, that is, when the total number of trees is less than or equal to the preset sliding window size, the initial θ value calculated previously is directly used to select features in the random subspace.

[0070] Step 402: When the number of trees exceeds the size of the sliding window, the sliding window starts to slide and the system enters the dynamic adjustment phase. After that, the random subspace used by each newly generated tree will be determined based on the adjusted and normalized θ value of the previous sliding window. Specifically, after counting the selection frequencies of all features in the window and adjusting according to step 3, the new tree will use the adjusted and normalized θ value, i.e., θ i′ To determine its own random subspace, when all probability selection features are selected, the window continues to slide.

[0071] Step 403, when the number of trees reaches a preset number, the sliding window stops and the value of θ no longer changes. All the trees selected according to the value of θ constitute the random forest based on the permutation feature importance and random subspace of the present invention.

[0072] The experiments of the present invention are all run in the python environment, and the data sets are Statlog (Vehicle Silhouettes), Adult, WDBC, Wine, and Spambase in the UCI machine learning library. The experimental results are obtained using 100 trees and ten five-fold cross validations, using average accuracy, average F1 score, and average Matthews Correlation Coefficient (MCC) as evaluation methods. The average accuracy, average F1 score, and average MCC of the present invention and the Breiman random forest method on different data sets are shown in Table 1, Table 2, and Table 3:

[0073] Table 1 Comparison of the average accuracy of the present invention and Breiman random forest method on different data sets

[0074] Dataset Breiman The present invention Statlog 0.7444 0.7503 Adult 0.8573 0.8640 WDBC 0.9526 0.9671 Wine 0.9775 0.9830 Spambase 0.9498 0.9535

[0075] Table 2 Comparison of average F1 scores between the present invention and Breiman random forest method on different data sets

[0076] Dataset Breiman The present invention Statlog 0.7373 0.7414 Adult 0.7921 0.8044 WDBC 0.9492 0.9645 Wine 0.9780 0.9832 Spambase 0.9472 0.9510

[0077] Table 3 Comparison of average MCC between the present invention and Breiman random forest method on different data sets

[0078] Dataset Breiman The present invention Statlog 0.6623 0.6703 Adult 0.5889 0.5961 WDBC 0.8996 0.9301 Wine 0.9665 0.9752 Spambase 0.8946 0.9024

[0079] The average accuracy of different thresholds α on different data sets is as follows: Figure 2 As shown, at this time, the number of probability selected features is half of the remaining features.

[0080] The average accuracy of the features selected by different number probabilities on different data sets is as follows: Figure 3 As shown, at this time, the threshold α is all 0.01.

[0081] In summary, the method of the present invention has simple steps and reasonable design. It changes the practice of ignoring high-importance features in conventional random forests and ensures the stability and robustness of the model.

[0082] The above contents are only preferred embodiments of the present invention and do not limit the present invention in any way. Any simple modifications, changes and equivalent structural changes made to the above embodiments based on the technical essence of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A random forest improvement method based on feature importance, characterized in that: The method comprises the following steps: Step 1: Calculation of feature importance weights; Based on the importance of the arrangement, calculate the importance value β of each feature, normalize β to get β', discard the features whose β' is lower than the threshold α, and sum and normalize the β' of the remaining features to get the importance weight θ; Step 2: Selection of random subspace; Based on the importance weight θ, a certain number of features are selected from the remaining features without replacement and added to the random subspace of each tree; Step 3: Dynamic adjustment of importance weight θ; Use a sliding window with a step size of 1 to monitor the feature selection frequency of all trees in the window and adjust the feature importance weight θ in real time; Step 4: Use the adjusted θ to determine the new random subspace; Combined with the feature selection without replacement based on the importance weights θ of the remaining features in step 2 and the monitoring of feature selection frequency through the sliding window in step 3, the importance weights θ of the remaining features are dynamically adjusted. When the total number of trees is less than or equal to the size of the sliding window, the initial θ value calculated previously is directly used to select features in the random subspace. When the number of trees exceeds the size of the sliding window, the random subspace is reselected based on the dynamically adjusted and summed normalized θ value.

2. The random forest improvement method based on feature importance according to claim 1, characterized in that: In step 1, the specific steps for calculating the feature importance weights are as follows: Step 1.1: Use the conventional random forest model to predict the data and obtain the conventional model error value; Step 1.2: Based on the importance of permutation, shuffle the values ​​of each feature of the data, and use the conventional random forest model for prediction again to obtain the error value of the model under each feature; Step 1.3: Subtract the error value of the conventional model from the error value of the model under each feature to obtain the importance value β of the feature. Normalize β to the minimum and maximum values ​​to get β'. Discard the features whose β' is lower than the threshold α. Normalize the β' of the remaining features by summing up to get the importance weight θ of the remaining features. θ represents the weight coefficient of feature selection. The weight θ obtained by summing up and normalizing is called the initial θ value of the feature, which is used in the subsequent feature selection process.

3. The random forest improvement method based on feature importance according to claim 1, characterized in that: In step 2, the specific steps of selecting the random subspace include: Step 2.1: Determine the number of probabilistically selected features. Random subspace refers to randomly selecting a feature subset from the entire remaining feature space when constructing each tree. This subset is fixed and used in the training process of the entire tree. All nodes are split based on this feature subset. The features contained in the random subspace are obtained by probabilistic selection according to the size of the weight coefficient θ, which is called probabilistically selected features. The θ values ​​of all remaining features are comprehensively analyzed to determine the appropriate number of feature selections. By evaluating the overall distribution of θ values ​​and the relative importance of each feature, it is ensured that the number of selected features can effectively reflect the key features of the data, while avoiding over-selection or under-selection; Step 2.2: According to the θ value of each remaining feature, a certain number of features are randomly selected from the remaining features without replacement. Features with higher θ values ​​have a higher probability of being selected, which means that each feature can only be selected once in a selection process, avoiding repeated selection of features.

4. The random forest improvement method based on feature importance according to claim 1, characterized in that: In step 3, the specific steps of dynamically adjusting the importance weight θ include: Step 3.1: Construct a sliding window with a step size of 1. This window is used to monitor the feature selection frequency of all features of each tree during the continuous tree construction process. The size of the sliding window is still considered comprehensively based on the total number of remaining features and the distribution of θ values. Within the sliding window, the feature selection frequency of each tree is continuously recorded to form a statistical data set of feature selection frequency; Step 3.2: The system monitors the features that appear repeatedly in the window. If the feature selection frequency of a feature in the sliding window is equal to the window size, the system will determine that the feature is overused, which may lead to overfitting of the model. The system will reduce the θ value of the repeated feature to reduce its probability of being selected in the subsequent feature selection process. The reduction of the θ value is set to λ dec , dynamically adjusted according to the importance of different features in different data sets, but can only be reduced to a preset minimum value, set as λ min , the formula is as follows: λ dec To dynamically adjust the importance of different features of different data sets; Step 3.3: The system monitors the features that are not selected in the sliding window. If the feature selection frequency of a feature in the sliding window is zero, it indicates that the feature may not become a probability selection feature due to a low θ value. In order to increase the probability of selection of these unselected features, the system will increase their θ value. The increase in θ value is set to λ inc , dynamically adjusted according to the importance of different features in different data sets, but can only be raised to a preset maximum value, set to λ max , the formula is as follows: λ inc To dynamically adjust the importance of different features of different data sets; Step 3.4: When the feature selection frequency of a feature is equal to half of the window size, the system will restore its θ value to the initial θ value, that is, the initial weight determined in the feature importance weight calculation step. This measure ensures that the dynamic adjustment of feature weights can be restored in time to maintain the stability and adaptability of the model during the feature selection process. When the feature selection frequency of a feature is equal to half of the window size, the system will restore its θ value to the initial θ value, that is The formula is as follows: Step 3.5: After completing the adjustment of all θ values, the system will perform sum normalization on the adjusted θ values. The purpose of sum normalization is to adjust the θ values ​​of all features to a unified scale range to ensure the comparability and fairness of weights in the subsequent feature selection process.

5. The random forest improvement method based on feature importance according to claim 1, characterized in that: In step 4, the specific steps of determining a new random subspace using the adjusted θ include: Step 4.1: When constructing the random subspace in the initial stage, that is, when the total number of trees is less than or equal to the preset sliding window size, the initial θ value calculated previously is directly used to select features in the random subspace. At this stage, there is no dynamic adjustment of the θ value. Step 4.2: When the number of trees exceeds the size of the sliding window, the sliding window starts to slide and the system enters the dynamic adjustment stage. After that, the random subspace used by each newly generated tree will be determined according to the adjusted and normalized θ value of the previous sliding window. Specifically, after counting the selection frequencies of all features in the window and adjusting according to the above step 3, the new tree will use the adjusted θ and the normalized value to determine its own random subspace. When all probability selection features are selected, the window continues to slide; Step 4.3: When the number of trees reaches the preset number, the sliding window stops and the value of θ no longer changes. All the trees selected according to the value of θ constitute a random forest based on the permutation feature importance and random subspace.

Images