Feature importance comprehensive screening method, system and equipment
By integrating multiple feature importance calculation methods and similarity analysis, a comprehensive similarity matrix is formed and community division is carried out, which solves the problem of traditional feature selection methods performing poorly in diverse application scenarios, and achieves more comprehensive feature screening and higher generalization capabilities.
Patent Information
- Application Number
- CN202411879310.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-19
- Publication Date
- 2025-04-15
AI Technical Summary
Traditional feature selection methods usually rely on a single algorithm or principle, resulting in poor performance in diverse application scenarios and inability to effectively handle nonlinear relationships.
A variety of different methods are used to calculate the importance of feature, form multiple feature sorting, and then the similarity between each pair of feature sorting is calculated by counting, Spearman, Kendall and cosine similarity methods to form a comprehensive similarity matrix. This matrix is used to construct a network graph, divide the community, and merge feature sorting through the waveda counting method to finally obtain the fusion sorting.
By integrating multiple feature importance calculation methods and similarity analysis, feature importance can be evaluated more comprehensively, the accuracy and generalization ability of feature screening can be improved, and it is suitable for diverse application scenarios.
Smart Images
Figure CN119829968A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of feature screening technology, and more specifically to a method, system and device for comprehensive screening of feature importance. Background Art
[0002] With the rapid development of machine learning and data analysis, feature selection as a preprocessing step has become increasingly important. It can not only simplify the model and increase the training speed, but also enhance the generalization ability and interpretability of the model.
[0003] However, most traditional feature selection methods are based on a single algorithm or principle, relying on the feature ranking obtained by its own logical calculation. For example, each method has its own specific application scenarios and assumptions. When these conditions are not met, its effect may be greatly reduced. Some methods may be sensitive to linear relationships, but perform poorly for nonlinear relationships, which limits their performance in diverse application scenarios.
[0004] Therefore, how to provide a method, system and equipment for comprehensive screening of feature importance is an urgent problem to be solved by those skilled in the art. Summary of the invention
[0005] In view of this, the present invention provides a method, system and device for comprehensive screening of feature importance to overcome the limitations of existing methods.
[0006] In order to achieve the above object, the present invention provides the following technical solutions:
[0007] A comprehensive feature importance screening method comprises the following steps:
[0008] Use a variety of different methods to calculate feature importance and obtain a variety of different feature rankings;
[0009] The similarity between each pair of feature rankings is calculated by four methods: counting, Spearman, Kendall, and cosine similarity, forming four sets of similarity matrices;
[0010] The four groups of similarity matrices are weighted and summed to obtain a comprehensive similarity matrix that integrates all information;
[0011] Obtaining an adjacency matrix based on the comprehensive similarity matrix;
[0012] Use the adjacency matrix to construct a network graph, and use the Louvain algorithm to divide the network graph into communities;
[0013] The Borda counting method was applied to merge the feature rankings within each community to obtain the representative ranking of each community;
[0014] The Borda counting method is applied to summarize the representative rankings of each community and obtain the final ranking based on the fused similarity matrix.
[0015] Furthermore, the feature importance is calculated using a variety of different methods, including:
[0016] Feature importance is calculated through various methods such as XGBoost built-in feature importance sorting, PermutationImportance permutation importance, Leave-one-out permutation, correlation analysis, recursive feature elimination, random forest built-in feature importance, principal component analysis, analysis of variance, chi-square test, shapley values sorting, mutual information, information gain, ridge regression, decision tree, linear discriminant analysis, autoencoder, T test sorting, and forward selection.
[0017] Furthermore, the four groups of similarity matrices are weighted and summed to obtain a comprehensive similarity matrix that integrates all information, and the expression is:
[0018] M final =λ1*M1+λ2*M2+λ3*M3+λ4*M4;
[0019] Among them, M i (i∈1,2,3,4) is the similarity matrix obtained by four similarities, λ i (i∈1,2,3,4) is the weight of each similarity matrix.
[0020] Furthermore, the integrated similarity matrix obtains an adjacency matrix, including:
[0021] The similarity results obtained based on the comprehensive similarity matrix are normalized and a threshold is set;
[0022] If the normalized similarity value is greater than the threshold, it is marked as 1, otherwise it is marked as 0, thus obtaining a set of adjacency matrices.
[0023] A comprehensive feature importance screening system, comprising:
[0024] Feature ranking module: uses a variety of different methods to calculate feature importance and obtain a variety of different feature rankings;
[0025] Similarity matrix generation module: Calculate the similarity between each pair of feature rankings through four methods: counting, Spearman, Kendall and cosine similarity, and form four sets of similarity matrices;
[0026] Similarity matrix synthesis module: weighted sum of four sets of similarity matrices to obtain a comprehensive similarity matrix that integrates all information;
[0027] Adjacency matrix construction module: obtaining an adjacency matrix based on the comprehensive similarity matrix;
[0028] Network graph construction module: Use the adjacency matrix to construct a network graph, and use the Louvain algorithm to divide the network graph into communities;
[0029] Clustering module: Apply the Borda counting method to merge the feature rankings within each community to obtain the representative ranking of each community;
[0030] Ranking result generation module: The Borda counting method is used to summarize the representative rankings of each community and obtain the final ranking based on the fusion similarity matrix.
[0031] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned method for comprehensive screening of feature importance are implemented.
[0032] It can be seen from the above technical solutions that compared with the prior art, the present invention discloses a method, system and device for comprehensive screening of feature importance, which adopts an integrated method for feature ranking obtained by different feature importance calculation methods, sorts the obtained features according to importance, and uses similarity analysis, complex network community division, boda counting and evolutionary algorithm and other methods to obtain a set of ultimate rankings containing all feature rankings and similarity information, and then uses the rankings again for feature screening, selects the top x features for training and prediction, and combines the advantages and thinking methods of multiple methods to obtain the most comprehensive prediction results. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0034] Figure 1 It is a schematic diagram of the method flow of the present invention;
[0035] Figure 2 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] The present invention essentially provides a method, system and device for comprehensive screening of feature importance, which is a method of fusion sorting. When applied in the field of feature screening, different existing feature importance calculation methods are actually used to obtain different importance sortings of a group of features. The feature sorting obtained by each feature importance calculation method is different, and the present invention does not limit the number of feature importance methods used, which can be customized. After obtaining the importance sorting of this group of features obtained by different methods, the purpose of the present invention is to merge and summarize these sortings to obtain a final sorting. This group of sortings integrates the results and experience of different feature importance methods, and finally selects the top x features as the data set according to this sorting for subsequent training models or other operations. Here, x can also be set according to demand. At the same time, in addition to being applied in the field of feature screening, the present invention can also be applied to any other place where different sortings of the same group of data need to be merged into one sorting. It is suitable for any task of structured feature screening. If there are m features, it is necessary to screen the top n features with the highest importance. Compared with other existing feature screening methods, other methods are very simple and rely on their own logical calculations to obtain feature rankings. The present invention designs a framework to summarize the results of different feature screening methods, learn from their strengths and weaknesses, and integrate them to obtain more comprehensive screening results.
[0038] See also Figure 1 , the embodiment of the present invention discloses a method for comprehensive screening of feature importance, comprising the following steps:
[0039] Use a variety of different methods to calculate feature importance and obtain a variety of different feature rankings;
[0040] The similarity between each pair of feature rankings is calculated by four methods: counting, Spearman, Kendall, and cosine similarity, forming four sets of similarity matrices;
[0041] The four groups of similarity matrices are weighted and summed to obtain a comprehensive similarity matrix that integrates all information;
[0042] Obtaining an adjacency matrix based on the comprehensive similarity matrix;
[0043] Use the adjacency matrix to construct a network graph, and use the Louvain algorithm to divide the network graph into communities;
[0044] The Borda counting method was applied to merge the feature rankings within each community to obtain the representative ranking of each community;
[0045] The Borda counting method is applied to summarize the representative rankings of each community and obtain the final ranking based on the fused similarity matrix.
[0046] Specifically, feature importance is calculated using a variety of different methods, including:
[0047] (1) XGBoost built-in feature importance ranking: Evaluate the importance of features by calculating the contribution of each feature in model training.
[0048] (2) Permutation Importance: The importance of features is evaluated by shuffling the order of feature values and observing the changes in model performance.
[0049] (3) Leave-one-out: Remove one feature at a time and observe the changes in model performance to evaluate the importance of the feature.
[0050] (4) Correlation analysis: Calculate the correlation coefficient between the feature and the target variable to evaluate the importance of the feature.
[0051] (5) Recursive Feature Elimination: Recursively consider smaller and smaller feature sets and select the features that contribute the most to the model.
[0052] (6) Random Forest built-in feature importance: The importance of a feature is evaluated by calculating the average impurity reduction of each feature in the tree.
[0053] (7) Principal Component Analysis (PCA): The importance of features is evaluated by the contribution of the principal components after dimensionality reduction to the original features.
[0054] (8) Analysis of variance (ANOVA): The importance of a feature is assessed by testing whether the means of different feature value groups are significantly different.
[0055] (9) Chi-square test: Test the independence between the feature and the target variable.
[0056] (10) Shapley values: The contribution of each feature to the model output is evaluated based on the Shapley value in cooperative game theory.
[0057] (11) Mutual information: measures the shared information between features and target variables.
[0058] (12) Information gain: In the decision tree algorithm, the importance of each feature is evaluated by calculating the information gain brought by each feature.
[0059] (13) Ridge regression: A regularized linear regression method that evaluates the importance of features through the regularization coefficient.
[0060] (14) Decision tree: Evaluate the importance of features based on a tree structure.
[0061] (15) Linear Discriminant Analysis (LDA): It evaluates the importance of features by maximizing the inter-class distance and minimizing the intra-class distance.
[0062] (16) Autoencoder: Reconstructs input data through unsupervised learning to evaluate the importance of features.
[0063] (17)T test: Test whether there is a significant difference between the feature and the target variable.
[0064] (18) Forward selection: Starting from an empty model, gradually add features and select the features that contribute the most to the model.
[0065] Specifically, in a specific embodiment, the following detailed steps are included:
[0066] (1) Using the above 18 methods for calculating feature importance, 18 different feature rankings are obtained.
[0067] Specifically, the input is processed through 18 methods for calculating feature importance to obtain 18 different feature rankings (or all other rankings that need to be fused). In a specific embodiment, the first column may be the 18 methods for calculating feature importance, and the second column and thereafter are all expressed in the form of rankings of the various features.
[0068] (2) Using counting, Spearman, Kendall and cosine similarity methods, calculate the similarity between each feature importance ranking and obtain four sets of corresponding similarity matrices. (Note: The calculation method of the similarity matrix can be changed and the number can also be changed)
[0069] Specifically, four similarity calculation methods are used to calculate the similarities between all the sorted pairs input in the first step to obtain a similarity matrix. Finally, four sets of such adjacency matrices, namely M1-M4, will be obtained.
[0070] (3) The weighted sum of the four similarity matrices is used to obtain a fused similarity matrix:
[0071] M final =λ1*M1+λ2*M2+λ3*M3+λ4*M4;
[0072] Among them, M i(i∈1,2,3,4) is the similarity matrix obtained by four similarities, λ i (i∈1,2,3,4) can be regarded as the weight of each similarity matrix.
[0073] (4) The similarity results are standardized and a threshold is set. If the standardized similarity value is greater than the threshold, it is marked as 1, otherwise it is marked as 0. Thus, a set of adjacency matrices is obtained.
[0074] Specifically, it is necessary to perform weighted fusion on the M1-M4 adjacency matrix obtained in the second step to obtain an adjacency matrix, that is, set M final =λ1*M1+λ2*M2+λ3*M3+λ4*M4, where M i (i∈1,2,3,4) is the similarity matrix obtained by four similarities, λ i (i∈1,2,3,4) can be regarded as the weight of each similarity matrix. Set a threshold threshold, if M final Each similarity value greater than the threshold is set to 1, otherwise it is set to 0, and a final adjacency matrix is obtained.
[0075] (5) A network is constructed based on the adjacency matrix, and the Louvain method is used to divide the communities. Each feature ranking within each community can be considered to have a certain similarity or correlation.
[0076] (6) Use Borda counting to aggregate the rankings within each community to obtain a ranking.
[0077] (7) The rankings obtained for each community are then summarized again using the Borda count, thus obtaining the final ranking under the fused similarity matrix.
[0078] (8) Final sorting is the final sorting obtained by integrating the sorting obtained by the above different methods in this patent. The purpose of feature screening is achieved by selecting the first x features of the final sorting, that is, selecting the most important x features obtained by integrating different methods.
[0079] Specifically, the λ of each similarity matrix from the second to the eighth step is i , the threshold and the number of selected features x are adaptively adjusted using evolutionary algorithms. i, threshold and x are used as gene points of the population for crossover, selection and mutation operations. The goal is fitness_result = 1 / accuracy + 1 / auc, so that fitness_result is maximized. Through algebraic iteration, the values of a group of gene points with the largest fitness_result are selected as parameters for sorting and the filtered features are obtained.
[0080] Specifically, the evolutionary algorithm is set to obtain the optimal parameters. i , threshold and select feature number x as the gene point of the population for crossover, selection and mutation operations, with the goal of fitness_result = 1 / accuracy + 1 / auc, to maximize fitness_result. That is, by setting the number of generations, the following steps are continuously iterated to modify the value of the gene point to maximize the target and finally obtain the optimal value of the gene point.
[0081] In a specific embodiment, the first step of the evolutionary algorithm is: the system first sets λ i , threshold and select feature number x, i+2 gene points to set an initial value. Then according to the obtained M final The adjacency matrix constructs a network, that is, the 18 sorting methods (or each sorting) are regarded as a network node. If two nodes are 1 in the adjacency matrix, they are connected by an edge, and if they are 0, they are not connected. The network graph is obtained.
[0082] Then use the Louvain method (this is a method for community division of the network, that is, each node is clustered according to the connection between the networks, and each node is divided into a finite number of communities, which is set to 3 in this experiment, and the community division method Louvain and the number of communities can be changed and personalized) to divide the community. For each feature ranking within each community, it can be considered to have a certain similarity or correlation.
[0083] The second step of the evolutionary algorithm: use Borda counting to aggregate the rankings within each community to obtain a ranking, where Borda counting is a voting counting method, that is, a final ranking is obtained based on different voting rankings. Here, Borda counting is used to aggregate the rankings within each community into a final ranking. In this way, because the number of communities set in this experiment is 3, three groups of final rankings will be obtained, and then Borda counting will be used again for these three groups of final rankings to obtain a final ranking.
[0084] The third step of the evolutionary algorithm: select the first x features of the feature sorting as the input of the machine learning model according to the value of x, and use the machine learning model to train and test the data set after feature screening. The evaluation indicators are accuracy and auc, that is, the target fitness_result = 1 / accuracy + 1 / auc is obtained.
[0085] The fourth step of the evolutionary algorithm: i , threshold and select feature number x, these i+2 gene loci are crossover, selection and mutation operations are performed, and the above three steps are repeated again, and the value of the gene locus with the largest target value is obtained by iterating the preset number of generations. The sorting results corresponding to these values are the final sorting results, and the first x features extracted by the final sorting are the features after the patent feature screening.
[0086] See also Figure 2 , a comprehensive feature importance screening system, comprising:
[0087] Feature ranking module: uses a variety of different methods to calculate feature importance and obtain a variety of different feature rankings;
[0088] Similarity matrix generation module: Calculate the similarity between each pair of feature rankings through four methods: counting, Spearman, Kendall and cosine similarity, and form four sets of similarity matrices;
[0089] Similarity matrix synthesis module: weighted sum of four sets of similarity matrices to obtain a comprehensive similarity matrix that integrates all information;
[0090] Adjacency matrix construction module: obtaining an adjacency matrix based on the comprehensive similarity matrix;
[0091] Network graph construction module: Use the adjacency matrix to construct a network graph, and use the Louvain algorithm to divide the network graph into communities;
[0092] Clustering module: Apply the Borda counting method to merge the feature rankings within each community to obtain the representative ranking of each community;
[0093] Ranking result generation module: The Borda counting method is used to summarize the representative rankings of each community and obtain the final ranking based on the fusion similarity matrix.
[0094] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned method for comprehensive screening of feature importance are implemented.
[0095] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0096] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for comprehensive screening of feature importance, characterized in that: The following steps are involved: Use a variety of different methods to calculate feature importance and obtain a variety of different feature rankings; The similarity between each pair of feature rankings is calculated by four methods: counting, Spearman, Kendall, and cosine similarity, forming four sets of similarity matrices; The four groups of similarity matrices are weighted and summed to obtain a comprehensive similarity matrix that integrates all information; Obtaining an adjacency matrix based on the comprehensive similarity matrix; Use the adjacency matrix to construct a network graph, and use the Louvain algorithm to divide the network graph into communities; The Borda counting method was applied to merge the feature rankings within each community to obtain the representative ranking of each community; The Borda counting method is applied to summarize the representative rankings of each community and obtain the final ranking based on the fused similarity matrix.
2. A method for comprehensive screening of feature importance according to claim 1, characterized in that: There are several different ways to calculate feature importance, including: Feature importance is calculated through various methods such as XGBoost built-in feature importance sorting, PermutationImportance permutation importance, Leave-one-out permutation, correlation analysis, recursive feature elimination, random forest built-in feature importance, principal component analysis, analysis of variance, chi-square test, shapley values sorting, mutual information, information gain, ridge regression, decision tree, linear discriminant analysis, autoencoder, T test sorting, and forward selection.
3. A method for comprehensive screening of feature importance according to claim 1, characterized in that: The weighted sum of the four groups of similarity matrices is performed to obtain a comprehensive similarity matrix that integrates all information, and the expression is: M final =λ1*M1+λ2*M2+λ3*M3+λ4*M4; Among them, M i( i∈1,2,3,4 ) The similarity matrix obtained by the four similarities, λ i( i∈1,2,3,4 is the weight of each similarity matrix.
4. A method for comprehensive screening of feature importance according to claim 1, characterized in that: The integrated similarity matrix obtains an adjacency matrix, including: The similarity results obtained based on the comprehensive similarity matrix are normalized and a threshold is set; If the normalized similarity value is greater than the threshold, it is marked as 1, otherwise it is marked as 0, thus obtaining a set of adjacency matrices.
5. A feature importance comprehensive screening system using the feature importance comprehensive screening method according to any one of claims 1 to 4, characterized in that: include: Feature ranking module: uses a variety of different methods to calculate feature importance and obtain a variety of different feature rankings; Similarity matrix generation module: Calculate the similarity between each pair of feature rankings through four methods: counting, Spearman, Kendall and cosine similarity, and form four sets of similarity matrices; Similarity matrix synthesis module: weighted sum of four sets of similarity matrices to obtain a comprehensive similarity matrix that integrates all information; Adjacency matrix construction module: obtaining an adjacency matrix based on the comprehensive similarity matrix; Network graph construction module: Use the adjacency matrix to construct a network graph, and use the Louvain algorithm to divide the network graph into communities; Clustering module: Apply the Borda counting method to merge the feature rankings within each community to obtain the representative ranking of each community; Ranking result generation module: The Borda counting method is used to summarize the representative rankings of each community and obtain the final ranking based on the fusion similarity matrix.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of a feature importance comprehensive screening method as described in any one of claims 1 to 4 are implemented.