A dimension reduction and feature extraction method based on mutual information and genetic algorithm

Abstract

The present application relates to the technical field of data processing, and more particularly to a dimension reduction and feature extraction method based on mutual information and genetic algorithm, comprising using an improved mutual information formula to calculate mutual information to measure the role of each dimension for each class; using the mutual information value as the fitness value of the feature dimension, first using the roulette method in the genetic algorithm to generate multiple information-carrying feature subsets; then using mutual information to optimize the generated feature subsets in dimension; optimizing the feature subsets, controlling the mutation and difference degree of the optimized feature subsets, and generating new feature subsets; and fusing the evaluation results of the multi-source feature subsets. The present application considers the fixity of the dimensionality after transformation, the high efficiency and small influence of feature extraction, and the neglect of small features; the feature subset is mutated, the dimensionality of the reduced feature subset is changed, and the effective difference degree between the feature subsets is effectively controlled.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for dimensionality reduction and feature extraction based on mutual information and genetic algorithms. Background Technology

[0002] With the rapid development of the information age, information is presented in increasingly rich and comprehensive forms. Data from various fields exhibits characteristics such as large volume, diverse data types, low data value density, and high processing speed. However, this also brings problems such as the "curse of dimensionality" and "algorithm failure" to data processing. How to quickly extract valuable information for practical applications from this massive and diverse dataset is a hot topic in current data mining and pattern recognition research. Since the original representation of high-dimensional data often contains a lot of redundancy, and some variables are highly correlated, for such variables, a new set of fewer, uncorrelated variables can be found to represent the original data. Therefore, dimensionality reduction or feature extraction is an effective way to solve problems such as the curse of dimensionality in high-dimensional data.

[0003] Existing technologies have already conducted relevant research and discussions. Feature extraction and feature transformation are two commonly used methods for data dimensionality reduction. Feature transformation refers to transforming the original feature space into a certain form to obtain new features. Among them, the most famous feature transformation algorithm is principal component analysis. This algorithm has a considerable dimensionality reduction effect in most learning tasks, but it can only fix the dimensionality reduction, which has low flexibility. Moreover, the new features of feature transformation are usually derived from all the original features, which does not reduce the workload of data processing.

[0004] Feature extraction refers to selecting a subset of features from the original feature set that is optimal for a certain evaluation criterion. Through feature extraction, some features that are irrelevant to the task or redundant are removed. The algorithm is simple and efficient, but it may also ignore some features that are not very relevant to the task when they are individually used, but are very relevant when combined with other features. Summary of the Invention

[0005] To address the shortcomings of existing algorithms, this invention considers the fixity of the transformed dimension due to feature transformation, the high efficiency of dimensionality reduction in feature extraction, and the ignoring of features with little impact. It fully explores the correlation between features and categories through mutual information, generates multiple superior feature subsets using a genetic algorithm roulette wheel, and changes the dimension of the reduced feature subsets by mutating the feature subsets. This ensures that each feature subset can effectively distinguish data from different categories, while effectively controlling the degree of difference between feature subsets.

[0006] The technical solution adopted in this invention is: a dimensionality reduction and feature extraction method based on mutual information and genetic algorithms, comprising the following steps:

[0007] Step 1: Calculate mutual information using the improved mutual information formula to measure the role of each dimension in each class;

[0008] Furthermore, specifically including:

[0009] S11. The dataset is preprocessed using the maximization method;

[0010] S12. Using the equal-width method, calculate the number of each attribute feature in the dataset in each interval;

[0011] S13. Do not calculate the mutual information formula. To determine how each attribute affects each category.

[0012] Step 2: Use the mutual information value as the fitness value of the feature dimension. First, use the roulette wheel algorithm in the genetic algorithm to generate multiple initial feature subsets; then use the mutual information to optimize the dimensions of the generated feature subsets.

[0013] Furthermore, the generation of multiple initial feature subsets using the roulette wheel algorithm in genetic algorithms specifically includes:

[0014] The fitness value is obtained by summing the mutual information row by row, and the fitness value is divided by the sum of mutual information to obtain the selection probability of the feature.

[0015] Calculate the cumulative probability of each feature occurring;

[0016] A random number between 0 and 1 is generated. If the first feature is not selected and the cumulative probability of the first feature is greater than the random number, then the first feature is selected. Starting from the second feature and moving to the last feature, if the second feature is not selected and the cumulative probability of the feature preceding the second feature is less than the random number but the cumulative probability of the second feature is greater than or equal to the random number, then the second feature is selected. This process is repeated until the last feature is selected.

[0017] Furthermore, the dimensional optimization of the generated feature subset using mutual information specifically includes:

[0018] Find the selected features corresponding to the initial T feature subsets in the mutual information table, sum the mutual information values ​​by category, and divide by the total mutual information value of each category to obtain the contribution rate of the T initial feature subsets to each category.

[0019] If the number of features in the T initial feature subsets whose contribution rate is less than a threshold exceeds half the total number of categories, then it is a weak feature subset, and optimization is performed on the weak feature subset.

[0020] Furthermore, optimizing weak feature subsets specifically includes:

[0021] Record the feature subsets whose contribution rate is less than the contribution rate threshold and their corresponding category numbers;

[0022] Based on the category, the mutual information is sorted in reverse order to obtain the dimension effect sort;

[0023] The weak feature subset is subjected to a difference operation with each row of the dimension-based sorting of the selected features to obtain at least one candidate feature.

[0024] Each candidate feature in the candidate feature group is added to the weak feature subset, and the contribution rate is calculated again. If the number of categories with a contribution rate greater than the contribution rate threshold is greater than half of the total number of categories, then an optimized feature group is established.

[0025] Randomly select one optimized feature from the optimized feature group to supplement the weak feature subset.

[0026] Step 3: Optimize the feature subset, and then perform mutation and dissimilarity control on the optimized feature subset to generate a new feature subset;

[0027] Furthermore, specifically including:

[0028] From the optimized T feature subsets, two feature subsets are sequentially selected, namely the control feature subset and the feature subset to be mutated, and a first mutation probability and a second mutation probability are given.

[0029] When two feature subsets with a difference of less than 0.5 select the same feature, a random number between 0 and 1 is generated. If the random number is less than the first mutation probability, the same feature in the feature subset to be mutated is set to 0.

[0030] When two feature subsets with a difference of less than 0.5 do not select the same feature, a random number between 0 and 1 is generated. If the random number is less than the second mutation probability, the same unselected feature in the feature subset to be mutated is set to 1.

[0031] Step 4: Fusion of evaluation results for multi-source feature subsets;

[0032] Furthermore, specifically including:

[0033] The dataset is divided according to the final feature subset, and T different learning models are trained.

[0034] The final result is obtained by combining T learning models using a voting method.

[0035] The beneficial effects of this invention are:

[0036] 1. By measuring the effect of each feature dimension on the category through mutual information, and using mutual information as the fitness value of the feature dimension, combined with the genetic algorithm, multiple feature subsets with small intra-class differences and large inter-class differences are selected to form multi-source low-dimensional feature subsets. The evaluation results of multi-source feature subsets are integrated to achieve dimensionality reduction and feature extraction of high-dimensional data.

[0037] 2. This invention belongs to the filtering feature selection method. The entire feature selection process does not involve the results of the final classifier or learner. It belongs to the data preprocessing process in machine learning. Therefore, the time complexity is low and the whole process is relatively simple.

[0038] 3. The mutual information obtained from the solution is not only used as the fitness value to control the generation of the initial feature subset using the roulette wheel method, but also, after crossover and mutation, the mutual information is used as the contribution of the feature subset to the classification to optimize each subset.

[0039] 4. To address the issue that current feature selection algorithms only produce one optimal feature subset as the output, this invention uses multiple feature subsets to merge the output results. By combining the output results of each feature subset through voting and other methods, a final judgment result is formed, which significantly improves the accuracy compared to using the original dataset. Attached Figure Description

[0040] Figure 1 This is a flowchart of the dimensionality reduction and feature extraction method based on mutual information and genetic algorithm of the present invention;

[0041] Figure 2 This is a diagram illustrating the method of applying the present invention to dimensionality reduction or feature extraction results. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0043] The dimensionality reduction effect of this invention was verified using 10 publicly available datasets from the UCI Machine Learning Library. The 10 datasets are vehicle, sonar, mfeatfac, semeion, Isolet, CANE9, ORL, COIL20, warpPIE10p, and TOX171. Detailed descriptions of the characteristics of the datasets are shown in Table 1.

[0044] Table 1: Dataset

[0045] Serial Number Dataset Feature number Sample size Number of class tags 1 vehicle 18 946 4 2 sonar 60 208 2 3 mfeatfac 216 2000 10 4 semeion 256 1593 10 5 Isolet 617 1560 26 6 CANE9 857 1080 9 7 ORL 1024 400 40 8 COIL20 1024 1440 20 9 warpPIE10p 2420 210 10 10 TOX171 5749 171 4

[0046] The method of this invention measures the effect of each feature dimension on the category through mutual information, and uses mutual information as the fitness value of the feature dimension. Combined with a genetic algorithm, it generates feature subsets with high accuracy and high diversity, so as to achieve dimensionality reduction of high-dimensional data.

[0047] like Figure 1 As shown, a dimensionality reduction and feature extraction method based on mutual information and genetic algorithms includes the following steps:

[0048] Step 1: Calculate mutual information using the improved mutual information formula to measure the role of each dimension in each class;

[0049] S11. Data preprocessing, using formulas Divide each column by the maximum absolute value of each column, |max|, to achieve standardization of the original data;

[0050] S12. Using the equal-width method, divide the dataset into n intervals from min (dataset) to max (dataset), calculate the number of each attribute feature in each interval, and transform the continuous data into discrete data.

[0051] S13. Mutual information measures the degree of independence between two random variables. For two random variables X and Y, the joint probability density function is p(X,Y), and the marginal probability density functions are p(X) and p(Y). The mutual information I(X;Y) is the relative entropy between the joint distribution p(X,Y) and the integral distribution p(X)p(Y), calculated by the following formula:

[0052] Here, p(x) is calculated, which is the probability of each feature attribute appearing in the interval. The probability of each attribute in each interval is obtained by dividing the number of attributes in each interval calculated in S12 by the number of samples.

[0053] To find p(y), which is the probability of each category, first use unique to find the sample label class format, then classify according to the label value to find the number of samples in each class, and finally divide the number of samples in each class by the total number of samples.

[0054] To find the value of p(x,y), which is the probability that the feature attribute belongs to a class, find the value of p(x) in each interval according to the dimension, find the number of samples in the interval whose class is equal to the class label according to the class label, divide the obtained sample number by the sample number in the interval to get p(y|x), and finally multiply p(y|x) by p(x) to get p(x,y).

[0055] Change the mutual information formula, Not seeking This yields the effect of each attribute feature on each category; the higher the mutual information value, the greater the correlation between the attribute and the category.

[0056] Step 2: Use the mutual information value as the fitness value of the feature dimension. First, use the roulette wheel method in the genetic algorithm to generate multiple feature subsets with more information. Then, use the mutual information to optimize the dimensions of the generated feature subsets to generate feature subsets.

[0057] S21. Use the roulette wheel algorithm in the genetic algorithm to generate different feature subsets, and each feature attribute corresponds to a different individual in the genetic algorithm;

[0058] Specifically, it includes:

[0059] S211. The mutual information is summed row by row to obtain the feature importance, i.e. the individual fitness value. The feature importance is divided by the sum of mutual information to obtain the selection probability of the feature.

[0060] S212. Calculate the cumulative probability of each feature. In the genetic algorithm, the cumulative probability represents the sum of the selection probabilities of all individuals before each individual. Here, the feature attribute is treated as the individual.

[0061] S213. Randomly generate a number between 0 and 1. If the first feature is not selected and the cumulative probability of the first feature is greater than the random number, then select the first feature. From the second feature to the last feature, if a feature is not selected and the cumulative probability of the feature preceding it is less than the random number but the cumulative probability of the feature is greater than or equal to the random number, then select the feature.

[0062] S214. If N features need to be selected, repeat step S213 N times. The size of the feature subset, i.e., N, is set to half of the original feature subset.

[0063] S22. Utilize mutual information to optimize the generation of T feature subsets, with each feature subset having a size of N;

[0064] Specifically, it includes:

[0065] S221. Find the selected features corresponding to the optimized T feature subsets in the mutual information table, sum the mutual information values ​​by category, and divide by the total mutual information value of each category to obtain the contribution rate of the T feature subsets to each category.

[0066] S222. If the number of features in a subset of T features whose contribution rate is less than u (where u is a given contribution rate threshold) exceeds half the total number of categories, then the feature subset is called a weak feature subset and needs to be optimized. The optimization sub-steps are as follows.

[0067] S2221. Record the feature subsets whose contribution rate is less than u and their corresponding category numbers;

[0068] S2222. Based on the corresponding categories, sort the mutual information in reverse order to obtain the dimension effect sort;

[0069] S2223. Perform a difference operation on each row of the weak feature subset selected features and the dimension effect ranking to obtain at least one candidate feature. The candidate feature refers to the weak feature subset not selected but has a strong effect on the corresponding category with a contribution rate of less than u. Multiple candidate features constitute a candidate feature group.

[0070] S2224. Add each candidate feature in the candidate feature group to the weak feature subset, and calculate the contribution rate again. If the number of categories with a contribution rate greater than u is greater than half of the total number of categories, then the candidate feature can be used as an optimized feature. Multiple optimized features constitute an optimized feature group.

[0071] S2225. In the optimized feature group, randomly select one optimized feature to supplement the weak feature subset;

[0072] Step 3: Optimize the feature subsets and perform mutation and dissimilarity control on the optimized feature subsets to generate multiple feature subsets with high accuracy and high dissimilarity.

[0073] S31. From the T feature subsets optimized in step S2, take two feature subsets in sequence. The first feature subset is used as the control feature subset, and the second feature subset is used as the feature subset to be mutated. Given mutation probability 1 and mutation probability 2, where mutation probability 1 takes a larger value, ranging from 0.5 to 1, the purpose of which is to remove more features that were selected in both feature subsets. Mutation probability 2 takes a smaller value, ranging from 0 to 0.5, which means adding fewer features that were not selected in both feature subsets to the feature subset to be mutated.

[0074] S32. When two feature subsets with a difference of less than 0.5 select the same feature (the difference is the ratio of the total number of the two feature subsets that select the same feature and the total number of the two feature subsets that do not select the same feature to the total number of features), generate a random number between 0 and 1. If the random number is less than the given mutation probability of 1, then set the feature in the feature subset to be mutated to 0.

[0075] S33. When two feature subsets with a difference of less than 0.5 do not select the same feature, generate a random number between 0 and 1. If the random number is less than the given mutation probability of 2, then set the feature in the feature subset to be mutated to 1.

[0076] Step 4: Fusion of evaluation results for multi-source feature subsets;

[0077] S41. Divide the dataset according to feature subsets and train T different learning models;

[0078] S42. The final result is obtained by combining T learning models using a voting method.

[0079] To increase the credibility of the experimental results, the above 10 datasets all used a 10x cross-validation method to divide the training and test sets.

[0080] like Figure 2 As shown, based on the final generated feature subsets, the training and test sets are further divided into two parts.

[0081] The training model is obtained by using the KNN classifier to train both the training set and the second-partitioned training set.

[0082] The test set and the test set obtained from the second partition are fed into the trained model for testing and verification. The results of the test set obtained from the second partition are used to obtain the final test result by the relative majority voting method.

[0083] The test results are compared with the test set labels, and the classification accuracy is calculated as (number of correct test results / number of test set samples) * 100%. Table 2 shows the accuracy results of the original dataset and the dataset after dimensionality reduction using the KNN classifier. The classification accuracy for each iteration is obtained through 10-fold cross-validation, and the average is taken to obtain the average classification accuracy before the ± sign in the table. The value after the ± sign is the standard deviation of the accuracy. Each row in the table selects the feature with the highest average classification accuracy, indicated in bold. The last row, "Average," represents the average value of each algorithm across all datasets.

[0084] Table 2: Accuracy results of the KNN classifier before and after dimensionality reduction.

[0085]

[0086]

[0087] As can be seen from Table 2, except for the mfeatfac and Isolet datasets, the accuracy after dimensionality reduction is slightly lower than the original accuracy, but the difference is very small. The dimensionality reduction method proposed in this invention has a significant improvement in accuracy on other datasets.

[0088] In addition to comparing the results with the original dataset, the accuracy of the five different algorithms was also compared with other existing dimensionality reduction methods. The average classification accuracy of the five different algorithms on the KNN classifier is shown in Table 3, and the average classification accuracy of the five different algorithms on the SVM classifier is shown in Table 4.

[0089] Table 3: Accuracy results of the method of the present invention on the KNN classifier compared with other methods

[0090]

[0091]

[0092] Table 3 shows that, on the selected 10 datasets, compared to the RelaxFS algorithm, the algorithm of this invention decreased by 2.7% on the TOX171 dataset; however, it outperformed other algorithms on all other datasets. The last row of the table indicates that the algorithm of this invention achieves better average classification accuracy on the KNN classifier compared to other mutual information-based feature extraction algorithms.

[0093] Table 4: Accuracy results of the method of the present invention on the SVM classifier compared with other methods

[0094] Dataset CIFE RelaxFS CFR WCFR Current accuracy vehicle 64.94±0.07 63.47±0.07 65.01±0.07 65.15±0.07 68.56±0.07 sonar 77.35±0.02 77.99±0.02 78.74±0.03 79.34±0.03 76.85±0.09 mfeatfac 88.76±0.11 90.24±0.11 90.39±0.11 90.58±0.11 97.50±0.02 semeion 63.63±0.10 68.58±0.12 67.27±0.11 68.41±0.12 72.82±0.05 Isolet 58.11±0.11 68.96±0.15 69.94±0.15 72.49±0.16 93.12±0.05 CANE9 67.98±0.13 73.29±0.16 73.17±0.16 73.71±0.16 81.94±0.03 ORL 57.24±0.14 82.21±0.21 79.62±0.21 81.81±0.22 96.55±0.05 COIL20 87.11±0.14 89.95±0.14 88.42±0.15 90.23±0.15 99.50±0.10 warpPIE10p 93.71±0.12 93.67±0.12 94.09±0.12 94.92±0.12 100±0.00 TOX171 76.72±0.09 79.63±0.07 80.97±0.12 80.52±0.11 94.12±0.03 Average 73.56±0.10 78.80±0.12 78.76±0.12 79.72±0.13 88.09±0.05

[0095] Table 4 shows that the two best-performing feature extraction algorithms are the MCRMCR algorithm and the algorithm of this invention. The MCRMCR algorithm achieved the highest classification accuracy on one dataset, while the algorithm of this invention achieved the highest classification accuracy on nine datasets.

[0096] Comparing Tables 3 and 4, we can see the impact of different classifiers on the algorithm of this invention and other feature extraction algorithms. Comparing the results of the algorithm of this invention using two classifiers, it can be seen that the number of times the algorithm of this invention achieves better results with the KNN classifier and the SVM classifier is equal. Therefore, the preprocessing effect of the algorithm of this invention on the classifier is universal.

[0097] The superiority of the data dimensionality reduction method of this invention lies in its ability to effectively solve the problem of irrelevant feature information interfering with the target task, meet the dual requirements of accuracy and operational efficiency for adapting to large-scale datasets, and have strong adaptability to low-dimensional data.

[0098] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for dimensionality reduction and feature extraction based on mutual information and genetic algorithm, characterized in that, The method comprises the following steps: Step one, obtaining vehicle, sonar, mfeatfac, semeion, Isolet, CANE9, ORL, COIL20, warpPIE10p, TOX171 data sets; using the improved mutual information formula to calculate mutual information to measure the role of each dimension in each class in the data set; Step one specifically comprises: S11, using the maximum method to preprocess the data set; S12, using the equal width method, the number of each attribute feature in each interval in the data set is calculated; S13, the mutual information formula is not solved , get each attribute feature on each class role; Step two, taking the mutual information value as the fitness value of the feature dimension, first using the roulette method in the genetic algorithm to generate multiple initial feature subsets; The method comprises the following steps: The mutual information is summed by row to obtain the fitness value, and the fitness value is divided by the total mutual information to obtain the selected probability of the feature; The cumulative probability of each feature is calculated; A 0-1 random number is generated, if the first feature is not selected and the cumulative probability of the first feature is greater than the random number, the first feature is selected; from the second feature to the last feature, if the second feature is not selected and the cumulative probability of the second feature is less than the random number but the cumulative probability of the second feature is greater than or equal to the random number, the second feature is selected; and the cycle is repeated until the last feature is selected; Then the generated feature subsets are optimized in dimension by using mutual information; Specifically, it comprises: In the mutual information table, find the selected features corresponding to the initial T feature subsets, sum the mutual information values by category, and divide by the total category mutual information value to obtain the contribution rate of the T initial feature subsets to each category; If the number of feature subsets with a contribution rate less than the threshold value in the T initial feature subsets exceeds half of the total number of categories, the feature subset is a weak feature subset, and the weak feature subset is optimized; The optimization of the weak feature subset specifically comprises: Record the feature subsets with a contribution rate less than the contribution rate threshold value and the corresponding category number; According to the category, the dimension effect ranking is obtained by inverse order sorting of the mutual information; The difference set operation is performed between the selected features of the weak feature subset and each row of the dimension effect ranking to obtain at least one candidate feature; The weak feature subset is added with each candidate feature in the candidate feature group, and the contribution rate is calculated again, if the number of categories with a contribution rate greater than the contribution rate threshold value is greater than half of the total number of categories, an optimized feature group is established; An optimized feature is randomly selected from the optimized feature group and supplemented into the weak feature subset; Step three, the optimized feature subset is subjected to mutation and difference degree control to generate a final feature subset; Step three specifically comprises: Two feature subsets are sequentially taken from the optimized T feature subsets, which are respectively a control feature subset and a to-be-mutated feature subset, and a first mutation probability and a second mutation probability are given; When the two feature subsets with a difference degree less than 0.5 both select the same feature, a 0-1 random number is generated, if the random number is less than the first mutation probability, the same feature in the to-be-mutated feature subset is set to 0; When the difference degree of two feature subsets is less than 0.5 and the same feature is not selected in the two feature subsets, a random number between 0 and 1 is generated, and if the random number is less than the second mutation probability, the same feature not selected in the feature subset to be mutated is set to 1; Step four, the evaluation results of the multi-source feature subsets are fused, and a classification result is output by using a KNN classifier.

2. The method of dimensionality reduction and feature extraction based on mutual information and genetic algorithm according to claim 1, characterized in that, Step four specifically comprises: The data set is divided according to the final feature subset, and different T' learning models are trained; A final result is obtained by using a voting method to integrate the T' learning models.

Images