Hybrid multi-objective optimization integrated feature selection method

Abstract

The invention discloses a hybrid multi-objective optimization integrated feature selection method, and relates to the technical field of multiple objectives. According to the method, a filtering type feature selection method and a package type feature selection method are effectively combined, association among all features is fully considered, and population initialization is carried out by adopting three filtering type feature selection methods including mutual information, variance threshold and chi-square verification; the problems of high calculation cost and overfitting of a traditional single package type feature selection method are solved; a multi-objective optimization algorithm-UNSGAIII package type method is adopted, and different from other optimization algorithms, the UNSGAIII method is an algorithm suitable for multi-objective tradeoff solving problems, so that the problems that multi-scheme solving can be provided for special users, and single filtering type feature selection prediction precision is low are solved; and the convergence speed and the global search capability are improved.

Description

Technical Field

[0001] The present invention relates to the field of multi-objective technology, and in particular to a hybrid multi-objective optimization integrated feature selection method. Background Art

[0002] With the development of modern medical technology and the exponential growth of large-scale biomedical data, how to effectively utilize this data and extract valuable information from it has become one of the current research hotspots. Feature selection is a key data preprocessing technology in data mining, which aims to select the most representative and high-information feature subsets from the original feature set, which is conducive to building efficient prediction models and extracting meaningful information. Feature selection of data not only reduces the number of features, but also improves the performance of the task model, helps to find important features for early identification and diagnosis of diseases, and has practical significance for identifying potential biomarkers, improving disease diagnosis and prognosis prediction.

[0003] Data dimensionality reduction is usually divided into two methods: feature extraction and feature selection. Feature extraction refers to mapping the original feature space to a new feature space through a certain transformation method. The new feature space usually has a lower dimension, but still retains the important information of the original data. Common feature extraction methods include principal component analysis, independent component analysis, linear discriminant analysis, etc. Feature selection is to select the most representative, most relevant or most predictive feature subset from the original feature set to reduce the data dimension and retain the most important information without changing the original structure of the data. Feature selection is usually divided into three methods: filtering, wrapping and hybrid.

[0004] A dataset consisting of N features can theoretically generate 2N feature subsets. Feature selection is to select features that are meaningful to a specific task from the original dataset to form a feature subset. The selected feature subset should not only have as few features as possible, but also make the model performance better. The feature selection process is roughly divided into four steps, namely, generating candidate subsets, evaluating subsets, termination conditions, and result verification.

[0005] Traditional single feature selection methods cannot fully consider the impact of feature subsets on specific learning tasks. Although the filtering feature selection method is computationally efficient, can perform feature selection independently of the model, and has no assumptions about the given learning algorithm, it does not consider the correlation and redundancy between features when selecting feature subsets, and cannot obtain the globally optimal feature subset. At the same time, the wrapping feature selection method is superior to the filtering feature selection method in terms of prediction performance, and directly uses the learning algorithm as the evaluation criterion, so it can select the globally optimal feature subset. However, since it needs to iteratively search the entire feature space and generate the optimal feature subset, it has a higher computational overhead than the filtering feature selection method, and there is also a risk of overfitting.

[0006] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0007] The purpose of the present invention is to provide a hybrid multi-objective optimization integrated feature selection method, which combines the advantages of filtering and wrapping feature selection algorithms. A filtering method is first used on the original feature subset to screen out a preliminary feature subset, and then the preliminary feature subset is carefully searched by a wrapping method to obtain the best feature subset. The test set data is imported into the model to obtain a higher accuracy, so as to solve the technical problems raised in the background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: a hybrid multi-objective optimization integrated feature selection method, comprising at least the following steps:

[0009] S1: Building a hybrid multi-objective optimization integrated feature selection - HMOEFS technical structure, which is used to select features with efficient classification significance from tabular digital data sets. The -HMOEFS technical structure includes a data preprocessing module, a parallel filtering feature selection module, a UNSGAIII feature selection module and a classification prediction module, and the classification prediction module is composed of an XGBoost classifier;

[0010] S2: The data set is processed and divided as input, and three feature subsets are selected through the parallel filtering selection module. The preliminary feature subsets after merging and deduplication are used as the initial population of the UNSGAIII module;

[0011] S3: In the UNSGAIII module, the population is gradually updated through non-dominated sorting, crowding distance calculation and combination of new parent populations to obtain the optimal Pareto solution set;

[0012] S4: Finally, the test set is classified and predicted based on the representative features selected by the Pareto solution.

[0013] Further, the data preprocessing module divides the data set into 85% and 15% according to the training set and the test set respectively;

[0014] The data preprocessing module is at least used to process data missing values, outliers and feature scaling;

[0015] When the data preprocessing module processes missing values, if there are few missing values, a small number of samples with missing values ​​can be removed, or the missing parts can be filled;

[0016] When the data preprocessing module processes abnormal values, the abnormal values ​​are processed using a Winsorization method.

[0017] Furthermore, the parallel filtering feature selection module includes three filtering feature selection algorithms, which are mutual information, variance threshold and chi-square test respectively. The parallel filtering feature selection module generates three feature subsets by combining the three filtering feature selections in parallel, and obtains the preliminary optimal feature subset by using feature subset merging and deduplication technology for the three feature subsets, which will improve the computational efficiency of subsequent training models.

[0018] Furthermore, the mutual information application process includes at least the following steps:

[0019] Mutual information is used to measure the mutual dependence between any two feature variables. If two random feature variables are independent of each other, their mutual information is 0.

[0020] If two feature variables are perfectly correlated, then the mutual information is equal to the entropy of one of the random feature variables;

[0021] For the selected preliminary feature subset, it is hoped that the feature with a larger mutual information value with the target variable is selected. The larger the mutual information value, the stronger the correlation between the feature and the target variable, and the more helpful it is in predicting the target variable.

[0022] In the mutual information method, the entropy of the feature set X is shown in Equation 1, where p i is x i ∈X probability density function;

[0023]

[0024] When finding the uncertainty of a given target variable Y and feature X, the conditional entropy H(X|Y) is as shown in Equation 2, where p(y,x) is the unified probability density function of Y and X;

[0025]

[0026] MI(X,Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)=H(X)+H(Y)-H(X,Y)(4)

[0027] The specific mutual information is shown in Equation 3 and Equation 4.

[0028] Furthermore, the process of applying the variance threshold comprises at least the following steps:

[0029] By calculating the variance threshold, taking the variance of the feature as the benchmark, some features with low variance are removed. Features with too small variance contribute less to the prediction ability of the model, because most of the values ​​of features with too small variance are concentrated in a small range, which is not conducive to the model distinguishing different samples.

[0030] A variance threshold is set, and features below the threshold are removed;

[0031] Assume that the threshold is set to h, for the i-th feature x i , as shown in Formula 5:

[0032]

[0033] If Var(x i )≤h, the feature will be removed;

[0034] in, is the mean of the i-th feature, and n is the total number of features in the data set.

[0035] Furthermore, the application process of the chi-square test includes at least the following steps:

[0036] By calculating the chi-square statistic between the feature and the target variable, the predictive ability of the feature on the target variable is evaluated. For a given feature and target variable, a Contingency table is constructed to count the observed and expected values, and then the chi-square statistic is calculated.

[0037] Assume that the data set has a total of N samples, the feature variable X has M different values, and the target variable Y has K different categories. It is necessary to construct an M×K Contingency table to indicate the number of samples corresponding to each value of feature X and each category of Y. For the element O in the i-th row and j-th column of the table, ij (Observed value) Calculate the corresponding expected value E ij , the expected value calculation formula is shown in Equation 6;

[0038]

[0039] Calculate the element O ij The expected value E ij , the chi-square statistic between feature X and target variable Y can be calculated as:

[0040]

[0041] Compare the chi-square statistic with the critical value corresponding to the degrees of freedom. If it is greater than the critical value, the null hypothesis is rejected and the feature is retained. A larger chi-square statistic value indicates that the feature has a strong correlation with the target variable. A smaller chi-square statistic value indicates a weaker correlation.

[0042] Furthermore, the application process of the feature subset merging and deduplication includes at least the following steps:

[0043] Three feature subsets are obtained by three feature selection algorithms, namely S1, S2, and S3. All subsets are merged into a large set S, that is, S = S1 ∪ S2 ∪ S3. The merged set S is deduplicated to obtain the deduplicated feature set F.

[0044] The specific merging and deduplication is shown in Equation 8;

[0045] F=((S1∪S2)∪S3)-((S1∩S2)∩S3)(8)

[0046] All sets are unioned, and then the intersection of all sets is subtracted. This ensures that the final F contains all the features selected by the algorithm and there are no duplicate features.

[0047] Furthermore, the HMOEFS technology includes three objectives: maximizing the mutual information between selected features, minimizing the number of selected features, and maximizing the classification prediction accuracy, and the UNSGAIII algorithm is used as the three objective optimization modules of the HMOEFS technology;

[0048] The UNSGAIII feature selection module is composed of a wrapped UNSGAIII algorithm and a filtering feature selection module. The process aims to find the optimal Pareto solution set. The UNSGAIII feature selection module selects a preliminary feature subset through filtering feature selection in the early stage, which saves a lot of time cost for the subsequent search for excellent initial populations. The UNSGAIII feature selection module at least includes generating the first generation population, merging the parent generation and the offspring generation, generating a new parent generation population, selecting, crossing, mutating, determining the number of iterations and outputting the Pareto solution. When the set number of iterations is reached, the search will stop;

[0049] The UNSGAIII algorithm is an upgraded algorithm of the NSGAII algorithm. On the basis of NSGAII, the reference point mechanism and widely distributed reference points are introduced to maintain the diversity of the population, and the non-dominated individuals of the population close to the reference point are retained.

[0050] Furthermore, the application of the UNSGAIII feature selection module comprises at least the following steps:

[0051] Generate the first generation population: In the filtering feature selection module, obtain a preliminary optimal feature subset, use this feature subset as the initial population of the UNSGAIII feature selection module, and then perform non-dominated sorting on the initial population, so that the individuals in the population can be divided into different levels according to the Pareto dominance relationship;

[0052] Next, let’s take the tth generation as an example. If P t It is the parent of the tth generation, with a size of N and a generated child of Q t , the size is also N;

[0053] First, combine the child and the parent, that is, R t =P t ∪Q t , R t The size is 2N, in R t Select N individuals from

[0054] To implement this selection process, first R t It is divided into multiple non-dominated layers (F1, F2, F3, ..., F h ), and then construct a new population S starting from F1 t , until its size reaches N or exceeds N for the first time. We call the last layer h layer, and the solutions above the h+1th layer will be eliminated. Usually, only part of the h layer will be accepted. In this case, it is necessary to use diversity to measure the solutions in the h layer for selection;

[0055] Merging of parent and offspring: In the UNSGAIII module, after the first generation of population is generated, starting from the second generation, each iteration will merge the parent generation P t With offspring Q t Merge into a new population R t , we will t Perform non-dominated sorting and calculate the crowding degree of individuals in each non-dominated layer. Select appropriate individuals to form a new parent population based on the non-dominated relationship and crowding degree.

[0056] Generate new parent population: When combining excellent individuals to form a new parent population, an elite selection strategy is used to select excellent individuals. In each generation of evolution, a portion of excellent individuals will be retained so that the next generation of population is selected from the combined population of the parent and child generations. This can prevent excellent solutions from being destroyed or lost, and can also speed up the convergence of the non-dominated solution set, thereby improving the convergence of the algorithm and the quality of the solution;

[0057] Crowding calculation: After the merger, the new population R t When performing non-dominated sorting, it is also necessary to calculate the individual crowding of each non-dominated layer. The crowding calculation is mainly to measure the similarity of each individual with other individuals in the target space. The greater the crowding, the more diverse the individuals are. The specific crowding distance calculation formula is shown in Equation 9;

[0058]

[0059] The number of objective functions is 3, which are maximizing the mutual information between features, minimizing the number of features in the selected feature subset, and maximizing the model classification prediction accuracy;

[0060] f m(i+1) and fm(i-1) are the function values ​​of the i+1 and i-1 individuals of the objective function m after the objective function values ​​are sorted, and f m(max) and f m(min) are the maximum and minimum values ​​of the objective function m respectively;

[0061] Divide the population according to the non-dominated layer: put the population members of the non-dominated layer level 1 to h into S t In, if |S t |=N, then directly P t +1 = S t ; if |S t |>N, then part of the next generation of solutions is P t +1=F1∪F2∪…∪F h-1 , the remaining part (K = N-|P t+1 |)From F h Select from;

[0062] Determination of reference points on the hyperplane: The UNSGAIII algorithm uses a set of predefined reference points to ensure the diversity of the solutions obtained. Using the system method of Das and Dennis, since the proposed HMOEFS technique optimizes three objectives simultaneously, the reference points are placed on a 3D plane. Each objective is divided into H parts, and the number of its reference points can be calculated by Equation 10;

[0063]

[0064] For a three-objective problem with H = 4, the reference points form a triangle. According to Equation 10, we can know that the 15 reference points generated are:

[0065] In the UNSGAIII algorithm, reference points are used to guide the diversity distribution of the population. Reference points are usually evenly distributed in the unit hypercube. Through appropriate resolution generation and normalization processing, the uniform distribution of the population in the entire target space is ensured, so that UNSGAIII can find a uniformly distributed Pareto optimal solution set in the high-dimensional target space. Reasonable distribution of ideal points and reference points is a key factor in ensuring the effectiveness of the UNSGAIII module.

[0066] Adaptive normalization of population individuals:

[0067] First, we need to identify each individual in S1∪S2∪S3∪...∪S τ The minimum value Zmin of the objective function i (i=1,2,3) on i , construct an ideal point To determine the population S t The ideal point is through Zmin i Subtract the target fi To translate each target value of St, the ideal point of St after translation becomes a zero vector, and the translated target is represented as f i '(x)=f i (x)-Zmin i ;

[0068] Then, the extreme points corresponding to each coordinate axis are found respectively. A three-dimensional linear hyperplane can be constructed through the extreme points. The formula for finding the extreme points is shown in Equation 11;

[0069] Finally, the intercept of the three-dimensional linear hyperplane and the coordinate axis can be used to adaptively normalize the objective function value;

[0070]

[0071] When i =0, use 10 -6 Replacement; the target direction after translation f i ', will generate an extreme value target vector z i ,max , and then use the three extreme value vectors to form a three-dimensional linear hyperplane, and calculate the intercept a between the i-th target axis and the linear hyperplane i , Figure 7 It is a schematic diagram of the intercepts of each target axis and the linear hyperplane; the objective function can be normalized as shown in Equation 12;

[0072]

[0073] Association operation: According to S in the target space t The size of the members,After adaptive regularization for each target, it is necessary to associate each population member with a reference point;

[0074] To do this, a reference line corresponding to each reference point on the hyperplane is defined by connecting the reference point with the origin;

[0075] Then, the vertical distance of each group member from each reference line was calculated;

[0076] The reference point whose reference line is closest to the group member in the normalized target space is considered to be associated with that group member;

[0077] By performing association operations, the UNSGAIII algorithm can determine the relative position of each individual in the three target spaces, which helps to select excellent solutions in the subsequent selection process to construct the next generation population;

[0078] The whole process helps maintain diversity and better approach the Pareto frontier;

[0079] Niche preservation operation: In UNSGAIII, the association operation associates each individual with the nearest reference point; each reference point and the individuals associated with it form a cluster category, which is called a niche; the goal of the niche preservation operation is to fill each niche so that it contains the required number of individuals, which can be selected from other niches to ensure that the number of individuals in each niche reaches the required size; when filling the niche, the individuals with the highest association with the reference point are usually selected to ensure that they are relatively close in the multi-objective space;

[0080] Selection: In the UNSGAIII module, the roulette wheel selection method is used to first calculate the fitness value f of each individual in the population i , and then calculate the selection probability of each individual. Selecting M individuals from a given population is equivalent to spinning the roulette wheel M times. There is no need to sort the individuals in the population before selecting them. In order to handle negative fitness values, the worst fitness value is set to f loss , if it is a negative value, then the probability of being selected is 0, and when the sum of the corrected fitness is 0, the selection probability of all individuals can be set to 1 / N;

[0081] Crossover: In the UNSGAIII module, the crossover operator used is a simulated binary crossover operator, which sets the two parents to A(a1, a2, ..., a n ) and B(b1,b2,...,b n ), the two offspring individuals C (c1, c2, ..., c n ) and D(d1,d2,...,d n ), can be further calculated by the following formula:

[0082]

[0083] Among them, β is dynamically and randomly determined by the distribution factor η according to the following formula. The larger the value of η, the greater the probability that the offspring individuals are close to the parent individuals;

[0084]

[0085] Therefore, the use of simulated binary crossover operator performs better in local optimization search;

[0086] Mutation: The mutation method used by the UNSGAIII module is the polynomial mutation method. The mutation operation simulates the gene mutation of organisms. Like the crossover operation, it is used to generate new individuals. The mutation form of the polynomial mutation operator is: i '=U i +ω(Ci -S i ), where ω is given by:

[0087]

[0088] Among them, ω1=(U i -S i ) / (C i -S i ),ω2=(C i -U i ) / (C i -S i ), C is a random number in the interval [0,1], β m is the distribution index.

[0089] Furthermore, the microhabitat preservation operation includes at least the following steps:

[0090] Determine the size of each niche, which is determined by the number of individuals P associated with the reference point j to be determined;

[0091] Select a microhabitat, determine a microhabitat set J with a minimum microhabitat count from the divided microhabitats, and use the microhabitat set J for the microhabitat operation;

[0092] For each selected niche j, repeat the following:

[0093] If the selected niche j is empty, that is, P j = 0, then it is necessary to select from other microhabitats F j Select an individual associated with a reference point j, select the individual with the shortest vertical distance to the reference line, and add it to the niche P t+1 Then, increase the count P of niche j in the set j ;

[0094] If niche j is not empty (P j ≥0), we need to randomly select an individual associated with reference point j from niche j and add it to niche P t+1 Then, increase the count P of niche j in the set j ;

[0095] Repeat the operation on each selected niche to ensure that the niche is filled; the whole process is repeated K times to ensure that P t+1 All empty positions in the collection are filled.

[0096] Compared with the prior art, the present invention has the following beneficial effects:

[0097] 1. In order to obtain a better population initialization scheme, the present invention proposes a new population initialization model. This model effectively combines the filtering feature selection method and the parcel feature selection method, fully considers the correlation between each feature, and uses three filtering feature selection methods, namely mutual information, variance threshold and chi-square verification, to initialize the population. Compared with the traditional evolutionary algorithm model, the present invention can collect the global optimal subset group faster, and then use the optimal subset group as the initial population of the UNSGAIII algorithm. The population initialization model proposed by the present invention solves the problems of high computational cost and overfitting of the traditional single parcel feature selection method.

[0098] 2. The present invention adopts a wrapper method of a multi-objective optimization algorithm, UNSGAIII. The UNSGAIII algorithm includes population initialization, non-dominated sorting, calculation of crowding distance, selection of parents, merging populations and offspring, generating new populations and updating Pareto frontiers. When the set number of iterations is reached or the optimal solution of the optimal Pareto frontier is obtained, the search will be stopped. The present invention studies three competing goals, and the three competing goals are: 1) maximizing the mutual information between selected features; 2) minimizing the number of selected features; 3) maximizing classification prediction accuracy. The greater the mutual information between features, the smaller the redundancy between features, which can indicate that the selected features are more meaningful for classification of the model, ensuring the diversity of the selected feature subsets; a smaller number of features can simplify the model, reduce computational costs and avoid the risk of overfitting; like other researchers, pursuing high classification accuracy is our common goal. High classification prediction accuracy means that the model is more accurate in classifying data, which can improve the practical application value of the model. The UNSGAIII algorithm is different from other optimization algorithms in that it is an algorithm suitable for solving multi-objective trade-off problems. It can provide a multi-solution solution for special users and the problem of low prediction accuracy of single filtering feature selection, and improves the convergence speed and global search capability.

[0099] 3. The present invention proposes an integrated feature selection model that combines a filtering feature selection model with a wrapping feature selection model in series. The model firstly initializes the features in the original data set through a population initialization model to obtain a preliminary optimal subset group, and uses the optimal subset group as the population input of the UNSGAIII module, and finally calculates the optimal Pareto solution, and finally uses the XGBoost classifier to perform classification prediction on the data set. The model solves the comprehensive solution problem of low prediction accuracy and high computational cost when using a single feature selection method. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for describing the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0101] Figure 1 It is a hybrid multi-objective optimization integrated feature selection structure diagram of the present invention;

[0102] Figure 2 This is a structural diagram of the data preprocessing module of the present invention;

[0103] Figure 3 This is a schematic diagram of feature subset merging and deduplication in the present invention;

[0104] Figure 4 It is the flow chart of UNSGAIII algorithm module of the present invention;

[0105] Figure 5 Optimizing the reference point distribution map for the three objectives of the present invention;

[0106] Figure 6 Schematic diagram of intercepts of target axes and linear hyperplanes of the present invention;

[0107] Figure 7 A schematic diagram showing the comparison of the microhabitat before and after filling of the present invention;

[0108] Figure 8 This is a visualization diagram of the accuracy of each method of the present invention on different data sets;

[0109] Fig. 9 It is a schematic diagram of Table 1 in the present invention;

[0110] Fig.10 It is a schematic diagram of Table 2 in the present invention;

[0111] Fig.11 It is a schematic diagram of Table 3 in the present invention;

[0112] Fig.12 It is a schematic diagram of Table 4 in the present invention. DETAILED DESCRIPTION

[0113] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0114] In this paper, we propose a new feature selection technique, called hybrid multi-objective optimization integrated feature selection (HMOEFS), which is used to select features with efficient classification significance from tabular digital data sets, as follows:

[0115] A hybrid multi-objective optimization integrated feature selection method includes at least the following steps:

[0116] S1: Building a hybrid multi-objective optimization integrated feature selection - HMOEFS technical structure, which is used to select features with efficient classification significance from tabular digital data sets, the -HMOEFS technical structure includes a data preprocessing module, a parallel filtering feature selection module, a UNSGAIII feature selection module and a classification prediction module;

[0117] S2: The data set is processed and divided as input, and three feature subsets are selected through the parallel filtering selection module. The preliminary feature subsets after merging and deduplication are used as the initial population of the UNSGAIII module;

[0118] S3: In the UNSGAIII module, the population is gradually updated through non-dominated sorting, crowding distance calculation and combination of new parent populations to obtain the optimal Pareto solution set;

[0119] S4: Finally, the test set is classified and predicted based on the representative features selected by the Pareto solution.

[0120] The structure of HMOEFS technology is as follows Figure 1 shown.

[0121] Data preprocessing module

[0122] In the invention we proposed, the data preprocessing module mainly deals with aspects such as missing data values, outliers, and feature scaling. When dealing with missing values, if there are few missing values, a small number of samples with missing values ​​can be removed, or the missing parts can be filled; when dealing with outliers, we use the Winsorization method to deal with outliers. The core idea of ​​this method is to limit the range of extreme values ​​in the data to a specific range, rather than completely deleting it. The specific approach is to replace the values ​​in the observations that are more extreme than a certain percentile with the values ​​of that percentile; in order to avoid the problem that some features dominate the objective function due to excessive values, the features of heteroscedasticity need to be standardized and normalized. Through data preprocessing, noise can be removed, data distribution can be balanced, outliers can be processed, and redundancy can be reduced, etc., to prepare for feature selection and subsequent modeling processes.

[0123] Filtering feature selection module

[0124] The filtering feature selection module is composed of three filtering feature selection algorithms in parallel, namely mutual information, variance threshold and chi-square verification algorithm. The module structure diagram is shown in Figure 2 As shown, we also need to have a certain understanding of the filtering feature selection algorithm:

[0125] The filtering feature selection method is a commonly used feature selection technique that evaluates and selects features through statistical indicators or methods independent of the learning algorithm. This method evaluates and ranks the features based on certain metrics and only selects the top few important features for modeling. Filtering feature selection has fast calculation speed and is suitable for modeling data with large sample sizes and high data dimensions. At the same time, since filtering feature selection has low prediction accuracy and ignores the interactions and dependencies between features, the feature subset selected by a single filtering feature selection is not reliable. Therefore, we integrate the three filtering feature selection methods in parallel as the population initialization of the subsequent wrapper algorithm, which can save more time cost and storage space.

[0126] Considering the compatibility of the three algorithms and the low redundancy between features, as well as the maximum correlation between features and classification labels, the present invention uses three methods, namely mutual information, variance threshold and chi-square verification. The three selected feature subsets are merged and deduplicated to obtain a preliminary optimal subset, which will improve the computational efficiency of the subsequent training model. The following will briefly introduce the three filtering feature selection methods:

[0127] (1) Mutual Information

[0128] In the present invention, we need to take into account the nonlinear dependency between features, maintain high correlation between features and categories, and low redundancy between features. Therefore, in the filtering feature selection module we proposed, we use the mutual information method to make the selected feature subset have high correlation and low redundancy.

[0129] We use mutual information to measure the mutual dependence between any two feature variables. If two random feature variables are independent of each other, their mutual information is 0; if two feature variables are completely correlated, then the mutual information is equal to the entropy of one of the random feature variables. For the selected preliminary feature subset, we all hope to select features with larger mutual information values ​​with the target variable. The larger the mutual information value, the stronger the correlation between the feature and the target variable, and the more helpful it is in predicting the target variable. In the mutual information method, the entropy of the feature set X is shown in Equation 1, where p i is x i The probability density function of ∈X.

[0130]

[0131] When finding the uncertainty of a given target variable Y and feature X, the conditional entropy H(X|Y) is as shown in Equation 2, where p(y,x) is the unified probability density function of Y and X.

[0132]

[0133] The specific mutual information is shown in Equation 3 and Equation 4.

[0134]

[0135] MI(X,Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)=H(X)+H(Y)-H(X,Y)(4)

[0136] (2) Variance threshold

[0137] In this invention, we calculate the variance threshold and use the variance of the feature as a benchmark to remove some features with low variance. Features with too small variance contribute less to the prediction ability of the model because most of the values ​​of features with too small variance are concentrated in a small range, which is not conducive to the model distinguishing different samples. Therefore, we set a variance threshold to remove features below the threshold. We assume that the threshold is set to h, and for the i-th feature x i ,have:

[0138]

[0139] If Var(x i )≤h, the feature will be removed. is the mean of the ith feature, and n is the total number of features in the data set. Since the variance threshold method used alone only considers the variance of a single feature, ignoring the correlation between the feature and the target variable and the redundancy between features, the present invention combines it with mutual information and chi-square validation to achieve better results.

[0140] (3) Chi-square verification

[0141] In the present invention, we evaluate the predictive ability of the feature for the target variable by calculating the chi-square statistic between the feature and the target variable. For a given feature and target variable, we construct a contingency table to count the observed values ​​and expected values, and then calculate the chi-square statistic. Assuming that our data set has a total of N samples, the feature variable X has M different values, and the target variable Y has K different categories, we need to construct an M×K contingency table, which indicates the number of samples corresponding to each value of feature X and each category of Y. For the element O in the i-th row and j-th column of the table, ij(observation value), we will calculate the corresponding expected value E ij , the expected value calculation formula is shown in Equation 6.

[0142]

[0143] Calculate the element O ij The expected value E ij , we can calculate the chi-square statistic between feature X and target variable Y as:

[0144]

[0145] We compare the chi-square statistic with the critical value corresponding to the degrees of freedom. If it is greater than the critical value, the null hypothesis (the feature is unrelated to the target) is rejected and the feature is retained. A larger chi-square statistic value indicates that the feature has a strong correlation with the target variable; otherwise, the correlation is weak.

[0146] (4) Feature subset merging and deduplication

[0147] The present invention uses three different filtering feature selection algorithms. The feature subsets obtained by each algorithm may be different. In order to obtain a more comprehensive and optimized feature subset, we need to merge and remove duplicates from these three feature subsets. The specific merging method is as follows: Figure 3 shown.

[0148] Three feature subsets are obtained by the three feature selection algorithms, namely S1, S2, and S3. We merge all the subsets into a large set S, that is, S = S1∪S2∪S3, and perform deduplication processing on the merged set S to obtain the deduplicated feature set F. The specific merging and deduplication is shown in Equation 8.

[0149] F=((S1∪S2)∪S3)-((S1∩S2)∩S3)(8)

[0150] All sets are unioned, and then the intersection of all sets is subtracted. This ensures that the final F contains all the features selected by the algorithm and there are no duplicate features.

[0151] UNSGAIII feature selection module

[0152] The present invention studies three competing goals, which are: 1) maximizing the mutual information between selected features; 2) minimizing the number of selected features; and 3) maximizing the classification prediction accuracy. The greater the mutual information between features, the smaller the redundancy between features, which can indicate that the selected features are more meaningful for the model, ensuring the diversity of the selected feature subsets. With the rapid development of current medical diagnosis, social media, online education, bioinformatics and other fields, the dimensions of data features show a slope-type growth trend. When selecting features for high-dimensional classified data, the number of selected features directly affects the computational efficiency of the model. A smaller number of features can simplify the model, reduce computational costs and avoid overfitting risks. Like other researchers, when doing feature selection classification tasks, pursuing high classification accuracy is our common goal. High classification prediction accuracy means that the model is more accurate in classifying data, which can improve the practical application value of the model. Especially in medical diagnosis or financial risk assessment tasks, high classification prediction accuracy will be more conducive to related companies making more informed decisions.

[0153] Since the present invention is designed for three objectives, and the UNSGAIII algorithm can optimize three objectives or multiple objectives simultaneously. Therefore, we will use the UNSGAIII algorithm as the three-objective optimization module of the HMOEFS technology. The flow chart of the UNSGAIII feature selection module is as follows: Figure 4 shown.

[0154] The UNSGAIII algorithm is a wrapper feature selection method, characterized in that the selected feature subset has higher prediction accuracy, but the single use of the UNSGAIII algorithm will cause a large computational overhead problem. Therefore, the present invention combines it with the filtering feature selection, and screens out the preliminary feature subset through filtering feature selection in the early stage, which saves a lot of time cost for the subsequent search for excellent initial populations. The UNSGAIII feature selection module mainly includes generating the first generation population, merging the parent generation and the offspring, generating a new parent population, selecting, crossing, mutating, determining the number of iterations and outputting the Pareto solution, and the search will be stopped when the set number of iterations is reached. In order to make it clearer that the HMOEFS technology uses the UNSGAIII module, the various processes of the UNSGAIII module will be briefly introduced:

[0155] UNSGAIII algorithm is an upgraded algorithm of NSGAII algorithm. On the basis of NSGAII, it introduces reference point mechanism and widely distributed reference points to maintain the diversity of population, and retains those non-dominated individuals close to the reference point.

[0156] In the filtering feature selection module, we get a preliminary optimal feature subset, which is used as the initial population of the UNSGAIII feature selection module. Then we will perform non-dominated sorting on the initial population, and we can divide the individuals in the population into different levels according to the Pareto dominance relationship. Next, we will use the tth generation to illustrate. If P t It is the parent of the tth generation, with a size of N and a generated child of Q t , the size is also N. First, combine the child and the parent, that is, R t =P t ∪Q t , R t The size is 2N, in R t In order to realize this selection process, firstly, R t It is divided into multiple non-dominated layers (F1, F2, F3, ..., F h ), and then construct a new population S starting from F1 t , until its size is N or exceeds N for the first time. We call the last layer the h layer, and the solutions above the h+1th layer will be eliminated. Usually, only part of the h layer will be accepted. In this case, it is necessary to use diversity to measure the solutions in the h layer for selection.

[0157] (1) Merging of parent and child generations

[0158] In the UNSGAIII module, after the first generation of population is generated, starting from the second generation, each iteration will t With offspring Q t Merge into a new population R t , we will t Perform non-dominated sorting and calculate the crowding degree of individuals in each non-dominated layer. Select appropriate individuals to form a new parent population based on the non-dominated relationship and crowding degree.

[0159] When combining excellent individuals to form a new parent population, we use an elite selection strategy to select excellent individuals. In each generation of evolution, a portion of excellent individuals will be retained so that the next generation of population can be selected from the combined population of the parent and child generations. This can prevent excellent solutions from being destroyed or lost, and can also speed up the convergence of the non-dominated solution set, thereby improving the convergence of the algorithm and the quality of the solution.

[0160] (2) Crowding calculation

[0161] In the new population R after the merger tWhen performing non-dominated sorting, we also need to calculate the individual crowding of each non-dominated layer. Calculating crowding mainly measures the similarity of each individual with other individuals in the target space. The greater the crowding, the more diverse the individuals are. The specific crowding distance calculation formula is shown in Equation 9.

[0162]

[0163] The number of objective functions in the present invention is 3, which are maximizing the mutual information between features, minimizing the number of features in the selected feature subset, and maximizing the model classification prediction accuracy. m(i+1) and f m(i-1) are the function values ​​of the i+1 and i-1 individuals of the objective function m after the objective function values ​​are sorted. m(max) and f m(min) are the maximum and minimum values ​​of the objective function m, respectively.

[0164] (3) Divide the population according to the non-dominant layer

[0165] Put the population members of non-dominated layer level 1 to h into S in sequence t In, if |S t |=N, then directly P t +1 = S t ; if |S t |>N, then part of the next generation of solutions is P t +1=F1∪F2∪…∪F h-1 , the remaining part (K = N-|P t+1 |)From F h Select in.

[0166] (4) Determination of reference points on the hyperplane

[0167] The UNSGAIII algorithm uses a set of predefined reference points to ensure the diversity of the solutions obtained. In this paper, we use the system method of Das and Dennis. Since our proposed HMOEFS technique optimizes three objectives simultaneously, the reference points are placed on a 3D plane. We divide each objective into H parts, and the number of its reference points can be calculated by Equation 10.

[0168]

[0169] For a three-objective problem with H = 4, the reference points form a triangle. According to Equation 10, the 15 reference points generated are as follows: Figure 5 shown.

[0170] In the UNSGAIII algorithm, reference points are used to guide the diversity distribution of the population. Reference points are usually evenly distributed within the unit hypercube. Through appropriate resolution generation and normalization processing, the uniform distribution of the population in the entire target space is ensured, so that UNSGAIII can find a uniformly distributed Pareto optimal solution set in the high-dimensional target space. Reasonable distribution of ideal points and reference points is a key factor in ensuring the effectiveness of the UNSGAIII module.

[0171] (5) Adaptive normalization of population individuals

[0172] First, we need to identify each individual in S1∪S2∪S3∪…∪S τ The minimum value Zmin of the objective function i (i=1,2,3) on i , construct ideal point To determine the population S t The ideal point is through Zmin i Subtract the target f i To translate each target value of St, the ideal point of St after translation becomes a zero vector. We represent the translated target as f i '(x)=f i (x)-Zmin i Then, we find the extreme points corresponding to each coordinate axis. We can construct a three-dimensional linear hyperplane through the extreme points. The formula for finding the extreme points is shown in Equation 11. Finally, we can adaptively normalize the objective function value by using the intercept of the three-dimensional linear hyperplane and the coordinate axis.

[0173]

[0174] When i =0, we can use 10 -6 Replace. The target direction after translation f i ', will generate an extreme value target vector z i,max , and then use the three extreme value vectors to form a three-dimensional linear hyperplane, and calculate the intercept a between the i-th target axis and the linear hyperplane i , Figure 6 is a schematic diagram of the intercepts of each target axis and the linear hyperplane. The objective function can be normalized as shown in Equation 12.

[0175]

[0176] (6) Association Operation

[0177] We use S in the target space tThe size of the member, after adaptive regularization for each target, each population member needs to be associated with a reference point. To do this, we define a reference line corresponding to each reference point on the hyperplane by connecting the reference point with the origin. Then, we calculate the perpendicular distance of each population member to each reference line. The reference point whose reference line is closest to the population member in the normalized target space is considered to be associated with the population member. By performing the association operation, the UNSGAIII algorithm can determine the relative position of each individual in the three target spaces, which helps us select excellent solutions in the subsequent selection process to build the next generation of population. The whole process helps to maintain diversity and better approach the Pareto frontier.

[0178] (7) Microhabitat preservation operation

[0179] In UNSGAIII, the association operation associates each individual with the nearest reference point. Each reference point and the individuals associated with it constitute a cluster category, which we call a microhabitat. The individuals in each microhabitat have similar objective function values ​​to the same reference point, so they are relatively close in the target space. In the selection operation, individuals are usually selected from different microhabitats to ensure that the selected solutions have a wide diversity and thus better approximate the Pareto frontier. The niche preservation operation is one of the key mechanisms for the UNSGAIII algorithm to maintain diversity.

[0180] The goal of the niche retention operation is to fill each niche to contain the required number of individuals, which can be selected from other niches to ensure that the number of individuals in each niche reaches the required size. When filling the niche, the individuals with the highest correlation with the reference point are usually selected to ensure that they are relatively close in the multi-objective space. The specific process of the retention operation will be described in detail below. The comparison diagram before and after the niche filling is shown in Figure 7 shown.

[0181] 1) Determine the size of each niche: The niche size is determined by the number of individuals P associated with the reference point j to be determined;

[0182] 2) Selecting microhabitats: We determine the microhabitat set J with the minimum microhabitat count from the already divided microhabitats, and we use the microhabitat set J for the microhabitat operation;

[0183] 3) Niche Operation: For each selected niche j, repeat the following operations:

[0184] ① If the selected niche j is empty, that is, P j = 0, then it is necessary to select from other microhabitats F jTo select an individual associated with a reference point j, we select the individual with the shortest vertical distance to the reference line and add it to the niche P. t+1 Then, increase the count P of niche j j .

[0185] ②If niche j is not empty (P j ≥0), we need to randomly select an individual associated with reference point j from niche j and add it to niche P t+1 Then, increase the count P of niche j j .

[0186] ③ Repeat the operation: The microhabitat operation described above will be repeated on each selected microhabitat to ensure that the microhabitat is filled. The whole process will be repeated K times to ensure that P t+1 All empty positions in the collection are filled.

[0187] 8) Selection, crossover, mutation

[0188] 1) Select

[0189] In the UNSGAIII module, we use the roulette wheel selection method, first calculating the fitness value f of each individual in the population i Then the selection probability of each individual is calculated. Selecting M individuals from a given population is equivalent to spinning the roulette wheel M times. There is no need to sort the individuals in the population before selecting them. In order to handle negative fitness values, we set the worst fitness value to f loss , if it is a negative value, then the probability of being selected is 0. When the sum of the corrected fitness is 0, the selection probability of all individuals can be set to 1 / N. The specific calculation formula is shown in Equation 13.

[0190]

[0191] 2) Cross

[0192] In the UNSGAIII module, the crossover operator we use is the simulated binary crossover operator. We set the two parents to be A(a1, a2, ..., a n ) and B(b1,b2,...,b n ), the two offspring individuals C (c1, c2, ..., c n ) and D(d1,d2,...,d n ) can be calculated by formula 14.

[0193]

[0194] Where β is dynamically and randomly determined by the distribution factor η according to formula 15. The larger the value of η, the greater the probability that the generated offspring individuals are close to the parent individuals. Therefore, the simulated binary crossover operator used in our invention performs better in local optimization search.

[0195]

[0196] 3) Mutation

[0197] In the present invention, the mutation method used by the UNSGAIII module is the polynomial mutation method, which was first proposed by Deb and is the main mutation operator used in the current study of multi-objective optimization problems. The mutation operation simulates the gene mutation of organisms. Like the crossover operation, it is used to generate new individuals. The mutation form of the polynomial mutation operator is: i '=U i +ω(C i -S i ), where ω is given by Equation 16.

[0198]

[0199] Among them, ω1=(U i -S i ) / (C i -S i ),ω2=(C i -U i ) / (C i -S i ), C is a random number in the interval [0,1], β m is the distribution index. In the present invention, we set the distribution index to 20.

[0200] Classification prediction module

[0201] The classifier used in the classification prediction module of the present invention is XGBoost, which is a boosting tree (BoostingTree) algorithm. In this multi-objective optimization feature selection task, the XGBoost classifier is mainly used to evaluate the performance of the candidate feature subset in classification accuracy, as an important goal in the optimization process. When the algorithm converges or reaches the maximum number of iterations, XGBoost can be used to evaluate the classification performance of the optimal individual (feature subset) on the test set, and draw accuracy line graphs of different methods. At the same time, it can also be compared with the XGBoost classifier without feature selection to view the effect of feature selection.

[0202] Experimental data introduction

[0203] In order to evaluate the HMOEFS method proposed in this invention, Fig. 9Table 1 is a brief introduction to the dataset. Table 1 shows the 13 datasets used in the experiment. These datasets are selected from the UCI machine learning repository and the Arizona State University (ASU) feature selection repository. Table 1 gives the research field and source of the data; the feature column gives the number of features ranging from dozens to thousands in different datasets, and the feature values ​​are all numerical types; at the same time, the number of classification categories and the number of samples of each dataset are also given.

[0204] Experimental results and analysis

[0205] The HMOEFS method proposed in the present invention includes three filtering feature selection algorithms, namely information gain (INFO), variance threshold (VAR) and chi-square test (CHI2), and a wrap-around multi-objective optimization feature selection UNSGAIII algorithm. In the present invention, we integrate the INFO, VAR and CHI2 algorithms in parallel to obtain a merge filtering method (MergeF). In the present invention, we conduct experimental comparisons on INFO, VAR, CHI2, MERGE-F, UNSGAIII and HMOEFS respectively, and finally calculate the average classification accuracy, standard deviation and number of selected features of each algorithm on all data sets. Fig.10 For Table 2, Fig.11 Table 3 is shown in Table 2 and Table 3. The specific experimental results are shown in Table 2 and Table 3.

[0206] Table 2 shows the classification accuracy and standard deviation of HMOEFS and each single algorithm. It can be seen from Table 2 that the HMOEFS algorithm proposed in the present invention has obtained higher classification accuracy than other algorithms on 13 data sets, and the average accuracy of the HMOEFS algorithm on all data sets is 3 percentage points higher than the second place. Among them, the UNSGAIII algorithm achieved the highest classification accuracy on 6 data sets and is the second best feature selection algorithm among the 6 algorithms. Table 3 shows the classification accuracy and number of selected features of HMOEFS and each single algorithm. It can be seen from Table 3 that the accuracy of each algorithm on the Das data set is 100%. UNSGAIII and HMOEFS algorithms only select 1 feature out of 37 features to ensure high accuracy. The average number of features selected by the HMOEFS algorithm on all data sets is far less than that of other algorithms, and it is nearly 100 features less than the second place. This fully proves that the HMOEFS algorithm has a higher efficiency in screening feature subsets than other single filtering and wrapping algorithms, and meets the multi-objective requirements of high accuracy and low feature quantity.

[0207] At the same time, we also experimentally compare the HMOEFS method proposed in this invention with other BDA, BWOA, HLBDA, ISSA, TMGWO and NSGAII wrapper methods. The experimental results are shown in Table 4. Fig.12 Table 4.

[0208] Table 4 shows the classification accuracy and standard deviation of HMOEFS and other advanced wrapper methods. As shown in Table 4, the BWOA algorithm achieved the best classification accuracy on 4 data sets, which is the second best algorithm among all algorithms. In all data sets, the average classification accuracy reached 89.82%; the BDA algorithm and the NSGAII algorithm achieved the best classification accuracy in Soybean and Colon, and the average classification accuracy in all data sets reached 88.08% and 88.77%, respectively; in contrast, the HMOEFS algorithm achieved better performance in classification accuracy than other methods, and obtained 10 best and 1 suboptimal in 13 data sets, and did not obtain the best on Sonar, Colon and Lung, and the gap with the best was very small. In all data sets, the average classification accuracy reached the best, which was 93.88%, 4.06% higher than the suboptimal BWOA algorithm; while the performance of HLBDA, ISSA and TMGWO algorithms was average, and only achieved the best performance on Soybean. Figure 8 This is a line graph of the accuracy of the six methods on 13 data sets. It can be seen from the figure that the HMOEFS method is ahead of other feature selection methods. This shows that the HMOEFS algorithm is superior to most other feature selection algorithms and has more in-depth research value in the future.

[0209] In summary, the present invention effectively combines the filtering feature selection method and the wrapping feature selection method, fully considers the correlation between the features, and uses three filtering feature selection methods, namely mutual information, variance threshold and chi-square verification, to initialize the population, thereby solving the problems of high computational cost and overfitting of the traditional single wrapping feature selection method.

[0210] And a multi-objective optimization algorithm - UNSGAIII wrapper method is adopted. Different from other optimization algorithms, the UNSGAIII method is an algorithm suitable for solving multi-objective trade-off problems. It solves the problem of providing a multi-solution solution for special users and low prediction accuracy of single filtering feature selection, and improves the convergence speed and global search capability.

[0211] An integrated feature selection method that combines the filtering feature selection model with the wrapping feature selection model in series is proposed. Finally, the XGBoost classifier is used to perform prediction on the data set, which solves the comprehensive solution problems of low prediction accuracy and high computational cost when using a single feature selection method.

[0212] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A hybrid multi-objective optimization integrated feature selection method, characterized by: At least the following steps are included: S1: Building a hybrid multi-objective optimization integrated feature selection - HMOEFS technical structure, which is used to select features with efficient classification significance from tabular digital data sets. The -HMOEFS technical structure includes a data preprocessing module, a parallel filtering feature selection module, a UNSGAIII feature selection module and a classification prediction module, and the classification prediction module is composed of an XGBoost classifier; S2: The data set is processed and divided as input, and three feature subsets are selected through the parallel filtering selection module. The preliminary feature subsets after merging and deduplication are used as the initial population of the UNSGAIII feature selection module; S3: In the UNSGAIII feature selection module, the population is gradually updated through non-dominated sorting, crowding distance calculation and combination of new parent populations to obtain the optimal Pareto solution set; S4: Finally, the test set is classified and predicted based on the representative features selected by the Pareto solution.

2. The hybrid multi-objective optimization integrated feature selection method according to claim 1, characterized in that: The data preprocessing module divides the data set into 85% and 15% according to the training set and the test set respectively; The data preprocessing module is at least used to process data missing values, outliers and feature scaling; When the data preprocessing module processes missing values, if there are few missing values, a small number of samples with missing values ​​can be removed, or the missing parts can be filled; When the data preprocessing module processes abnormal values, the abnormal values ​​are processed using a Winsorization method.

3. The hybrid multi-objective optimization integrated feature selection method according to claim 1, characterized in that: The parallel filtering feature selection module includes three filtering feature selection algorithms, which are mutual information, variance threshold and chi-square test respectively. The parallel filtering feature selection module generates three feature subsets by combining the three filtering feature selections in parallel, and obtains the preliminary optimal feature subset by using feature subset merging and deduplication technology for the three feature subsets, which will improve the computational efficiency of subsequent training models.

4. The hybrid multi-objective optimization integrated feature selection method according to claim 3, characterized in that: The mutual information application process comprises at least the following steps: Mutual information is used to measure the mutual dependence between any two feature variables. If two random feature variables are independent of each other, their mutual information is 0. If two feature variables are perfectly correlated, then the mutual information is equal to the entropy of one of the random feature variables; For the selected preliminary feature subset, it is hoped that the feature with a larger mutual information value with the target variable is selected. The larger the mutual information value, the stronger the correlation between the feature and the target variable, and the more helpful it is in predicting the target variable. In the mutual information method, the entropy of the feature set X is shown in Equation 1, where p i is x i ∈X probability density function; When finding the uncertainty of a given target variable Y and feature X, the conditional entropy H(X|Y) is as shown in Equation 2, where p(y,x) is the unified probability density function of Y and X; MI(X,Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)=H(X)+H(Y)-H(X,Y)(4) The specific mutual information is shown in Equation 3 and Equation 4.

5. The hybrid multi-objective optimization integrated feature selection method according to claim 3, characterized in that: The process of applying the variance threshold comprises at least the following steps: By calculating the variance threshold, taking the variance of the feature as the benchmark, some features with low variance are removed. Features with too small variance contribute less to the prediction ability of the model, because most of the values ​​of features with too small variance are concentrated in a small range, which is not conducive to the model distinguishing different samples. A variance threshold is set, and features below the threshold are removed; Assume that the threshold is set to h, for the i-th feature x i , as shown in Formula 5: If Var(x i )≤h, the feature will be removed; in, is the mean of the i-th feature, and n is the total number of features in the data set.

6. The hybrid multi-objective optimization integrated feature selection method according to claim 3, characterized in that: The application process of the chi-square test includes at least the following steps: By calculating the chi-square statistic between the feature and the target variable, the predictive ability of the feature on the target variable is evaluated. For a given feature and target variable, a Contingency table is constructed to count the observed and expected values, and then the chi-square statistic is calculated. Assume that the data set has a total of N samples, the feature variable X has M different values, and the target variable Y has K different categories. It is necessary to construct an M×K Contingency table to indicate the number of samples corresponding to each value of feature X and each category of Y. For the element O in the i-th row and j-th column of the table, ij (Observed value) Calculate the corresponding expected value E ij , the expected value calculation formula is shown in Equation 6; Calculate the element O ij The expected value E ij , the chi-square statistic between feature X and target variable Y can be calculated as: Compare the chi-square statistic with the critical value corresponding to the degrees of freedom. If it is greater than the critical value, the null hypothesis is rejected and the feature is retained. A larger chi-square statistic value indicates that the feature has a strong correlation with the target variable. A smaller chi-square statistic value indicates a weaker correlation.

7. The hybrid multi-objective optimization integrated feature selection method according to claim 1, characterized in that: The application process of merging and deduplication of feature subsets includes at least the following steps: Three feature subsets are obtained by three feature selection algorithms, namely S1, S2, and S3. All subsets are merged into a large set S, that is, S = S1 ∪ S2 ∪ S3. The merged set S is deduplicated to obtain the deduplicated feature set F. The specific merging and deduplication is shown in Equation 8; F=((S1∪S2)∪S3)-((S1∩S2)∩S3)(8) All sets are unioned, and then the intersection of all sets is subtracted. This ensures that the final F contains all the features selected by the algorithm and there are no duplicate features.

8. The hybrid multi-objective optimization integrated feature selection method according to claim 1, characterized in that: The HMOEFS technology includes three objectives: maximizing the mutual information between selected features, minimizing the number of selected features, and maximizing the classification prediction accuracy. The UNSGAIII algorithm is used as the three objective optimization modules of the HMOEFS technology; The UNSGAIII feature selection module is composed of a wrapped UNSGAIII algorithm and a filtering feature selection module. The process aims to find the optimal Pareto solution set. The UNSGAIII feature selection module selects a preliminary feature subset through filtering feature selection in the early stage, which saves a lot of time cost for the subsequent search for excellent initial populations. The UNSGAIII feature selection module at least includes generating the first generation population, merging the parent generation and the offspring generation, generating a new parent generation population, selecting, crossing, mutating, determining the number of iterations and outputting the Pareto solution. When the set number of iterations is reached, the search will stop; The UNSGAIII algorithm is an upgraded algorithm of the NSGAII algorithm. On the basis of NSGAII, the reference point mechanism and widely distributed reference points are introduced to maintain the diversity of the population, and the non-dominated individuals of the population close to the reference point are retained.

9. A hybrid multi-objective optimization integrated feature selection method according to claim 8, characterized in that: The application of the UNSGAIII feature selection module comprises at least the following steps: Generate the first generation population: In the filtering feature selection module, obtain a preliminary optimal feature subset, use this feature subset as the initial population of the UNSGAIII feature selection module, and then perform non-dominated sorting on the initial population, so that the individuals in the population can be divided into different levels according to the Pareto dominance relationship; Next, let’s take the tth generation as an example. If P t It is the parent of the tth generation, with a size of N and a generated child of Q t , the size is also N; First, combine the child and the parent, that is, R t =P t ∪Q t , R t The size is 2N, in R t Select N individuals from To implement this selection process, first R t It is divided into multiple non-dominated layers (F1, F2, F3, ..., F h ), and then construct a new population S starting from F1 t , until its size reaches N or exceeds N for the first time. We call the last layer h layer, and the solutions above the h+1th layer will be eliminated. Usually, only part of the h layer will be accepted. In this case, it is necessary to use diversity to measure the solutions in the h layer for selection; Merging of parent and offspring: In the UNSGAIII module, after the first generation of population is generated, starting from the second generation, each iteration will merge the parent P t With offspring Q t Merge into a new population R t , we will t Perform non-dominated sorting and calculate the crowding degree of individuals in each non-dominated layer. Select appropriate individuals to form a new parent population based on the non-dominated relationship and crowding degree. Generate new parent population: When combining excellent individuals to form a new parent population, an elite selection strategy is used to select excellent individuals. In each generation of evolution, a portion of excellent individuals will be retained so that the next generation of population is selected from the combined population of the parent and child generations. This can prevent excellent solutions from being destroyed or lost, and can also speed up the convergence of the non-dominated solution set, thereby improving the convergence of the algorithm and the quality of the solution; Crowding calculation: After the merger, the new population R t When performing non-dominated sorting, it is also necessary to calculate the individual crowding of each non-dominated layer. The crowding calculation is mainly to measure the similarity of each individual with other individuals in the target space. The greater the crowding, the more diverse the individuals are. The specific crowding distance calculation formula is shown in Equation 9; The number of objective functions is 3, which are maximizing the mutual information between features, minimizing the number of features in the selected feature subset, and maximizing the model classification prediction accuracy; f m(i+1) and f m(i-1) are the function values ​​of the i+1 and i-1 individuals of the objective function m after the objective function values ​​are sorted, and f m(max) and f m(min) are the maximum and minimum values ​​of the objective function m respectively; Divide the population according to the non-dominated layer: put the population members of the non-dominated layer level 1 to h into S t In, if |S t |=N, then directly P t +1 = S t ; if |S t |>N, then part of the next generation of solutions is P t +1=F1∪F2∪…∪F h-1 , the remaining part (K = N-|P t+1 |)From F h Select from; Determination of reference points on the hyperplane: The UNSGAIII algorithm uses a set of predefined reference points to ensure the diversity of the solutions obtained. Using the system method of Das and Dennis, since the proposed HMOEFS technique optimizes three objectives simultaneously, the reference points are placed on a 3D plane. Each objective is divided into H parts, and the number of its reference points can be calculated by Equation 10; For a three-objective problem with H = 4, the reference points form a triangle. According to Equation 10, we can know that the 15 reference points generated are: In the UNSGAIII algorithm, reference points are used to guide the diversity distribution of the population. Reference points are usually evenly distributed in the unit hypercube. Through appropriate resolution generation and normalization processing, the uniform distribution of the population in the entire target space is ensured, so that UNSGAIII can find a uniformly distributed Pareto optimal solution set in the high-dimensional target space. Reasonable distribution of ideal points and reference points is a key factor in ensuring the effectiveness of the UNSGAIII module. Adaptive normalization of population individuals: First, we need to identify each individual in S1∪S2∪S3∪...∪S τ The minimum value Zmin of the objective function i (i=1,2,3) on i , construct ideal point To determine the population S t The ideal point is through Zmin i Subtract the target f i To translate each target value of St, the ideal point of St after translation becomes a zero vector, and the translated target is represented as f i '(x)=f i (x)-Zmin i ; Then, the extreme points corresponding to each coordinate axis are found respectively. A three-dimensional linear hyperplane can be constructed through the extreme points. The formula for finding the extreme points is shown in Equation 11; Finally, the intercept of the three-dimensional linear hyperplane and the coordinate axis can be used to adaptively normalize the objective function value; When i =0, use 10 -6 Replacement; the target direction after translation f i ', will generate an extreme value target vector z i,max , and then use the three extreme value vectors to form a three-dimensional linear hyperplane, and calculate the intercept a between the i-th target axis and the linear hyperplane i , Figure 7 is a schematic diagram of the intercepts of each target axis and the linear hyperplane; the objective function can be normalized as shown in Equation 12; Association operation: According to S in the target space t The size of the members,After adaptive regularization for each target, it is necessary to associate each population member with a reference point; To do this, a reference line corresponding to each reference point on the hyperplane is defined by connecting the reference point with the origin; Then, the vertical distance of each group member from each reference line was calculated; The reference point whose reference line is closest to the group member in the normalized target space is considered to be associated with that group member; By performing association operations, the UNSGAIII algorithm can determine the relative position of each individual in the three target spaces, which helps to select excellent solutions in the subsequent selection process to construct the next generation population; The whole process helps maintain diversity and better approach the Pareto frontier; Niche preservation operation: In UNSGAIII, the association operation associates each individual with the nearest reference point; each reference point and the individuals associated with it form a cluster category, which is called a niche; the goal of the niche preservation operation is to fill each niche so that it contains the required number of individuals, which can be selected from other niches to ensure that the number of individuals in each niche reaches the required size; when filling the niche, the individuals with the highest association with the reference point are usually selected to ensure that they are relatively close in the multi-objective space; Selection: In the UNSGAIII module, the roulette wheel selection method is used to first calculate the fitness value f of each individual in the population i , and then calculate the selection probability of each individual. Selecting M individuals from a given population is equivalent to spinning the roulette wheel M times. There is no need to sort the individuals in the population before selecting them. In order to handle negative fitness values, the worst fitness value is set to f loss , if it is a negative value, then the probability of being selected is 0, and when the sum of the corrected fitness is 0, the selection probability of all individuals can be set to 1 / N; Crossover: In the UNSGAIII module, the crossover operator used is a simulated binary crossover operator, which sets the two parents to A(a1, a2, ..., a n ) and B(b1,b2,...,b n ), the two offspring individuals C (c1, c2, ..., c n ) and D(d1,d2,...,d n ), can be further calculated by the following formula: Among them, β is dynamically and randomly determined by the distribution factor η according to the following formula. The larger the value of η, the greater the probability that the offspring individuals are close to the parent individuals; Therefore, the use of simulated binary crossover operator performs better in local optimization search; Mutation: The mutation method used by the UNSGAIII module is the polynomial mutation method. The mutation operation simulates the gene mutation of organisms. Like the crossover operation, it is used to generate new individuals. The mutation form of the polynomial mutation operator is: i '=U i +ω(C i -S i ), where ω is given by: Among them, ω1=(U i -S i ) / (C i -S i ),ω2=(C i -U i ) / (C i -S i ), C is a random number in the interval [0,1], β m is the distribution index.

10. A hybrid multi-objective optimization integrated feature selection method according to claim 9, characterized in that: The microhabitat preservation operation comprises at least the following steps: Determine the size of each niche, which is determined by the number of individuals P associated with the reference point j to be determined; Select a microhabitat, determine a microhabitat set J with a minimum microhabitat count from the divided microhabitats, and use the microhabitat set J for the microhabitat operation; For each selected niche j, repeat the following: If the selected niche j is empty, that is, P j = 0, then it is necessary to select from other microhabitats F j Select an individual associated with a reference point j, select the individual with the shortest vertical distance to the reference line, and add it to the niche P t+1 Then, increase the count P of niche j in the set j ; If niche j is not empty (P j ≥0), we need to randomly select an individual associated with reference point j from niche j and add it to niche P t+1 Then, increase the count P of niche j in the set j ; Repeat the operation on each selected niche to ensure that the niche is filled; the whole process is repeated K times to ensure that P t+1 All empty positions in the collection are filled.

Images