A PSO-XGBoost system construction method with hybrid feature screening and hyperparameter optimization

Abstract

The present invention relates to a method for constructing a PSO-XGBoost system for hybrid feature screening and hyperparameter optimization, including: previewing and preprocessing a data set; using the Pearson correlation coefficient to analyze the correlation between features in the preprocessed data set, deleting features whose linear correlation in the original data set is greater than a preset threshold, and obtaining a feature subset; dividing the feature subset into a training set and a test set; performing feature selection on the training set and hybrid optimization of the hyperparameters of the XGBoost model based on the PSO algorithm, outputting the optimal feature subset and the optimal hyperparameter combination of the XGBoost model; constructing a PSO-XGBoost system based on the optimal feature subset and the optimal hyperparameter combination of the XGBoost model; testing the PSO-XGBoost system based on the test set, outputting a prediction result, and completing the construction of the PSO-XGBoost system when the prediction result meets the preset conditions. Its beneficial effect is that feature screening is realized when a machine learning algorithm processes big data to achieve parallel processing of the two key issues of data dimensionality reduction and model hyperparameter optimization.

Description

Technical Field

[0001] The present invention relates to the technical field of big data mining and machine learning, and in particular to a PSO-XGBoost system construction method for hybrid feature screening and hyperparameter optimization. Background Art

[0002] With the rapid development of Internet technology, massive amounts of data are generated and collected through various means, including social networks, e-commerce, and the Internet of Things. These data contain valuable information, which can be analyzed to gain business value, scientific development, and social insights. Big data mining technology involves extracting, converting, and presenting useful information and patterns from huge amounts of data, which requires the assistance of machine learning and artificial intelligence technologies.

[0003] However, the original data set is often huge in size and contains many redundant and irrelevant features. Using the original data set directly as the input of the machine learning model will result in a waste of time and computing resources. Secondly, the machine learning algorithm needs to adjust its own parameters when it is used. Since the parameters can directly control the behavior of the algorithm, different parameter values ​​will greatly affect the model performance. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a PSO-XGBoost system construction method for hybrid feature screening and hyperparameter optimization, which solves the technical problems of high dimensionality of original data sets and unclear optimal parameter configuration of the model in machine learning algorithms for big data mining.

[0006] (II) Technical solution

[0007] In order to achieve the above object, the main technical solutions adopted by the present invention include:

[0008] The present invention provides a PSO-XGBoost system construction method for hybrid feature screening and hyperparameter optimization, comprising:

[0009] Preview and preprocess the data set;

[0010] The Pearson correlation coefficient is used to analyze the correlation between the features in the preprocessed data set, and the features with linear correlation greater than the preset threshold in the original data set are deleted to obtain the feature subset;

[0011] Divide the feature subsets into training and test sets;

[0012] Based on the PSO algorithm, the feature selection of the training set and the hyperparameter hybrid optimization of the XGBoost model are performed to output the optimal feature subset and the optimal hyperparameter combination of the XGBoost model;

[0013] Construct a PSO-XGBoost system based on the optimal feature subset and the optimal hyperparameter combination of the XGBoost model;

[0014] The PSO-XGBoost system is tested based on the test set and the prediction results are output. When the prediction results meet the preset conditions, the construction of the PSO-XGBoost system is completed.

[0015] Optionally, feature selection of the training set and hybrid optimization of the hyperparameters of the XGBoost model are performed based on the PSO algorithm, and the optimal feature subset and the optimal hyperparameter combination of the XGBoost model are output, including:

[0016] Initialize the particle swarm parameters in the PSO algorithm, where the particle swarm parameters include: particle swarm size, particle dimension, maximum number of iterations, inertia weight, learning factor, population range, and iteration step range;

[0017] Randomly initialize the position and velocity of each particle in the particle swarm;

[0018] Determine the individual's best historical position and the individual's best historical fitness, the group's best historical position and the group's best historical fitness;

[0019] During the iteration, the position and velocity of each particle are updated and the fitness of each particle is calculated;

[0020] During the iteration process, when the fitness of the particle at the current moment is greater than the individual historical optimal fitness of the particle, the individual historical optimal fitness and individual historical optimal position of the particle are updated; when the maximum value of the individual historical optimal fitness of the current particle is greater than the group historical optimal fitness, the group historical optimal fitness and group historical optimal position are updated;

[0021] When the maximum number of iterations is reached, the current group's historical optimal fitness and the current group's historical optimal position are output as the optimal solution;

[0022] The optimal solution is divided into two parts, the first part is converted into the optimal feature subset based on the feature indicator function, and the second part is used as the optimal hyperparameter combination of the XGBoost model.

[0023] Optionally, the fitness is calculated according to the following formula:

[0024]

[0025] Among them, ACC is the classification accuracy of the model; S selected is the number of features selected in each calculation, S total is the total number of features in the data set, α∈(0,1).

[0026] Optionally, the particle position is updated according to the following formula:

[0027]

[0028] in, is the position of the particle at the next moment, is the particle's current position.

[0029] Optionally, the particle velocity is updated according to the following formula:

[0030]

[0031] in, is the velocity of the particle at the next moment, is the velocity of the particle at the current moment, ω is the inertia weight, c1 is the individual learning factor, c2 is the group learning factor, r1 and r2 are random numbers, r1∈[0,1), r2∈[0,1), is the individual's best historical position at the current moment, is the historical optimal position of the group at the current moment.

[0032] Optionally, the characteristic indicator function is:

[0033]

[0034] Among them, X fi is the original particle position vector, X′ fi is the converted feature indicator vector,

[0035] Optionally, preview and preprocess the dataset, including:

[0036] Get the size, mean, variance, and number of missing items of a data set based on functions in Python;

[0037] Based on the data processing tools in Python, outlier detection and processing, missing value processing and data normalization are performed on the data set according to the size, mean, variance and number of missing items of the data set.

[0038] Optionally, the Pearson correlation coefficient is used to analyze the correlation between the features in the preprocessed data set, including:

[0039] The linear correlation between two variables in a data set is calculated according to the following formula:

[0040]

[0041] Among them, ρ X,Y is the correlation coefficient between variables X and Y, σ X , σ Y are the standard deviations of variables X and Y, respectively, cov(X, Y) is the covariance between variables X and Y, n is the number of values ​​the variable can take, are the means of variables X and Y respectively.

[0042] (III) Beneficial effects

[0043] The beneficial effects of the present invention are as follows: a hybrid feature screening and hyperparameter optimization PSO-XGBoost system construction method of the present invention mixes two types of PSO algorithms: a continuous value version and a discrete value version. The continuous value version is used to optimize the optimal hyperparameter combination of XGBoost, and the discrete value version is used to search for the optimal feature subset. It can simultaneously solve the problems of data set dimensionality disaster and unclear model hyperparameters in the existing big data mining field. Compared with the related art, the present invention realizes the parallel processing of the two key problems of feature screening to achieve data dimensionality reduction and model hyperparameter optimization when the machine learning algorithm processes big data, which saves computing resources and time while overcoming the limitations that arise when the two are performed separately. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a process for constructing a PSO-XGBoost system for hybrid feature screening and hyperparameter optimization provided by an embodiment of the present invention;

[0045] Figure 2 The Pearson correlation coefficient diagram of the first ten features in the original WAT dataset provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below in conjunction with the accompanying drawings through specific embodiments. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a clearer and more thorough understanding of the present invention and to be able to fully convey the scope of the present invention to those skilled in the art.

[0047] Embodiment 1

[0048] Reference Figure 1, this embodiment provides a PSO-XGBoost system construction method for hybrid feature screening and hyperparameter optimization, including:

[0049] S101, preview and preprocess the data set.

[0050] Optionally, the data set is previewed and preprocessed, including: obtaining the size, mean, variance, and number of missing items of the data set based on the functions in Python. Based on the data processing tools in Python, outlier detection and processing, missing value processing, and data normalization are performed on the data set according to the size, mean, variance, and number of missing items of the data set.

[0051] Furthermore, outlier detection can select but is not limited to the following two detection methods: box plot and 3-Sigma principle. L -1.5IQR or greater than Q U Values ​​of +1.5IQR are considered outliers. L It is called the lower quartile, which means that one quarter of all sample data have values ​​smaller than it; Q U It is called the upper quartile, which means that one quarter of the data in the sample has a value greater than it; IQR is called the interquartile range, which is Q U and Q L The difference contains half of all sample data. The results of the box plot to identify outliers are more objective and have certain advantages in identification. The 3-Sigma principle is more suitable for data that obeys the normal distribution. Under the assumption of normal distribution, the probability of the value x that differs from the mean value μ by three times the standard deviation 3σ is P(|x-μ|>3σ)≤0.003, which is a very rare low-probability event. Therefore, if the deviation of the measured value from the mean value in a set of data exceeds 3 times the standard deviation, it will be regarded as an outlier. Set the detected outliers as null values ​​and process them together with the missing values ​​in the next step.

[0052] The processing of missing values ​​can be selected from but is not limited to the following methods: deleting missing values; filling missing values ​​with the mean, mode, or median; establishing a prediction model based on existing data and using the results to fill missing values.

[0053] Data normalization can be done by selecting but not limited to the following three methods: standard normalization, maximum and minimum value normalization, and robust normalization. Standard normalization is to process sample data into a standard normal distribution with a mean of 0 and a variance of 1. The formula for the standard normalization method is: In the formula, x′ is the normalized data, is the mean of all sample data, and S is the standard deviation of all sample data. Maximum and minimum normalization uses the two extreme values ​​of the sample data to scale all data and normalize all data to between 0 and 1. The formula is In the formula, x is the original sample data, Max and Min are the maximum and minimum values ​​of the sample data respectively. Robust normalization uses the concept of quartiles. By calculating the difference between the first quartile and the third quartile, the interquartile range (IQR) is obtained, and then normalized. The formula is: In the formula, median is the median of the original sample data, IQR 1,3 (X) represents the interquartile range. These three methods have their own advantages and disadvantages. The standard normalization method can only get the best results when the sample data is closer to the normal distribution, while the maximum and minimum normalization method is easily affected by outliers. Compared with the first two normalization methods, robust normalization is more effective in dealing with outliers. Since quartiles and interquartile ranges are used when calculating robust normalization, outliers have little impact on the calculation results. In this method, because the data set has been detected and processed for outliers before the normalization operation, the impact of outliers on the maximum and minimum normalization results is eliminated, the maximum and minimum normalization method can be used.

[0054] S102, using the Pearson correlation coefficient to analyze the correlation between the features in the preprocessed data set, deleting the features in the original data set whose linear correlation is greater than a preset threshold, and obtaining a feature subset.

[0055] Among them, the preset threshold is 0.7.

[0056] Optionally, the Pearson correlation coefficient is used to analyze the correlation between the features in the preprocessed data set, including:

[0057] The linear correlation between two variables in a data set is calculated according to the following formula:

[0058]

[0059] Among them, ρ X，Y is the correlation coefficient between variables X and Y, σ X , σ Y are the standard deviations of variables X and Y, respectively, cov(X, Y) is the covariance between variables X and Y, n is the number of values ​​the variable can take, are the means of variables X and Y respectively.

[0060] In Pearson correlation analysis, the sign of the correlation coefficient represents whether the two variables are positively correlated or negatively correlated. In addition, the absolute value of the correlation coefficient close to 0 indicates that the correlation between the two variables is weak, and close to 1 indicates that the correlation between the two variables is significant. The relationship between the Pearson coefficient and the correlation between variables is shown in Table 1. When the correlation coefficient between two variables is greater than 0.7, it can be considered that the two variables are highly linearly correlated, and only one of the variables needs to be retained as the subsequent model input.

[0061] Table 1

[0062] Pearson correlation coefficient absolute value Variable correlation <![CDATA[|ρ X,Y |<0.3]]> No linearity <![CDATA[0.3≤|ρ X,Y |<0.5]]> Weak linear correlation <![CDATA[0.5≤|ρ X,Y |<0.7]]> Moderate linear correlation <![CDATA[|ρ X,Y |≥0.7]]> Highly linear correlation

[0063] S103, dividing the feature subset into a training set and a test set.

[0064] The feature subset without redundant features is divided into a training set and a test set, where the training set accounts for 80% and the test set accounts for 20%, and it is ensured that there are no duplicates between the two sub-datasets.

[0065] S104, performing feature selection on the training set and hybrid optimization of the hyperparameters of the XGBoost model based on the PSO algorithm, and outputting the optimal feature subset and the optimal hyperparameter combination of the XGBoost model.

[0066] Optionally, feature selection of the training set and hybrid optimization of the hyperparameters of the XGBoost model are performed based on the PSO algorithm, and the optimal feature subset and the optimal hyperparameter combination of the XGBoost model are output, including:

[0067] Initialize the particle swarm parameters in the PSO algorithm, where the particle swarm parameters include: particle swarm size, particle dimension (i.e., the number of parameters to be optimized), maximum number of iterations, inertia weight, learning factor, population range, and iteration step range.

[0068] Randomly initialize the position and velocity of each particle in the particle swarm.

[0069] Determine the individual's historical optimal position and individual's historical optimal fitness, the group's historical optimal position and group's historical optimal fitness.

[0070] Among them, the individual historical optimal position is the optimal position obtained by the individual during the search process. The individual historical optimal fitness is the best fitness value obtained by the particle during the search process. Traversing the entire particle group, the position of the particle with the best fitness value is the group historical optimal position. The fitness value corresponding to the group historical optimal position can be directly selected as the group historical optimal fitness.

[0071] During the iteration, the position and velocity of each particle are updated and the fitness of each particle is calculated.

[0072] During the iteration process, if the fitness of the particle at the current moment is greater than the individual historical optimal fitness of the particle, the individual historical optimal fitness and individual historical optimal position of the particle are updated. If the maximum value of the individual historical optimal fitness of the current particle is greater than the group historical optimal fitness, the group historical optimal fitness and group historical optimal position are updated.

[0073] When the maximum number of iterations is reached, the current group's historical optimal fitness and the current group's historical optimal position are output as the optimal solution.

[0074] The optimal solution is divided into two parts, the first part is converted into the optimal feature subset based on the feature indicator function, and the second part is used as the optimal hyperparameter combination of the XGBoost model.

[0075] During the iteration process, the position of each particle is represented as a vector of size 1×dim. The size of dimension dim is determined by the number of features to be selected in the dataset and the number of hyperparameters to be optimized in the XGBoost model. The dimension size of the particle position vector is equal to the sum of the two. When calculating the particle fitness, the position vector X of each particle is i Split into two partsX fi and X hi , Part I X fi After the feature indicator function is converted, the feature selection of the data set is realized. The second part X hi Directly used for hyperparameter optimization of XGBoost models.

[0076] Optionally, the fitness is calculated according to the following formula:

[0077]

[0078] Among them, ACC is the classification accuracy of the model. selected is the number of features selected in each calculation, S total is the total number of features in the data set, α∈(0,1).

[0079] Optionally, the particle position is updated according to the following formula:

[0080]

[0081] in, is the position of the particle at the next moment, is the particle's current position.

[0082] Optionally, the particle velocity is updated according to the following formula:

[0083]

[0084] in, is the velocity of the particle at the next moment, is the velocity of the particle at the current moment, ω is the inertia weight, c1 is the individual learning factor, c2 is the group learning factor, r1 and r2 are random numbers, r1∈[0,1), r2∈[0,1), is the individual's best historical position at the current moment, is the historical optimal position of the group at the current moment.

[0085] Optionally, the characteristic indicator function is:

[0086]

[0087] Among them, X fi is the original particle position vector, X′ fi is the converted feature indicator vector,

[0088] If a component in the feature indicator vector is 1, it means that the corresponding feature is selected; if a component is 0, it means that the corresponding feature is not selected.

[0089] S105, constructing a PSO-XGBoost system based on the optimal feature subset and the optimal hyperparameter combination of the XGBoost model.

[0090] S106, testing the PSO-XGBoost system based on the test set, outputting the prediction results, and completing the construction of the PSO-XGBoost system when the prediction results meet the preset conditions.

[0091] The system construction is completed when the accuracy of the prediction results is significantly improved, otherwise the steps of this application are re-executed if the preset conditions are not met.

[0092] The present embodiment provides a method for constructing a PSO-XGBoost system for hybrid feature screening and hyperparameter optimization, which mixes two types of PSO algorithms: a continuous value version and a discrete value version. The continuous value version is used to optimize the optimal hyperparameter combination of XGBoost, and the discrete value version is used to search for the optimal feature subset. It can simultaneously solve the problems of data set dimensionality disaster and unclear model hyperparameters in the existing big data mining field. Compared with related technologies, the present invention realizes the parallel processing of the two key problems of feature screening to achieve data dimensionality reduction and model hyperparameter optimization when the machine learning algorithm processes big data, which saves computing resources and time while overcoming the limitations that arise when the two are performed separately.

[0093] Embodiment 2

[0094] This embodiment provides a PSO-XGBoost system construction method for hybrid feature screening and hyperparameter optimization.

[0095] First, the dataset is previewed and preprocessed.

[0096] In this example, we use a set of real WAT (Wafer Acceptance Test) data as sample data. The sample features include the drain-source current I of the chip at each test point on the wafer. DS , substrate current I sub , threshold voltage V T , contact resistance R e The entire dataset contains 99 features in total. The sample label is the CP (circuit yield) test yield of the wafer, and the entire dataset contains more than 800 samples.

[0097] By using the function in Python to preview the WAT dataset, we can obtain the dataset size, mean, variance, number of missing items and other properties. The missing items can be classified as outliers and deleted. Then, the maximum and minimum value normalization method is used to scale all data to (0, 1). The formula is as follows:

[0098]

[0099] Then, the Pearson correlation coefficient was used to perform correlation analysis on the preprocessed data set.

[0100] Use Python to compile the Pearson correlation coefficient function to perform correlation analysis on the preprocessed data set. The Pearson correlation coefficient can be used to detect the linear correlation between two variables. Usually, we regard two variables with a correlation coefficient greater than or equal to 0.7 as redundant variables, and only one of them can be retained.

[0101] In this example, the original WAT dataset contains 99 features, and there is some redundancy between these features. Since there are too many features in the original dataset, it is impossible to visualize all of them. The correlation coefficients between the first 10 features in the original dataset are selected and represented in Figure 2 In. From Figure 2 As can be seen from the figure, the correlation coefficients between some data are very strong, even exceeding 0.9, and these features can be regarded as redundant features. Therefore, we choose to randomly delete one of the redundant feature pairs, and finally reduce the dimension of the original WAT dataset to 62.

[0102] Furthermore, the feature subsets are divided into training sets and test sets, where the training set accounts for 80% and the test set accounts for 20%, and there are no duplicates between the two sub-datasets.

[0103] Furthermore, the training set is used as the input of the pre-training process of the PSO-XGBoost intelligent system. The PSO algorithm is combined with the XGBoost model to realize the hybrid optimization of WAT dataset feature selection and XGBoost hyperparameters, and finally the optimal WAT feature subset and the optimal XGBoost hyperparameter configuration are output simultaneously.

[0104] Extreme Gradient Boosting (XGBoost) is developed based on gradient boosting, and has been enhanced in terms of loss function and loss optimization process. Suppose we have a database consisting of n samples:

[0105] D={(x1, y1), (x2, y2),..., (x n ,y n )}

[0106] Where i = 1, 2, ..., n, x i is the input variable, y i is the output variable. In XGBoost, the loss function L is defined as the error between the predicted value and the true value, and its formula is as follows:

[0107]

[0108] Among them, l represents the actual training loss between the true value and the predicted value, Ω(f k ) is the added regularization term, which can be used to measure the complexity of the model. The formula is as follows:

[0109]

[0110] Where T represents the number of leaf nodes, ω k is the leaf score (or weight), γ and λ are penalty coefficients. The introduction of regularization term reduces the risk of model overfitting. XGBoost has some hyperparameters that can be defined when using it. In this example, four XGBoost hyperparameters to be optimized are selected, namely the number of weak learners n_eatimators, the maximum depth of the decision tree max_depth, the loss reduction threshold gamma caused by splitting when making decisions, and the learning rate learning_rate.

[0111] Furthermore, the XGBoost model and PSO swarm optimization algorithm were compiled based on Python. First, the particle swarm parameters of the PSO algorithm were initialized, including the particle swarm size sizepop=30, particle dimension (i.e. the number of parameters to be optimized) dim=66, maximum number of iterations maxgen=400, minimum weight ω min =0.4, maximum weight ω max =0.9, learning factor c1=c2=2 and iteration step range (-0.5, 0.5).

[0112] Then, the position and velocity of each particle in the population are randomly initialized, and the individual historical optimal position and fitness value, the group historical optimal position and fitness value are calculated; among them, the fitness function of the PSO algorithm is defined as:

[0113]

[0114] In the formula, ACC is the classification accuracy of the model, S selected is the number of features selected in each calculation, S total is the total number of features in the data set, and α is a constant that can take values ​​in the interval (0, 1). The larger the value, the more important the classification accuracy of the model is in this example. In this example, we take α = 0.8.

[0115] Start iteration. During the iteration, update the speed and position of each particle according to the following formula, calculate the fitness value of each particle, and if the fitness value of the particle at the current moment is greater than the individual historical optimal fitness value of the particle, update the individual historical optimal fitness value and position of the particle. If the maximum value of the current individual historical optimal fitness value of the particle is greater than the historical optimal fitness value of the group, update the historical optimal fitness value and position of the group. When the maximum number of iterations is reached, output the current historical optimal fitness value and position of the group, which is the optimal solution.

[0116]

[0117] In the formula, Represent the speed of the particle at the next moment and the current moment respectively, ω represents the inertia weight, c1 and c2 represent the individual learning factor and the group learning factor respectively, r1 and r2 are random numbers between [0, 1), represents the optimal historical position of the individual at the current moment, Indicates the historical optimal position of the population at the current moment.

[0118]

[0119] In the formula, They represent the position of the particle at the next moment and the current moment respectively.

[0120] In the PSO algorithm, the position of each particle is represented as a vector of size 1×dim. The size of dimension dim represents the number of features to be screened and the number of hyperparameters to be optimized. Updating the particle position vector in each iteration is the process of feature selection and hyperparameter optimization, and finally outputting the historical optimal position of the group: p d，gbest (t) = [p[0], p[1], ..., p

[63] , p

[64] , p

[65] ]. The optimal position vector of the particle is divided into two parts. The first part p fd，gbest (t) = [p[0], p[1], ..., p

[61] ] is converted into a discrete vector containing only 0 and 1 through the feature indicator function, where 0 represents that the corresponding feature is not selected and 1 represents that the corresponding feature is selected. The feature indicator function formula is as follows.

[0121]

[0122]

[0123] The second part of the optimal position vector p hd,gbest (t) = [p

[62] , p

[63] , p

[64] , p

[65] ] represents the four hyperparameters of the final optimized XGBoost, where p

[62] represents the maximum number of weak learners n_estimators in the XGBoost framework parameters, p

[63] represents the maximum depth max depth in the XGBoost parameters, p

[64] represents the loss reduction threshold gamma brought about by splitting during decision making, and p

[65] represents the learning rate learning_rate.

[0124] In this example, the two parts of the optimal position vector output by the PSO algorithm are P fd =[0,1,0,0,1,0,1,0,0,0,0,0,0,0,0,1,0,1,0,1,0,1,1,1,1,0,0,0,1,1,0,1,1,1,0,1,1,1,0,1,1,0,1,0,0,1,0,1,0,0,1,0,1,0,1,0,1,1,0,0,0,0,0,1,0]

[0125] P hd =[800,5,0.9,0.026]

[0126] That is, the PSO algorithm that mixes feature screening and hyperparameter optimization further screens the feature subset to 26, and the best hyperparameter combination of XGBoost found is n_estimators = 800, max_depth = 5, gamma = 0.9, and learning_rate = 0.026.

[0127] Finally, the obtained optimal feature subset and optimal XGBoost hyperparameter combination are used to build the final intelligent system, which is tested using the test set, and the wafer yield prediction results of the system on the test set are finally output.

[0128] The PSO-XGBoost intelligent system with hybrid feature screening and hyperparameter optimization proposed in this embodiment and the XGBoost without feature selection and hyperparameter optimization were used for the above real data set, and the results are shown in Table 2. As can be seen from Table 2, compared with the unoptimized XGBoost, the prediction accuracy of the PSO-XGBoost system proposed in this embodiment is improved by 3.5%, and the model generalization ability is improved by 3%, which proves the effectiveness of the present invention.

[0129] Table 2

[0130] PSO-XGBoost XGBoost Accuracy 0.802 0.767 Auc_score 0.681 0.651

[0131] Embodiment 3

[0132] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the PSO-XGBoost system construction method for hybrid feature screening and hyperparameter optimization in any one of the first aspects described above.

[0133] Embodiment 4

[0134] An embodiment of the present invention provides a storage device, including a storage medium and a processor, wherein the storage medium stores a computer program, and when the program is executed by the processor, the PSO-XGBoost system construction method for hybrid feature screening and hyperparameter optimization in any one of the first aspects described above is implemented.

[0135] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.

[0137] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may alter, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for constructing a PSO-XGBoost system with hybrid feature screening and hyperparameter optimization, characterized in that: include: Previewing and preprocessing a data set, wherein the data set is a set of real wafer acceptance test data, and the features of the data set include drain-source current, substrate current, threshold voltage and contact resistance of chips at each test point on the wafer; The Pearson correlation coefficient is used to analyze the correlation between the features in the preprocessed data set, and the features with linear correlation greater than the preset threshold in the original data set are deleted to obtain the feature subset; Divide the feature subsets into training and test sets; Based on the PSO algorithm, the training set is subjected to feature selection and the hyperparameters of the XGBoost model are mixed optimized to output the optimal feature subset and the optimal hyperparameter combination of the XGBoost model. In each iteration, the particle position vector is updated, which is the process of feature selection and hyperparameter optimization. Finally, the historical optimal position of the group is output: ; Divide the particle's optimal position vector into two parts. The first part The feature indicator function is converted into a discrete vector containing only 0 and 1, where 0 represents that the corresponding feature is not selected and 1 represents that the corresponding feature is selected; the feature indicator function formula is as follows: , ; The second part of the optimal position vector Represents the four hyperparameters of the final optimized XGBoost, where p[62] represents the maximum number of weak learners n_estimators in the XGBoost framework parameters, p[63] represents the maximum depth max_depth in the XGBoost parameters, p[64] represents the loss reduction threshold gamma brought by splitting during decision making, and p[65] represents the learning rate learning_rate; Construct a PSO-XGBoost system based on the optimal feature subset and the optimal hyperparameter combination of the XGBoost model; The PSO-XGBoost system is tested based on the test set and the prediction results are output. When the prediction results meet the preset conditions, the construction of the PSO-XGBoost system is completed.

2. The method for constructing a PSO-XGBoost system for hybrid feature screening and hyperparameter optimization according to claim 1, characterized in that: The method of performing feature selection on the training set and hybrid optimization of the hyperparameters of the XGBoost model based on the PSO algorithm, and outputting the optimal feature subset and the optimal hyperparameter combination of the XGBoost model, includes: Initialize the particle swarm parameters in the PSO algorithm, wherein the particle swarm parameters include: particle swarm size, particle dimension, maximum number of iterations, inertia weight, learning factor, population range and iteration step range; Randomly initialize the position and velocity of each particle in the particle swarm; Determine the individual's best historical position and the individual's best historical fitness, the group's best historical position and the group's best historical fitness; During the iteration, the position and velocity of each particle are updated and the fitness of each particle is calculated; During the iteration process, when the fitness of the particle at the current moment is greater than the individual historical optimal fitness of the particle, the individual historical optimal fitness and individual historical optimal position of the particle are updated; when the maximum value of the individual historical optimal fitness of the current particle is greater than the group historical optimal fitness, the group historical optimal fitness and group historical optimal position are updated; When the maximum number of iterations is reached, the current group's historical optimal fitness and the current group's historical optimal position are output as the optimal solution; The optimal solution is divided into two parts. The first part is converted into the optimal feature subset based on the feature index function, and the second part is used as the optimal hyperparameter combination of the XGBoost model.

3. The method for constructing a PSO-XGBoost system for hybrid feature screening and hyperparameter optimization according to claim 2, characterized in that: The fitness is calculated according to the following formula: in, is the classification accuracy of the model; is the number of features selected for each calculation, is the total number of features in the dataset, ∈ (0, 1).

4. The method for constructing a PSO-XGBoost system for hybrid feature screening and hyperparameter optimization according to claim 3, characterized in that: Update the particle position according to the following formula: in, is the position of the particle at the next moment, is the particle's current position, Indicates the velocity of the particle at the next moment.

5. The method for constructing a PSO-XGBoost system for hybrid feature screening and hyperparameter optimization according to claim 4, characterized in that: The particle velocity is updated according to the following formula: in, is the velocity of the particle at the next moment, is the particle's current velocity, is the inertia weight, is the individual learning factor, Divided into group learning factors, and is a random number, ∈[0,1), ∈[0,1), is the individual's best historical position at the current moment, is the historical optimal position of the group at the current moment.

6. The method for constructing a PSO-XGBoost system for hybrid feature screening and hyperparameter optimization according to claim 1, characterized in that: The previewing and preprocessing of the data set includes: Get the size, mean, variance, and number of missing items of a data set based on functions in Python; Based on the data processing tools in Python, outlier detection and processing, missing value processing and data normalization are performed on the data set according to the size, mean, variance and number of missing items of the data set.

7. The method for constructing a PSO-XGBoost system for hybrid feature screening and hyperparameter optimization according to claim 1, characterized in that: The Pearson correlation coefficient is used to analyze the correlation between features in the preprocessed data set, including: The linear correlation between two variables in a data set is calculated according to the following formula: in, For variables , The correlation coefficient between , Variables , The standard deviation of For variables , The covariance between , n is the number of values ​​the variable takes, Variables The mean of .

Images