Hyper-parameter optimization method based on meta-learning and improved genetic algorithm

Abstract

The invention provides a hyper-parameter optimization method based on meta-learning and an improved genetic algorithm for improving the precision of a neural network on the basis of determining a data set and not modifying the network in a real scene, and the method comprises the steps: extracting the sample features of each sample of the data set through a data sample feature extractor; using a K-means clustering algorithm to extract a clustering center as a feature centroid to obtain data set element features, calculating element feature cosine distances in pairs, and obtaining a data set element feature similarity matrix; calculating an optimal hyper-parameter combination by using an optimization genetic algorithm added with reverse learning and an adaptive crossover mutation rate; the execution time of the algorithm is reduced through an automatic early stop strategy. According to the method, the optimal hyper-parameter of the neural network can be searched under the condition of less time overhead, and the precision of the neural network is improved.

Description

Technical Field

[0001] The present invention relates to the field of neural network hyperparameter optimization, and in particular to a hyperparameter optimization method based on meta-learning and improved genetic algorithm. Background Art

[0002] With the development of deep learning, it has been widely used in many fields such as image recognition, target detection, and image re-identification. The success of deep learning models depends largely on the performance of deep learning models, and the model performance is significantly affected by the hyperparameter settings. When using neural networks, researchers often ignore the optimization of the hyperparameters of the network itself, but the performance of the algorithm is also affected by the hyperparameters. Therefore, it is of great significance to study an excellent hyperparameter optimization algorithm to optimize the hyperparameters of the detection network to improve the importance of baggage detection. Hyperparameters are different from model parameters, which must be set before training the model, such as the depth of the network architecture, learning rate, batch size, and regularization coefficient. Correct hyperparameter settings can significantly improve the learning ability and generalization performance of the model. However, the hyperparameter space is usually high-dimensional and non-convex, and finding the optimal hyperparameter combination has become a challenging problem, which has prompted the development of hyperparameter optimization technology. However, the hyperparameter space is usually high-dimensional and complex, and it often takes a lot of time to optimize the hyperparameters and it is difficult to achieve good results. Therefore, how to design an algorithm that can ensure a small time cost and a high optimization effect has become a research-worthy problem.

[0003] The current hyperparameter optimization algorithms can be divided into four categories. The first category is the brute force search type. The core idea of ​​this type of algorithm is to adopt the principle of trial and error and regard the hyperparameter optimization problem as an exhaustive process. The algorithm continuously generates new hyperparameter combinations and selects the one with the best performance. This type of algorithm relies on the selected candidate solution and has poor stability. At the same time, since the brute force search method has a very expensive time overhead, this type of algorithm is rarely used now. The second category is the algorithm based on Baye optimization, which is based on the global optimization strategy of Bayesian theory and is mainly used to optimize the objective function with high computational cost. Many scholars have improved it. For example, Johannes Kirschner et al. proposed an algorithm (LineBO) to solve the problem that non-convex optimization problems are difficult to expand to high dimensions. Watanabe S et al. proposed constrained TPE (C-TPE) for restricted hyperparameter optimization. However, the Bayesian method has the problem that the accuracy of the proxy model in high-dimensional space is difficult to guarantee and the optimization of the acquisition function may become more difficult, resulting in performance degradation. The third category is the algorithm based on multi-fidelity optimization, which combines low-fidelity models and high-fidelity models. Although low-fidelity models are not accurate enough, they have low computational cost and can quickly provide a large amount of approximate information. Although high-fidelity models are accurate, they have high computational cost. By intelligently balancing the use of these two models, multi-fidelity optimization algorithms can achieve high-precision optimization results at a lower overall computational cost. However, it is necessary to carefully design or select low-fidelity and high-fidelity models according to different problems to ensure that they have good correlation. The fourth category is meta-heuristic algorithms. These algorithms are mainly optimization methods developed under the influence of biological theory, simulating the behavior of biological groups or group intelligence in nature to solve problems. For example, Zhong et al. proposed a beluga optimization algorithm (BWO) based on the behavior of beluga whales, and JIA et al. proposed an enhanced chimpanzee optimization algorithm (EChOA). These algorithms perform well in dealing with complex, high-dimensional and nonlinear problems because they can effectively explore the solution space and find effective solutions, but they are also sensitive to the initial solution and fall into the local optimum. Therefore, how to design an algorithm with low time cost and good algorithm stability is still a challenge. Summary of the invention

[0004] The purpose of the present invention is to propose a hyperparameter optimization method based on meta-learning and improved genetic algorithm to find the optimal hyperparameter combination of the network within a limited time overhead, thereby improving the accuracy of the network.

[0005] The object of the present invention is achieved in this way:

[0006] First, the sample feature extractor and meta-feature extractor are used to extract the meta-feature vector of the data set. Secondly, the meta-feature vector is used to compare the similarity between data sets, and the initial solution of the algorithm is selected based on the similarity. Finally, an optimized genetic algorithm with adaptive crossover mutation rate and reverse learning is used to iterate the hyperparameter combination, and an automatic early stopping strategy is used to reduce the algorithm time overhead.

[0007] It specifically includes the following steps:

[0008] Step 1: Use the data sample feature extractor to extract sample features of each sample in the data set.

[0009] Step 2: Use the dataset meta-feature extractor to extract the meta-feature vector representing the meta-feature of the dataset from the sample feature vector of the dataset, and select the initial solution of the hyperparameter according to the similarity of the meta-feature vector.

[0010] Step 3: Use the optimization genetic algorithm to find the optimal hyperparameter combination.

[0011] Step 4: Reduce the execution time of the algorithm through automatic early stopping strategy.

[0012] Furthermore, in step 1, the data owner uses a ResNet-152 as a data sample feature extractor to extract a vector of size 1×1×2048 from the sample of the input data set as the feature vector of the sample.

[0013] Furthermore, in step 2, after obtaining the result of step 1, the sample feature vector, the meta-feature vector that can represent the meta-feature of the data set is extracted in the following manner: after initializing the feature centroid, the meta-feature extractor extracts the cluster center as the feature centroid through the K-means clustering algorithm to obtain the meta-feature of the data set. Then, the cosine distance of the meta-features is calculated pairwise. The cosine distance of the meta-features of the data set can be expressed as in and are the meta-features of the two data sets respectively. Then we can get the meta-feature similarity matrix of the data sets.

[0014] Furthermore, in step 3, after obtaining the meta-feature similarity matrix of step 2, the first k most similar data sets with the highest similarity are selected, and their optimal hyperparameters are extracted as the initial solution to be input into the optimization genetic algorithm. The optimization genetic algorithm calculation method is: the first k selected initially by the hyperparameters are used as the initial population P of the optimization genetic algorithm. i =(P i,1 ,P i,2 ,P i,3 ,...,P i,m ). Among them, P i,jrepresents the chromosome of each individual. So its reverse solution can be expressed as in Represents the reverse chromosome of a chromosome in an individual. The calculation method is to add the maximum and minimum values ​​of this chromosome and then subtract the original value. Then use the adaptive crossover mutation rate to adaptively control the value of the crossover mutation rate to improve the efficiency of the algorithm. Adaptive mutation rate r = r min +(r initial -r min )*L, where r min is the minimum mutation rate, r initial is the initial mutation rate, where L is a value related to the maximum number of iterations and the current number of iterations. In the formula, iter cur Indicates the current number of iterations, iter max represents the maximum number of iterations, and k is a hyperparameter used to control the speed at which L decreases.

[0015] Furthermore, the automatic early stopping strategy in step 4 applies the hyperparameters to the given model for training and stops the training after reaching the early stopping judgment training times. Then, the model is tested on the test set and its performance index is obtained. Finally, the obtained index is compared with the historical hyperparameter performance. If the set conditions are met, it is considered that the current hyperparameter combination should be stopped directly. Otherwise, the training will continue until the maximum number of iterations is reached and the performance is counted.

[0016] The positive effects of the present invention are:

[0017] (1) The present invention proposes a hyperparameter optimization algorithm MLES-IGA based on meta-learning and improved genetic algorithm. The core idea of ​​MLES-IGA is to use the prior knowledge of the existing data set to extract meta-features through feature extraction and clustering, select the initial solution through the similarity of the meta-features, and iterate the genetic algorithm based on this. First, the present invention proposes to extract the meta-features of the data set through feature extraction and clustering. Secondly, the present invention proposes an improved genetic algorithm IGA, which enhances the global learning ability of the algorithm through a reverse learning mechanism to avoid falling into local optimality. IGA meets the need for a balance between global search and local search through an adaptive mutation crossover rate. Finally, the present invention also designs an automatic early stopping mechanism, which stops the training of poorly performing hyperparameter combinations at an early stage, saving time overhead on the basis of meeting model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of the hyperparameter optimization method based on meta-learning and improved genetic algorithm.

[0019] Figure 2 Flowchart of meta-feature similarity matrix generation.

[0020] Figure 3 It is the optimization genetic algorithm flow chart. DETAILED DESCRIPTION

[0021] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0022] Step 1 and step 2 are as follows Figure 1 Phase 1 and Figure 2 As shown, we extract the meta-feature matrix of the dataset from the data sample feature extractor and the dataset meta-feature extractor, and generate a similarity matrix to obtain the initial population input for the next stage.

[0023] Step 3, such as Figure 1 Phase II and Figure 3 As shown, the initial population is iteratively optimized based on the optimization genetic algorithm.

[0024] Step 4, the detailed structure of the automatic early stopping strategy is as follows Figure 1 as shown in stage three.

Claims

1. A hyperparameter optimization method based on meta-learning and improved genetic algorithm, characterized in that The steps include: Step 1: Use a data sample feature extractor to extract sample features of each sample in the data set; Step 2: Use a dataset meta-feature extractor to extract a meta-feature vector representing the meta-features of the dataset from the sample feature vectors of the dataset, and select an initial solution of the hyperparameter based on the similarity of the meta-feature vectors; Step 3: Use the optimization genetic algorithm to find the optimal hyperparameter combination; Step 4: Reduce the execution time of the algorithm through automatic early stopping strategy.

2. In step 1, the data owner uses a ResNet-152 as a data sample feature extractor to extract a vector of size 1×1×2048 from the sample of the input data set as the feature vector of the sample. In step 2, after obtaining the result of step 1, the sample feature vector, the meta-feature vector that can represent the meta-feature of the data set is extracted in the following manner: after initializing the feature centroid, the meta-feature extractor extracts the cluster center as the feature centroid through the K-means clustering algorithm to obtain the meta-feature of the data set, and then calculates the cosine distance of the meta-features pairwise. The cosine distance of the meta-features of the data set can be expressed as in and are the meta-features of the two data sets respectively, and then the meta-feature similarity matrix of the data sets can be obtained.

3. In step 3, after obtaining the meta-feature similarity matrix of step 2, the first k most similar data sets with the highest similarity are selected, and their optimal hyperparameters are extracted as the initial solution and input into the optimization genetic algorithm. The optimization genetic algorithm calculation method is: the first k selected initially by the hyperparameters are used as the initial population P of the optimization genetic algorithm. i =(P i,1 ,P i,2 ,P i,3 ,…,P i,m ),in, P i,j represents the chromosome of each individual, so its reverse solution can be expressed as in It represents the reverse chromosome of a chromosome in an individual. The calculation method is to add the maximum and minimum values ​​of this chromosome and then subtract the original value. Then, the adaptive crossover mutation rate is used to adaptively control the value of the crossover mutation rate to improve the efficiency of the algorithm. The adaptive mutation rate r = r min +(r initial -r min )*L, where r min is the minimum mutation rate, r initial is the initial mutation rate, where L is a value related to the maximum number of iterations and the current number of iterations, In the formula, iter cur Indicates the current number of iterations, iter max represents the maximum number of iterations, and k is a hyperparameter used to control the speed at which L decreases.

4. The automatic early stopping strategy of step 4 applies the hyperparameters to the given model for training and suspends the training after reaching the early stopping judgment training times. Then, the model is tested on the test set and its performance index is obtained. Finally, the obtained index is compared with the historical hyperparameter performance. If the set conditions are met, it is considered that the current hyperparameter combination should be stopped directly. Otherwise, the training is continued until the maximum number of iterations is reached and the performance is counted.

Images