An improved CNN architecture optimization design method

Abstract

A method for optimizing and designing a convolutional neural network architecture based on an improved CNN architecture: the convolutional neural network framework design problem is modeled as a constrained combinatorial optimization problem, and a network framework that meets the target parameter quantity requirements is designed. A variable-length coding strategy is proposed to encode hyperparameters such as the number of convolutional layers and the number of output channels of the convolutional layers to represent the convolutional neural network framework, and then an improved particle swarm algorithm is used to search for the optimal network framework using the classification accuracy on the CIFAR10 and CIFAR100 test data sets as the fitness evaluation value to obtain the optimal convolutional neural network framework. The method for automatically designing a convolutional neural network framework based on an improved particle swarm algorithm proposed in the present invention has the characteristics of being fully automatic and highly efficient, and is also very friendly to inexperienced scientific research beginners.

Description

Technical Field

[0001] The present invention relates to the fields of convolutional neural network architecture optimization design, agent-assisted evolutionary algorithm, etc. in deep learning. Background Art

[0002] 1. Convolutional Neural Network Framework

[0003] Convolutional neural network (CNN) is a type of feedforward neural network with deep structure and convolution calculation. It is one of the representative algorithms of deep learning. Unlike fully connected neural network, it uses a feature extractor composed of convolution layer and pooling layer to extract data features. In the convolution layer of CNN, a neuron is only connected to some of its adjacent neurons. Each convolution layer usually contains multiple channels. The weights updated by neurons in each channel are shared, that is, the convolution kernel parameters are shared. Pooling is usually divided into two forms: average pooling and maximum pooling. It is a special convolution process that can be used for data dimensionality reduction. Convolution and subsampling greatly simplify the complexity of the model and reduce the parameters of the model. Convolutional neural network generally includes feature extraction and fully connected neural network. Feature extraction includes convolution and pooling (the two parts can be repeated). After convolution and pooling, the data achieves the purpose of feature extraction and data dimensionality reduction. The extracted features are flattened and input into the fully connected neural network to obtain the predicted label through learning and training model. The general schematic diagram of convolutional neural network is shown as follows Figure 1 shown.

[0004] 2. Convolutional neural network technology based on residual blocks

[0005] The residual network (ResNet) is an advanced CNN network proposed in recent years. The success of ResNet is largely attributed to its building blocks. The skip connection is applied to the residual block (RB) inside it, which alleviates the gradient vanishing problem of deep neural networks caused by increasing depth, making it easy to optimize and the accuracy is still good when the model depth is high.

[0006] RBs can alleviate the adverse effects of the gradient vanishing problem caused by increasing network depth. On this basis, the deep architecture can effectively learn the hierarchical representation of input data, thereby improving the final classification accuracy. The algorithm proposed in the present invention selects RBs as a building block.

[0007] 3. Evolutionary Algorithms

[0008] A classic evolutionary computing method mainly includes the following three steps: initialization, update, and termination. In the initialization stage, some individuals are randomly generated to form an initial population, and the fitness of these individuals is evaluated. In each iteration, the algorithm searches for new individuals based on a search strategy, then evaluates the fitness of the new individuals, and finally selects a group of individuals from the new individuals and the original individuals according to the individual fitness to form a new population to update the population. In the termination stage, the algorithm stops iterative updating of the population after reaching the termination condition, and outputs the individual with the best fitness in the current population. This individual will be used as the final solution to the problem being solved. The schematic diagram of evolutionary computing solving optimization problems is shown in Figure 2 As shown in the figure, it can be seen that when using evolutionary computing to solve optimization problems, the first thing to do is to design an encoding method to map the solution of the problem into individuals in the evolutionary computing population. On the other hand, the individual fitness evaluation method is designed based on the principle that the higher the fitness of the individual, the closer it is to the optimization goal of the problem. Figure 2 It can also be seen that the main calculation of evolutionary computing comes from the iterative update of the population, and in the iterative update of the population, a large number of individual fitness evaluations are required. Therefore, the computational efficiency of the fitness evaluation method determines the efficiency of the evolutionary computing method.

[0009] 4. Traditional convolutional neural network architecture design method based on evolutionary computing

[0010] In recent years, some studies have applied evolutionary computing to convolutional neural network architecture design, effectively improving the performance of convolutional neural networks.

[0011] In the traditional convolutional neural network architecture design method based on evolutionary computation, each individual corresponds to a network architecture, and the fitness evaluation of the individual is performed in the following way: build a corresponding convolutional neural network according to the architecture corresponding to the individual, train the network based on the training set, and test the performance of the trained network based on the test set. Therefore, in the traditional convolutional neural network architecture design method based on evolutionary computation, the fitness evaluation of each individual requires a complete convolutional neural network training. We know that neural network training takes a lot of time, so in the traditional convolutional neural network design method based on evolutionary computation, individual fitness evaluation is very time-consuming.

[0012] 5. Proxy Model

[0013] At present, the commonly used proxy models include Kriging model, artificial neural network (ANN), radial basis function (RBF), support vector regression model (SVM) and polynomial regression model (PRG).

[0014] The surrogate model is an approximation of the exact evaluation function. It takes individuals as input and outputs predicted fitness. By finding a mapping between individuals and their exact fitness values, the predicted fitness is approximated to the exact fitness. For each individual x i It can be evaluated by an expensive function y = f(x i ) to get its exact fitness value y i If we use some input x i and output y i To build a dataset for training the proxy model. The proxy model is an approximate function To evaluate the input x i .

[0015] In order to solve the difficulty of black box optimization problems, relevant scholars have proposed an optimization algorithm based on surrogate models (Surrogate-based optimization, SBAO). In SBAO, the surrogate model mainly reflects the characteristics of the true response value function in the design space by fitting a smooth true response function or constraint function, thereby accelerating the optimization process. Summary of the invention

[0016] The technical problem solved by the present invention is to propose a convolutional neural network framework automatic design method that is simple and easy to use, very friendly to inexperienced junior scientific researchers, and the obtained network architecture performance can meet the performance requirements, in view of the problems that the existing convolutional neural network framework design completely relies on the traditional manual trial and error method of experienced scientific research experts, and the design cost is high and the efficiency is low.

[0017] The technical solution adopted by the present invention is: an improved CNN architecture optimization design method, comprising the following steps:

[0018] 1. The automatic design method of convolutional neural network framework based on residual connection is modeled as a constrained combinatorial optimization problem, with the constraints being the number of residual blocks and the input and output dimensions of each convolutional layer;

[0019] 2. Design a new variable-length encoding strategy (Resnet Block Encoded, RBE) to map particles to the corresponding network architecture and obtain the mathematical representation of the convolutional neural network framework;

[0020] 3. The improved PSO algorithm is used to design the convolutional neural network framework, with the image classification accuracy on the CIFAR10 and CIFAR100 test data sets as the fitness evaluation value, and the continuous improvement of classification accuracy as the optimization goal. The improved PSO algorithm is continuously iterated and optimized to obtain the convolutional neural network framework with the best performance;

[0021] 4. Fully train the optimal convolutional neural network framework again; BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a diagram of a convolutional neural network.

[0023] Figure 2 This is a schematic diagram of how evolutionary computing solves optimization problems.

[0024] Figure 3 It is a flow chart of the particle swarm algorithm in the present invention.

[0025] Figure 4 It is a schematic diagram of the variable length encoding strategy.

[0026] Figure 5 It is a flow chart of automatic design of convolutional neural network framework based on improved particle swarm algorithm of the present invention. DETAILED DESCRIPTION

[0027] An improved CNN architecture optimization design method of the present invention is described in detail below with reference to examples and drawings.

[0028] The flowchart of the particle swarm algorithm is as follows: Figure 3 shown.

[0029] A typical PSO algorithm steps are as follows:

[0030] Step 1: Initialize particles, predefine the maximum number of generation T, and initialize the counter t=0.

[0031] Step 2: Evaluate the fitness of the particle.

[0032] Step 3: For each particle i, select the best pBest from its memory i .

[0033] Step 4: Select the best particle gBest from the history of all particles.

[0034] Step 5: Calculate the velocity {v1, v2, ...} from the position {x1, x2, ...} of each particle according to equation (1).

[0035] Step 6: Update the position of each particle according to equation (2).

[0036] Step 7: t=t+1, if t<T, repeat steps 2-6, otherwise execute step 8.

[0037] Step 8: Get gBest's location

[0038] v i ←ω·v i +c1·r1·(p g -pi )+c2·r2·(p p -p i ) (1)

[0039] p i ←p i +v i (2)

[0040] In (1), ω·v i represents inertia update, w is the inertia weight; c1·r1·(p g -p i ) represents global update, c2·r2·(p p -p i ) represents local update. c1 and c2 are acceleration constants, r1 and r2 are random numbers between 0 and 1, and p g 、p p are the positions of gBest and pBest respectively. i and p i Represents the i-th particle x i By integrating the three terms of "inertia", "global search" and "local search" into the velocity update, the particle is expected to find the best position.

[0041] In the particle swarm optimization algorithm, individuals in the population are called particles. In each generation, each particle records the optimal solution pBest in its own memory (using pBest i Represents the i-th particle), and the population records the best solution (represented by gBest) in the historical records of all particles. In this process, particles are expected to cooperate and interact with pBest and gBest to enhance the search ability and seek the optimal solution. PSO has been widely used in various practical applications, including the architecture design of neural networks, because it can overcome the non-convexity and non-differentiability of optimization problems. In the optimization of neural network structures, these algorithms use implicit methods to encode each connection of the neural network and use PSO or its variants to search for the optimal value.

[0042] Coding enables the particle swarm algorithm to model real-world problems, and encoding is achieved through corresponding encoding strategies, which is the first step of the evolutionary algorithm. When designing the CNN architecture using the EA-based method, before the algorithm starts, we need to encode the CNN architecture into an individual in the population, and after the algorithm ends, decode the optimal individual into the corresponding CNN architecture. It should be noted that although RB blocks are used in the proposed algorithm, users do not need to have professional knowledge of such blocks when using the proposed algorithm.

[0043] In the algorithm of the present invention, based on the residual block BottleNeck in ResNet, a fully convolutional neural network is constructed based on RBs blocks, without considering the pooling layer and the fully connected layer. The main reason is that the pooling layer needs to introduce additional parameters, and the fully connected characteristics of the fully connected layer are prone to overfitting. In order to prevent the fully connected layer from overfitting, other techniques such as dropout need to be adopted. However, these techniques will also generate additional parameters, which will increase the computational complexity of the algorithm. We use a convolution layer with a convolution kernel of 1*1 instead of a fully connected layer to output the classification results.

[0044] We proposed a new variable-length coding strategy that can represent the CNN architecture as particles of different lengths during the evolution process to search for the optimal CNN architecture. For the RB blocks used, according to the design of the classic ResNet, we set the filter size of the convolutional layers conv1 and conv2 to 3×3. The first RB starting from the second RBs block uses a convolutional layer with a stride of 2×2 to replace the pooling layer, so that the input dimension is halved after one RBs block. The unknown parameters of each RBs are set to the input space size and the number of RB blocks k. The size of the output space is equal to the size of the input space × expansion (expansion is a hyperparameter set for us). In short, the encoding of the CNN architecture is the number of RBs, and the encoding information of each RBs example is the number of RB blocks and the input space size, such as Figure 4 shown.

[0045] The complete steps of the CNN architecture design based on SO-PSO in the present invention are as follows: Figure 5 shown.

[0046] First, the particles randomly initialize the population according to the proposed encoding strategy and find their exact fitness values, then combine them into the dataset to train the proxy model, and then the particles begin to evolve. After a given number of iterations, the best individuals are selected from the best individuals according to the fitness of the final population to decode the best CNN architecture.

[0047] Step 1: Initialize the population and perform accurate fitness evaluation on each particle

[0048] Population initialization provides a basic population for the subsequent evolution process, and each individual in the population should represent a CNN architecture. The fitness of particles provides a quantitative measure of how they adapt to the environment. According to the principle of evolutionary algorithms, a particle with a higher fitness is more adaptable to the environment and the corresponding CNN architecture is better. In order to evaluate fitness, each individual in SR-PSO is decoded into a corresponding CNN and then added to a classifier for training.

[0049] Step 2: Update v and x of each particle according to (1) and (2), use the proxy model to perform an approximate fitness evaluation on each particle to select potential particles, and then perform an accurate fitness evaluation on the particles selected by the proxy model to update gBest and pBest.

[0050] Step 3: Add particles with accurate fitness evaluation to the dataset to update the proxy model

[0051] Step 4: Determine whether the termination condition is met. If so, proceed to the next step; if not, go to step 2.

[0052] Step 5: After the PSOA0 evolution is completed, the best particle gBest is selected for accurate fitness evaluation to obtain the final result.

[0053] The best parameter settings are given below:

[0054] 1. The best CNN architecture finally found is:

[0055]

[0056] 2. Parameter settings

[0057] In SR-PSO, the population size is set to 12 and the number of evolution generations is set to 20. Ideally, a larger population size and a larger number of iterations should lead to better performance, but it is also expected to consume more computing resources. As we described before, particle fitness evaluation is expensive because it requires training and testing CNNs. In order to reduce the computing time, we made some simplifications. In each fitness calculation, CNN was only trained for 60 epochs and only 30% of the random samples of the training and testing datasets were used. The fitness obtained was used as an approximation of the exact fitness evaluation during the particle evolution process. After the SR-PSO was finished, the performance of the obtained optimal CNN architecture was tested. Without any data augmentation or preprocessing, we used all the training datasets with more epochs (epochs=200) and the prediction accuracy of the test dataset as the exact fitness value of the particle. In particular, if the parameters of an individual exceeded the memory during the training process, the fitness of the individual was set to 0. In the training process of each particle, the ratio of training set to validation set is 4:1, the SGD optimizer is used, the batch size is 128, the activation function is ReLu, the learning rate is initialized to 0.1, and the cosine annealing is used for learning rate decay. Its parameters are set to the number of training epochs 200, the weight decay rate is set to 5×〖10〗^(-4), and the momentum is 0.9.

[0058] In addition, the maximum number of RBs in CNN is set to 4, the number of RBs in RBs is set to 3 to 6, and the input dimension of each RB block is in the range of [32, 256]. These settings are mainly based on our available computing resources, because any number beyond these settings will easily exceed the GPU memory. If the user's computing platform is equipped with a more powerful GPU, they can set the number to any one.

Claims

1. An improved CNN architecture optimization design method, comprising the following steps: 1) The automatic design method of convolutional neural network framework based on residual connection is modeled as a constrained combinatorial optimization problem, with the constraints being the number of residual blocks and the input and output dimensions in each residual block; 2) Design a new variable-length encoding strategy (Resnet Block Encoded, RBE) to map particles to the corresponding network architecture and obtain the mathematical representation of the convolutional neural network framework; including: (21) Define a parameter as the number of residual block sets; (22) For the i-th residual block set (RBs), it includes k i Residual blocks (RB), the network input dimension in each residual block (RB) is i , the output dimension is out n ; (23) stacking the residual blocks, that is, stacking and connecting multiple residual blocks; (24) The first residual block at the beginning of the second residual block set uses a convolutional layer with a step size of 2×2 to replace the pooling layer, so that the input dimension is halved after passing through a residual block, and a convolutional layer with a convolution kernel of 1*1 is used to replace the fully connected layer to output the classification result, forming a convolutional neural network framework. 3) The improved PSO algorithm is used to design the convolutional neural network framework, with the image classification accuracy on the CIFAR10 and CIFAR100 test data sets as the fitness evaluation value, and the continuous improvement of classification accuracy as the optimization goal. The improved PSO algorithm is continuously iterated and optimized to obtain the convolutional neural network framework with the best performance; including: (31) Set the initial parameters of the improved particle swarm algorithm, including: the number of populations N, the range of the number of residual block sets [Ncmin, Ncmax], the range of the number of RBs for each RBs [Rmin, Rmax], and the range of input dimensions [Inmin, Inmax]; (32) The population initialization parameters are used to initialize the population and encode according to the variable length coding strategy (Resnet Block Encoded, RBE). After a particle is generated, it is first determined whether the model parameter quantity of the convolutional neural network corresponding to the particle meets the constraint range. If so, the particle is valid. Otherwise, a new particle is generated again until N particles that meet the model parameter quantity constraint range are generated to form a particle population. (33) Evaluate the fitness of the generated population; (a) The classic CIFAR10 and CIFAR100 datasets are divided into a training set and a test set. The number of images in the training set is set to be , and the number of images in the test set is set to be . In order to reduce the training time, when performing fitness evaluation, only randomly select images from the entire dataset for training and testing, i.e., , training images and , test images; (b) Set the number of particles in the population n, reverse decode the particle according to the encoding method, and obtain the corresponding convolutional neural network framework; (c) Using the training images to train the decoded convolutional neural network framework, and then testing it on the test images to obtain the test accuracy as the fitness evaluation value; (d) The number of particles in the population n is increased by 1. If the number of particles in the population is less than the total number of particles in the population N, go to step (b). Otherwise, output the fitness values ​​of all particles in the population. (34) Sort the entire particle population in descending order according to the fitness value of the population, and select the global optimal particle in the entire population; (35) Update the particle's speed and position Update speed: Calculate the new speed by weighted summation based on the particle's current speed, position, and the particle's best historical position and the group's best historical position. The particle's best historical position is used to guide the particle to move in the direction of the particle's optimal solution, while the group's best historical position is used to guide the particle to move in the direction of the global optimal solution. Update position: Update the particle's position according to the newly calculated speed, and increase the number of iterations iter by one; (36) If the number of iterations is less than the maximum number of iterations of the algorithm, go to step (33), otherwise, output the global optimal particle and obtain the optimal convolutional neural network framework; In the improved PSO algorithm, the proxy model is used to perform approximate fitness evaluation on each particle to select potential particles, and then the precise fitness evaluation of the particles selected by the proxy model is performed to update the historical best position of the particles and the historical best position of the group. The particles with precise fitness evaluation are added to the data set to update the proxy model. 4) Fully train the optimal convolutional neural network framework again.

2. According to the improved CNN architecture optimization design method according to claim 1, step 1) comprises: The automatic design method of convolutional neural network framework is modeled as a combinatorial optimization problem under the constraint of model parameter quantity.

3. According to the improved CNN architecture optimization design method according to claim 1, step 4) comprises: (1) Set the number of iterations required for a full training process as iiter, and use all the data images in the CIFAO10 and CIFAR100 datasets, i.e., training images and test images; (2) Train the optimal convolutional neural network framework on the training data, test it on the test data, and obtain the final accuracy as the actual application accuracy.

Images