A convolutional neural network search method for image classification

Abstract

The present invention discloses a convolutional neural network design method based on a genetic algorithm. Firstly, data preprocessing and sampling operations are performed on an image data set to form a proxy data set and a verification data set, and then node blocks required in a search process are designed; an individual population is randomly initialized through a specific encoding method, and a decoding operation corresponding to the encoding is used to decode the individual into a candidate convolutional neural network; then, a proxy model is introduced to predict whether a new individual is better than the worst individual so far; the candidate convolutional neural network is trained using the proxy data set, and the performance of the candidate convolutional neural network is evaluated using the verification data set; the size and performance of the candidate convolutional neural network model are weightedly summed to obtain a fitness value, individuals with good performance are selected according to the fitness value, and then mutation operations are performed on the individuals to form the next generation population until the iteration is completed and the optimal individual is determined.

Description

Technical Field

[0001] The present invention belongs to the technical field of neural network architecture search, and specifically relates to a convolutional neural network search method for image classification. Background Art

[0002] The concept of convolutional neural network (CNN) search has long existed. In 1989, Miller et al. proposed a neuroevolution method for evolving the weights of artificial neural networks, which has had a profound impact on today's CNN search methods and has attracted widespread attention from academia and industry. Neural network architecture search samples a CNN structure in the search space through a search strategy, then uses a specific performance evaluation strategy to evaluate the model performance, and then feeds the evaluated value back to the search strategy.

[0003] The development of neural network architecture search has become a key method to solve important problems in the field of computer vision. Reference 1 (Stanley KO, Miikkulainen R. Evolving Neural Networks through Augmenting Topologies [J]. Evolutionary Computation, 2002, 10 (2): 99-127) proposed a genetic topological neural network (NEAT algorithm) by combining genetic algorithms and neural networks. However, the algorithm only uses one unit to represent the minimum structural information and cannot effectively evolve neural networks with a large number of parameters. Later, researchers improved the NEAT algorithm and proposed a more efficient HyperNEAT algorithm, which pioneered the automated design of neural networks. Reference 2 (Baker B, Gupta O, Naik N, et al. Designing neural network architectures using reinforcement learning [J]. 2016. https: / / arxiv.org / pdf / 1611.02167v3.pdf.) uses Q-learn and ε-greedy exploration strategies to search for neural network structures, and uses 10 GPUs for 10 days to search for different structures on CIFAR10, CIFAR100 and other datasets. Reference 3 (Wang C, Xu C, Yao X, et al. Evolutionary genetic adversarial networks [J]. IEEE Transactions on Evolutionary Computation, 2019, 23 (6): 921-934.) In the EGAN paper, a genetic algorithm is used to initialize a population of generators and discriminators for adversarial training, which greatly improves the stability of GAN throughout the training process.

[0004] At present, there are many kinds of neural network structure design methods, which can be roughly divided into three categories: methods based on evolutionary computation, methods based on reinforcement learning, and differentiable architecture search. The existing research methods have the following problems. First, most of the existing neural network design methods use deep modules such as ResNet Block and DenseNet Block, and do not introduce wide modules similar to Inception, which limits the feature extraction ability of CNN; second, for specific tasks, the optimal CNN depth is unknown, and how to achieve variable length search and find the optimal depth is worth exploring; third, candidate network evaluation is an extremely time-consuming process, and existing network search methods use a lot of computing power for evaluation. Summary of the invention

[0005] To solve the aforementioned problems, the present invention provides a convolutional neural network search method for image classification. In image classification tasks, compared with traditional manually designed models and models obtained by other automatic search methods, the method of the present invention has improved accuracy and search efficiency.

[0006] The specific implementation scheme of the present invention to solve the technical problem is as follows:

[0007] A convolutional neural network search method for image classification includes the following steps:

[0008] S1: Divide the dataset into training dataset and test dataset, further randomly sample the training dataset to form proxy dataset and validation dataset, and then perform data augmentation and preprocessing on the proxy dataset;

[0009] S2: Design the node blocks that constitute the convolutional neural network, and use a decimal and binary fusion encoding method to achieve deep search of candidate convolutional neural networks;

[0010] S3: Design a proxy model to predict whether the candidate convolutional neural network represented by the new solution is better than the candidate convolutional neural network represented by the worst solution so far, and transform the problem of predicting the performance of the convolutional neural network into a binary classification problem;

[0011] S4: Randomly create candidate convolutional neural networks as the initial population, use the proxy model for the evaluation process, select individuals with higher fitness in the current population according to the fitness value, and perform mutation operations on the individuals to form the next generation population until the iteration is completed and the optimal individual is determined.

[0012] Beneficial effects of the present invention:

[0013] 1) In view of the shortcomings of existing neural network architecture search methods such as low efficiency and high computing power consumption during the evaluation process, this paper proposes an efficient proxy model for predicting whether the CNN represented by the new solution is better than the CNN represented by the worst solution to date. The data obtained during the evaluation process is used to train the proxy model, converting the problem of predicting CNN performance into a binary classification task of comparing a pair of CNNs.

[0014] 2) In view of the fact that existing neural network architecture search methods are prone to fall into the trap of small models during the search process, this paper designs a fitness evaluation mechanism to measure the quality of candidate convolutional neural networks, retaining only individuals with good performance and conducting next-generation training. At the same time, a method for forming a proxy dataset is proposed to further accelerate the search process.

[0015] 3) The search space of the existing neural network architecture search method only uses deep modules such as ResNet Block and DenseNetBlock. In addition to deep modules such as ResNet Block and DenseNet Block, the present invention also introduces Inception and Inception width modules with residual blocks to design the search space of the neural network from the depth and width structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of the present invention;

[0017] Figure 2 It is the node block design diagram of the present invention;

[0018] Figure 3 Schematic diagram of the neural network search method of the present invention. DETAILED DESCRIPTION

[0019] In order to make the technical scheme and technical advantages of the present invention more clear, the present invention is further described below with reference to the accompanying drawings and specific implementation schemes.

[0020] like Figure 1 As shown, a convolutional neural network search method for image classification includes the following steps:

[0021] S1: Divide the dataset into training dataset and test dataset, further sample the training dataset to form proxy dataset and validation dataset, and then perform data enhancement and preprocessing on the proxy dataset;

[0022] Two methods are used to form a proxy dataset. The first method is to randomly sample a subset from the entire dataset as the evaluation dataset, and the remaining data as the test dataset, with a sampling rate of 80%. The random sampling method retains the characteristics of the entire dataset, thereby reducing the loss of the model. Another method is to further divide the evaluation dataset to form a proxy dataset and a verification dataset, with a division ratio of 8:2. In fact, as long as the proxy dataset accounts for more than 70% of the evaluation dataset, it can be used. However, after a large number of experiments, the inventors found that the experimental effect is best when the division ratio is 8:2.

[0023] The proxy dataset is enhanced by random rotation of 15°, random mirror flipping, and random cropping, and then the entire dataset is normalized.

[0024] S2: Design the node blocks that constitute the convolutional neural network, and use a decimal and binary fusion encoding method to achieve deep search of candidate convolutional neural networks;

[0025] The node blocks that make up the convolutional neural network include ResNet Block and DenseNet Block depth modules, Inception and Inception with residual blocks width modules, TransitionBlock for downsampling feature maps, disabled modules, and fully connected modules, such as Figure 2 As shown in the figure, Res_33 and Res_131 respectively represent a ResNet Block consisting of two 3×3 convolutional layers and a ResNet Block consisting of a 1×1 convolutional layer, a 3×3 convolutional layer and a 1×1 convolutional layer; Den_5 and Den_3 respectively represent a DenseNet block consisting of five 3×3 convolutional layers and a DenseNet block consisting of three 3×3 convolutional layers; Incep_135M indicates that the module is an Inception module consisting of 1×1, 3×3, 5×5 and Maxpool, and Incep_13M indicates that the module is composed of Incep_135M with the 5×5 convolutional layer removed; Incep_135M_R and Incep_13M_R are Inception blocks with residual blocks formed by adding skip connections on the basis of Incep_135M and Incep_13M.

[0026] According to the characteristics of node blocks and genetic algorithms, a decimal and binary fusion encoding method is adopted. Since the optimal depth for a specific problem is unknown before the final CNN structure is determined, the present invention introduces a disabled module to realize CNN depth search. The depth of CNN has a significant impact on its performance. After the disabled module is introduced, the encoding method is not limited by the initial length, and there is a greater chance of searching for the optimal structure.

[0027] In this embodiment, indexes 0 and 1 are set to ResNet Block; indexes 2 and 3 are set to DenseNet Block; indexes 4 and 5 are set to Inception Block; indexes 6 and 7 are set to Inception Block with residual blocks; index 8 represents TransitionBlock; index 9 represents a disabled block, and when the code is 9, the layer is disabled; indexes 10 and 11 are fully connected layer modules, and the specific encoding method is shown in Table 1, where Full_512 indicates that the number of fully connected neurons is 512, and Full_256 indicates that the number of fully connected neurons is 256.

[0028] Table 1 Encoding method

[0029] Decimal encoding 0 1 2 3 4 5 6 Binary encoding 0000 0001 0010 0011 0100 0101 0110 Modules Res_33 Res_131 Den_5 Den_3 Incep_13M Incep_135M Incep_13M_R

[0030] Table 1 (Continued) Coding method

[0031] Decimal encoding 7 8 9 10 11 Binary encoding 0111 1000 1001 1010 1011 Modules Incep_135M_R Transition Disable Blocks Full_512 Full_256

[0032] S3: Design a proxy model to predict whether the candidate convolutional neural network represented by the new solution is better than the candidate convolutional neural network represented by the worst solution so far, and transform the problem of predicting the performance of the convolutional neural network into a binary classification problem;

[0033] When the present invention uses the genetic algorithm to select individuals, only individuals with higher fitness values ​​are selected, so when the new solution shows a bad trend, it does not need to spend too much time to obtain the fitness value. Therefore, a proxy model with short training time and evaluation time is designed to predict the entire training process. The specific expression of the proxy model M(x) is as follows:

[0034] f(z)=max(0,x)

[0035] M(x)=f(wixi+b)

[0036] In the formula, f is a nonlinear activation function, xi is the number of input features, wi is the weight coefficient, and b is the bias.

[0037] A nonlinear function is chosen as a proxy model to predict the performance of the two CNNs. The main reasons are as follows: the training data collected during the network evaluation process is limited, and nonlinear functions have better fitting effects than linear functions; compared with neural networks, nonlinear functions do not require a large amount of data for training and have lower computational costs.

[0038] The proxy model is used to predict whether the candidate convolutional neural network represented by the new solution is better than the candidate convolutional neural network represented by the worst solution so far, converting the problem of predicting the performance of the convolutional neural network into a binary classification problem.

[0039] Proxy model pre-training: The evaluation accuracy, optimal accuracy, and network coding method collected during the evaluation process are constructed into a proxy dataset for training the proxy model.

[0040] S4: Randomly create candidate convolutional neural networks as the initial population, use the proxy model for the evaluation process, select individuals with high fitness in the current population according to the fitness value, and perform mutation operations on the individuals to form the next generation population until the iteration is completed and the optimal individual is determined. The principle of the method is as follows Figure 3 As shown:

[0041] a. Randomly create candidate convolutional neural network individuals: initialize an individual list of length 15, with a maximum of 12 node blocks. The first node in the individual list must select the feature extraction node, the last node must be the classification layer, the second node to the twelfth node can select the feature extraction module and the disabled module, and the thirteenth node to the fourteenth node can only select the disabled module and the fully connected module. All nodes constitute a single individual in the initialized population;

[0042] In this step, it is necessary to use the above method to create multiple convolutional neural network individuals to form an initial population.

[0043] b. Calculate the individual fitness value according to the evaluation method of the image classification task. In this embodiment, the candidate convolutional neural network is trained for 20 epochs using the proxy data set, and the proxy model is used to predict whether the neural network can outperform the convolutional neural network represented by the worst solution so far. The loss function Ls formula used in the training process is as follows:

[0044]

[0045] Where y i Represents the feature value received by the Softmax loss function, N represents the number of images in a batch, and n represents the classification category;

[0046] In order to avoid the network model falling into the small model trap during the search process, in this embodiment, the accuracy of the candidate individual on the verification data set and the model size of the candidate individual are weighted and summed to obtain the fitness value F, and the specific calculation formula is as follows:

[0047] x mid =CNN i (D train )

[0048]

[0049] F d =Para(CNN i )

[0050] F=αF q +βF d

[0051] Where CNN represents the current convolutional neural network model, D train represents the proxy dataset, D val represents the validation data set, para(x mid ) represents the weight information of the neural network during the training process, T(·) represents whether the convolutional neural network predicts correctly on the validation data set, len(D val) represents the number of images in the validation dataset, Para(CNN i ) represents the number of parameters of the convolutional neural network, α and β are two hyperparameters, where α = 1 and β = 0.5;

[0052] c. Sort the individual fitness values, select individuals with high fitness in the current population according to the fitness value, and perform mutation operations on the individuals to form the next generation population until the iteration is completed and the optimal individual is determined.

[0053] In this embodiment, the 10 individuals with the highest fitness values ​​in the current population are selected as the parents of the next generation population, 6 mutation points are randomly selected, the values ​​of the 6 mutation points are modified, and 10 offspring are formed; 3 mutation points are randomly selected to add modules, 3 mutation points are deleted, and 10 offspring are generated again, and a total of 20 offspring are generated to form the next generation population. After the iteration is completed, the best individual is selected to be trained from scratch on the training data set.

[0054] In order to further verify the claims of the present invention, specific examples are used for illustration below.

[0055] The Cifar-10 dataset is a widely used dataset in the field of image processing, and its image size, classification category, and noise are close to real-world data. The Cifar-10 dataset contains 10 categories (such as airplanes, birds, ships, etc.), consisting of 60,000 RGB images with a size of 32×32×3, of which the training set and test set have 50,000 and 10,000 images respectively.

[0056] The classification accuracy of the method of the present invention on the Cifar-10 dataset is 94.1%, and the search takes 3 days on an NVIDIA GeforceRTX2080Ti graphics card. Compared with the manually designed network ResNet, DenseNet, the semi-automatic search algorithm Genetic CNN, the fully automatic search algorithm NAS, and Meta-QNN, the method of the present invention shows the best performance advantages in classification accuracy and search time, as shown in Table 2.

[0057] Table 2 Performance comparison on Cifar-10 dataset

[0058]

[0059] The above experimental results show that the method of the present invention can efficiently learn data features and achieve good results, and is feasible and applicable in the field of neural network search.

Claims

1. A convolutional neural network search method for image classification, characterized in that: The following steps are involved: S1: Divide the image dataset into a training image dataset and a test image dataset, and further randomly sample the training image dataset to form a proxy image dataset and a verification image dataset, and then perform data enhancement and preprocessing on the proxy image dataset; S2: Design the node blocks that constitute the image classification convolutional neural network, and use a decimal and binary fusion encoding method to achieve deep search of candidate convolutional neural networks that perform image classification tasks; S3: Design a proxy model to predict whether the candidate convolutional neural network represented by the new solution has better neural network performance than the candidate convolutional neural network represented by the worst solution so far when using the proxy image dataset for image classification tasks, and convert the problem of predicting convolutional neural network performance into a binary classification problem; S4: Randomly create candidate convolutional neural networks as the initial population, use the proxy model for the evaluation process, select individuals with higher fitness in the current population according to the fitness value calculated by the image classification accuracy and the convolutional neural network parameters, and perform mutation operations on the individuals to form the next generation population until the iteration is completed, determine the optimal individual, and obtain the optimal convolutional neural network search result for the image classification task; the specific operations are: a) Randomly create candidate convolutional neural network individuals: initialize an individual list of length L, with a maximum of B node blocks, the first node in the individual list must select a feature extraction node, the last node must be a classification layer, the second node to the Bth node can select feature extraction modules and disabled modules, and the (B+1)th node to the (L-1)th node can only select disabled modules and fully connected modules; all nodes constitute a single individual in the initialized population; b) According to the evaluation method of the image classification task, the candidate convolutional neural network is trained using the proxy dataset. The loss function Ls used in the training process is as follows: Where y i Represents the feature value received by the Softmax loss function, N represents the number of images in a batch, and n represents the classification category; The validation data set is used to evaluate the accuracy of the candidate individuals. The fitness value F is obtained by weighted summing up the model size and accuracy of the candidate individuals. The specific formula in the evaluation process is as follows: x mid =CNN i (D train ) F d =For(CNN i ) F=αF q +βF d Where CNN represents the current convolutional neural network model, D train represents the proxy dataset, D val represents the validation data set, para(x mid ) represents the weight information of the neural network during the training process, T(·) represents whether the convolutional neural network predicts correctly on the validation data set, len(D val ) represents the number of images in the validation dataset, Para(CNN i ) represents the number of parameters of the convolutional neural network, α and β are two hyperparameters; c) Sort the individual fitness values, select individuals with higher fitness in the current population according to the size of the fitness value, and perform mutation operations on the individuals to form the next generation population until the iteration is completed and the optimal individual is determined.

2. The method according to claim 1, characterized in that In step S1, the specific operations are: a) Divide the dataset into a training dataset and a test dataset, and further randomly sample the training dataset into a proxy dataset and a validation dataset, with a sampling ratio of 8:2; b) Perform image enhancement on the proxy dataset by randomly rotating it by a certain angle, randomly folding it horizontally, and randomly cropping it; c) Perform normalization on the proxy dataset.

3. The method according to claim 1, characterized in that In step S2, the specific operations are: a) The node blocks that make up the convolutional neural network include ResNet Block and DenseNet Block depth modules, Inception and Inception width modules with residual blocks, TransitionBlock for downsampling feature maps, and disabled modules; b) encoding the convolutional neural network using a decimal and binary fusion encoding method, wherein the decimal and binary fusion encoding method is specifically expressed as using decimal encoding to form a candidate convolutional neural network and using binary encoding to implement a mutation operation; c) Implementing variable depth search of convolutional neural networks, where the variable depth search manifests itself as a disabled module appearing during encoding, which is automatically deleted during decoding.

4. The method according to claim 1, characterized in that: In step S3, the specific operations are: a) Designing a proxy model: using a proxy model with short training time and evaluation time to predict the entire training process; b) The proxy model is used to predict whether the candidate convolutional neural network represented by the new solution is better than the candidate convolutional neural network represented by the worst solution so far, converting the problem of predicting the performance of the convolutional neural network into a binary classification problem; c) Proxy model pre-training: The evaluation accuracy, optimal accuracy, and network coding method collected during the evaluation process are constructed into a proxy dataset for training the proxy model.

Images