General neural architecture search framework based on mathematical programming

Abstract

The invention discloses a general neural architecture search framework based on mathematical programming, the general neural architecture search framework based on mathematical programming adopts a network performance evaluation method based on module performance, and the network performance can be directly calculated and predicted by a mathematical formula without training and verification on a data set. Therefore, the method can greatly reduce the time of network performance evaluation, further effectively reduces the time cost of network search, converts the traditional neural architecture search into a dynamic planning equation with constraint conditions, and performs network search by solving a mathematical equation, thereby ensuring the search accuracy and improving the network search efficiency. And thus, the searched network not only meets delay and power consumption constraint conditions, but also has high accuracy on the target task.

Description

Technical Field

[0001] The present invention relates to the field of neural architecture search technology, and specifically to a general neural architecture search framework based on mathematical programming. Background Art

[0002] Neural architecture search is an automated method for searching and discovering the optimal neural network structure for a specific task. First, a predefined search space is set, which is a superset of all possible neural network architectures. Then, a controller network is used to randomly generate a series of network architectures. Each generated architecture will be trained independently and evaluated on a validation set. The controller network updates the parameters based on the performance feedback of the selected architecture. The above architecture generation and parameter update process is iterated until the controller network selects a high-performance network architecture.

[0003] In traditional neural architecture search, the controller network selects architectures from the entire search space and trains all architectures independently. When searching for large neural networks, the search space often contains a large number of neural network architectures, and the training of a single network architecture requires dozens of hours of computer graphics card calculations, which makes the architecture search of large-scale networks require expensive search time and computing power. To solve this problem, several effective search strategies have been proposed, including one-time neural architecture search, network conversion, and architecture optimization. However, with the in-depth research in the field of neural architecture search, these methods have been proven by many experiments that the performance of the searched networks still needs to be improved or the search time cost still needs to be reduced.

[0004] In order to reduce the cost of the search network performance evaluation process, an existing method is to use a performance predictor to speed up the search process. Its core idea is to train a model that can predict the performance of a given neural network architecture, such as the prediction accuracy on a specific task. This prediction model is usually a machine learning model, such as a neural network or a gradient boosting tree. Once trained, the performance predictor can quickly evaluate a large number of candidate architectures, thereby greatly reducing the search time.

[0005] The general search process of neural architecture search based on performance predictors is as follows: in the predictor training phase, a set of neural network architectures are randomly sampled and trained to obtain their performance on the target task, and then the performance predictor is trained using the generated data set, so that the predictor can more accurately estimate the performance of any given neural network architecture. In the search and evaluation phase, a large number of candidate neural network architectures are evaluated using the performance predictor, and the optimal architecture is selected based on the predicted performance. Finally, the architecture with the best predicted performance is selected for training and verification on the target task dataset to confirm its actual performance;

[0006] Although the performance predictor can quickly evaluate the architecture, its prediction accuracy may be insufficient, which may cause the optimal architecture to be mistakenly ignored or the suboptimal architecture to be wrongly selected. At the same time, training an accurate performance predictor itself requires a lot of computing resources and time, especially in the pre-training stage. The architecture search method based on the performance predictor needs to train and evaluate a large number of neural network architectures to obtain sufficient training data. Therefore, the above methods still have room for performance improvement;

[0007] In view of this, we conducted in-depth research on the above issues and came up with this case, proposing a general neural architecture search framework based on mathematical programming. Summary of the invention

[0008] The purpose of the present invention is to provide a general neural architecture search framework based on mathematical programming to solve the problem that the existing neural architecture search mentioned in the above background technology is time-consuming and low-precision.

[0009] To achieve the above object, the present invention provides the following technical solution: a general neural architecture search framework based on mathematical programming, comprising the following steps:

[0010] Step 1: Construct search space S = {N1,...,N k ,...}, where each network architecture N k Contains m module positions, each of which has n selectable modules. i = {b i,1 ,...,b i,j ,...,b i,n}, for each module position, calculate the average FLOPs of all optional modules at that position, and select the module with the FLOPs value closest to the average of that position, and define the network composed of these modules as the benchmark network of the search space;

[0011] Step 2: To estimate the optional modules b i,j The performance of the benchmark network is switched to b i,j Keeping other modules in the network unchanged, test the accuracy, latency and power consumption of the switched network on the target tasks and devices.

[0012] Step 3: Switch the i-th module in the benchmark network to the optional module with the largest FLOPs at the i-th module position, while keeping other modules in the network unchanged, and test the accuracy, latency, and power consumption of the switched network on the target task and device;

[0013] Step 4: Calculate the difference in network accuracy, delay and power consumption, that is, subtract the corresponding values ​​recorded in step 2 from the network accuracy, delay and power consumption values ​​recorded in step 3, and use the difference to represent the optional module b i,jperformance;

[0014] Step 5: Determine whether the performance of all optional modules in the search space has been estimated. If yes, proceed to the next step. If not, repeat step 2.

[0015] Step 6: Locate the network with the largest FLOPs in the search space, that is, each module position of the network is the module with the largest FLOPs at that position, test the accuracy, latency and power consumption of the network on the target task and device, and construct a network performance prediction formula in the search space to quickly calculate and predict the performance of any network in the search space;

[0016] Step 7: Set the delay constraints and power consumption constraints for the searched network architecture, and establish the dynamic programming equation for architecture search under this condition.

[0017] As a preferred technical solution of the present invention, the network performance prediction formula in the search space in step 6 is:

[0018]

[0019] As a preferred technical solution of the present invention, in the network performance prediction formula in the search space, N=(b 1,j1 ,b 2,j2 ,...,b m,jm ) is the predicted network architecture, and are the network with the largest FLOPs in the search space and the benchmark network, FLOPs(N) is the FLOPs value of network N, Acc(N), Lat(N) and Eng(N) represent the accuracy of network N on the target task, the latency on the target device and the power consumption on the target device, respectively. They are network performance indicators. A i,jm 、b L i,jm and b E i,jm Respectively represent module b i,jm Accuracy performance, latency performance and power consumption performance.

[0020] As a preferred technical solution of the present invention, the dynamic programming equation in step seven is:

[0021]

[0022] As a preferred technical solution of the present invention, the dynamic programming equation and are the network delay constraints and power consumption constraints set respectively, F i,j For module b i,jFLOPs value, b B i,j Represents module b i,j Whether it is selected by the searched network, b B i,j Can only be 0 or 1, b B i,j =0 represents b i,j Not in the searched network, b B i,j =1 represents b i,j In the searched network, the first two constraints of the equation respectively constrain the delay and power consumption of the searched network architecture to not exceed the constraint values. The third constraint limits the structure of the searched network so that the searched network only selects one module at each module position. By using a mathematical solver to solve the set of equations, the network architecture with the highest accuracy that meets the delay and power consumption constraints in the search space can be obtained.

[0023] Compared with the prior art, the beneficial effects of the present invention are: the general neural architecture search framework based on mathematical programming adopts a network performance evaluation method based on module performance. The network performance can be directly calculated and predicted by mathematical formulas without the need for training and verification on data sets. Therefore, this method can greatly reduce the time for network performance evaluation, thereby effectively reducing the time cost of network search. In addition, the traditional neural architecture search is converted into a dynamic programming equation with constraints, and the network search is performed by solving mathematical equations, which ensures the accuracy of the search, so that the searched network not only meets the delay and power consumption constraints but also has a high accuracy rate in the target task. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of a neural architecture search framework of the present invention;

[0025] Figure 2 It is a schematic diagram of the process of the present invention;

[0026] Figure 3 This is a schematic diagram of the SuperViT search space of Experiment 1 of the present invention;

[0027] Figure 4 This is a schematic diagram of the experimental performance curve of Experiment 1 of the present invention;

[0028] Figure 5 This is a schematic diagram of the system flow of Experiment 2 of the present invention;

[0029] Figure 6 This is a schematic diagram of the experimental structure curve of Experiment 2 of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] See also Figure 1-6 , the technical solution of the present invention: a general neural architecture search framework based on mathematical programming, comprising the following steps:

[0032] Step 1: Construct search space S = {N1,...,N k ,...}, where each network architecture N k Contains m module positions, each of which has n selectable modules. i = {b i,1 ,...,b i,j ,...,b i,n}, for each module position, calculate the average FLOPs of all optional modules at that position, and select the module with the FLOPs value closest to the average of that position, and define the network composed of these modules as the benchmark network of the search space;

[0033] Step 2: To estimate the optional modules b i,j The performance of the benchmark network is switched to b i,j Keeping other modules in the network unchanged, test the accuracy, latency and power consumption of the switched network on the target tasks and devices.

[0034] Step 3: Switch the i-th module in the benchmark network to the optional module with the largest FLOPs at the i-th module position, while keeping other modules in the network unchanged, and test the accuracy, latency, and power consumption of the switched network on the target task and device;

[0035] Step 4: Calculate the difference in network accuracy, delay and power consumption, that is, subtract the corresponding values ​​recorded in step 2 from the network accuracy, delay and power consumption values ​​recorded in step 3, and use the difference to represent the optional module b i,j performance;

[0036] Step 5: Determine whether the performance of all optional modules in the search space has been estimated. If yes, proceed to the next step. If not, repeat step 2.

[0037] Step 6: Locate the network with the largest FLOPs in the search space, that is, each module position of the network is the module with the largest FLOPs at that position, test the accuracy, latency and power consumption of the network on the target task and device, and construct a network performance prediction formula in the search space to quickly calculate and predict the performance of any network in the search space;

[0038] Step 7: Set the delay constraints and power consumption constraints for the searched network architecture, and establish the dynamic programming equation for architecture search under this condition.

[0039] The network performance prediction formula in the search space in step 6 is:

[0040]

[0041] In the network performance prediction formula in the search space, N = (b 1,j1 ,b 2,j2 ,...,b m,jm ) is the predicted network architecture, and are the network with the largest FLOPs in the search space and the benchmark network, FLOPs(N) is the FLOPs value of network N, Acc(N), Lat(N) and Eng(N) represent the accuracy of network N on the target task, the latency on the target device and the power consumption on the target device, respectively. They are network performance indicators. A i,jm 、b L i,jm and b E i,jm Respectively represent module b i,jm Accuracy performance, latency performance and power consumption performance.

[0042] The dynamic programming equation in step seven is:

[0043]

[0044] In the dynamic programming equation and are the network delay constraints and power consumption constraints set respectively, F i,j For module b i,j FLOPs value, b B i,j Represents module b i,j Whether it is selected by the searched network, b B i,j Can only be 0 or 1, b B i,j =0 represents b i,j Not in the searched network, b Bi,j =1 represents b i,j In the searched network, the first two constraints of the equation respectively constrain the delay and power consumption of the searched network architecture to not exceed the constraint values. The third constraint limits the structure of the searched network so that the searched network only selects one module at each module position. By using a mathematical solver to solve the set of equations, the network architecture with the highest accuracy that meets the delay and power consumption constraints in the search space can be obtained.

[0045] Experiment 1

[0046] The above neural architecture search framework is applied to the SuperViT (Super Vision Transformer) search space to automatically design the ViT network. The SuperViT search space is a layer connection search space with a fixed macro architecture. It consists of a convolution block, three reverse residual blocks, four self-attention modules, and a pooling block connected in sequence. Each module is allowed to vary in width, depth, convolution kernel size, and expansion ratio. The classification task of the color image dataset ImageNet is used as the target task. The dataset contains more than one million images, covering one thousand different object categories, and each category contains about one thousand training samples and hundreds of verification samples.

[0047] In this experiment, through Figure 4 Experimental results show that on the ImageNet real dataset, the searched network architecture can achieve the best trade-off between image recognition accuracy and network FLOPs;

[0048] Through the above experiments, this neural architecture search framework introduces the definition of network module performance based on the traditional neural architecture search, and then transforms the problem of searching the network architecture in the entire search space into the problem of searching the module in the module space. The key to using the neural architecture search method to complete the architecture design is to ensure the effectiveness of the searched network while reducing the search time as much as possible. It not only uses the idea of ​​network splitting to reduce the complexity of the architecture search, but also ensures the effectiveness of the searched architecture by solving mathematical programming problems. Therefore, the network search time can be greatly reduced, and a higher accuracy network can be searched under the same constraints.

[0049] Experiment 2

[0050] The neural architecture search framework is deployed on the hardware devices NVIDIA Jetson TX2 CPU and NVIDIA Jetson TX2 GPU to quickly search for a suitable network architecture on the edge device and switch the running architecture according to the current device resource conditions, so that the target task can be better completed on the hardware device. This example is performed on the MobileNetV3 search space. The MobileNetV3 search space follows a layer-by-layer paradigm, in which the building blocks use reverse residual blocks, squeezing and excitation mechanisms to build efficient neural networks. The space is divided into five modules, each of which contains multiple network layers ranging from 2 to 4. The kernel size of each module can be selected from {3, 5, 7}, and the expansion ratio can be selected from {1, 4, 6}. The search space encapsulates about 10 19 There are 7371 optional modules in each module position, and the classification task of the color image dataset ImageNet is used as the target task;

[0051] like Figure 5 As shown, in the offline stage, the accuracy performance and delay performance of all optional modules in the search space are tested and calculated, and five optional modules with Pareto optimal accuracy and delay performance are selected at each module position;

[0052] The offline phase includes the following 4 steps:

[0053] Step 1: Pre-train the largest network in the search space and share the weights of the trained network with other modules in the search space;

[0054] Step 2: Test and calculate the accuracy performance of all optional modules in the search space on the target task set;

[0055] Step 3: Test the latency of all optional modules on the target hardware device;

[0056] Step 4: For each module position, select five modules with the best accuracy performance and latency, and record their structure information, accuracy performance information and latency information. Deploy the Pareto optimal modules and their information on the hardware device. The contents not described in detail in this manual belong to the prior art known to professional and technical personnel in this field;

[0057] In the offline phase, delay constraints are set, and the dynamic programming equations for architecture search are established based on the performance information of the module. The equations are solved using the equation solver on the hardware device, and then the solved network is run. However, when the delay of the running network exceeds the delay constraint, the delay table is updated and the architecture search is performed again. The online phase includes the following four steps:

[0058] Step 1: Set the delay constraints of the running network;

[0059] Step 2: Establish a set of dynamic programming equations for neural architecture search based on the delay constraints and the performance information of the optional modules, use the solver on the hardware device to solve the dynamic programming equations for neural architecture search, and deploy the solved architecture on the hardware to run;

[0060] Step 3: Monitor the delay of the running network to determine whether the running delay of the current network exceeds the set delay constraint;

[0061] Step 4: Update the module's delay information table: b L i,j =b L i,j *NewLat(N) / OladLat(N)

[0062] Where NewLat(N) and OladLat(N) represent the detected delay of network N and the delay of network N calculated using the old delay information table, respectively. The delay information of all deployed optional modules is updated, and the process goes to step 2 for the next round of architecture search.

[0063] In this experiment, through Figure 6 Experimental results show that on the real ImageNet dataset, the searched network architecture can achieve the best trade-off between image recognition accuracy and network operation latency. At the same time, whether on the NVIDIA JetsonTX2 CPU or the NVIDIA Jetson TX2 GPU, the searched architecture has shown an accuracy rate that is better than the baseline, which further demonstrates its versatility for target hardware devices.

[0064] Through the above experiments, this neural architecture search framework has moved the traditional neural architecture search from the GPU server to the target hardware device, realizing real-time and fast search on the target hardware device, thereby making the neural network running on the target hardware device have higher accuracy and ensuring that the delay constraints are met. The key to running the network searched out by the neural architecture on the hardware device is that the neural network needs to achieve as high accuracy as possible on the task while satisfying the delay constraints. Not only does the delay monitoring module monitor the network's running delay in real time to ensure that the delay of the running network does not exceed the delay constraint, but also the accuracy of the searched architecture is guaranteed by solving mathematical programming problems. Therefore, it can more effectively ensure that the network running on the hardware device has high accuracy on the task set without exceeding the running delay constraint.

[0065] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A general neural architecture search framework based on mathematical programming, characterized in that: The following steps are involved: Step 1: Construct search space S = {N1,...,N k ,...}, where each network architecture N k Contains m module positions, each of which has n selectable modules. i = {b i,1 ,...,b i,j ,...,b i,n }, for each module position, calculate the average FLOPs of all optional modules at that position, and select the module with the FLOPs value closest to the average of that position, and define the network composed of these modules as the benchmark network of the search space; Step 2: To estimate the optional modules b i,j The performance of the benchmark network is switched to b i,j Keeping other modules in the network unchanged, test the accuracy, latency and power consumption of the switched network on the target tasks and devices. Step 3: Switch the i-th module in the benchmark network to the optional module with the largest FLOPs at the i-th module position, while keeping other modules in the network unchanged, and test the accuracy, latency, and power consumption of the switched network on the target task and device; Step 4: Calculate the difference in network accuracy, delay and power consumption, that is, subtract the corresponding values ​​recorded in step 2 from the network accuracy, delay and power consumption values ​​recorded in step 3, and use the difference to represent the optional module b i,j performance; Step 5: Determine whether the performance of all optional modules in the search space has been estimated. If yes, proceed to the next step. If not, repeat step 2. Step 6: Locate the network with the largest FLOPs in the search space, that is, each module position of the network is the module with the largest FLOPs at that position, test the accuracy, latency and power consumption of the network on the target task and device, and construct a network performance prediction formula in the search space to quickly calculate and predict the performance of any network in the search space; Step 7: Set the delay constraints and power consumption constraints for the searched network architecture, and establish the dynamic programming equation for architecture search under this condition.

2. A general neural architecture search framework based on mathematical programming according to claim 1, characterized in that: The network performance prediction formula in the search space in step 6 is:

3. A general neural architecture search framework based on mathematical programming according to claim 2, characterized in that: In the network performance prediction formula in the search space, N = (b 1,j1 ,b 2,j2 ,...,b m,jm ) is the predicted network architecture, and are the network with the largest FLOPs in the search space and the benchmark network, FLOPs(N) is the FLOPs value of network N, Acc(N), Lat(N) and Eng(N) represent the accuracy of network N on the target task, the latency on the target device and the power consumption on the target device, respectively. They are network performance indicators. A i,jm 、b L i,jm and b E i,jm Respectively represent module b i,jm Accuracy performance, latency performance and power consumption performance.

4. A general neural architecture search framework based on mathematical programming according to claim 1, characterized in that: The dynamic programming equation in step seven is:

5. A general neural architecture search framework based on mathematical programming according to claim 4, characterized in that: The dynamic programming equation and are the network delay constraints and power consumption constraints set respectively, F i,j For module b i,j FLOPs value, b B i,j Represents module b i,j Whether it is selected by the searched network, b B i,j Can only be 0 or 1, b B i,j =0 represents b i,j Not in the searched network, b B i,j =1 represents b i,j In the searched network, the first two constraints of the equation respectively constrain the delay and power consumption of the searched network architecture to not exceed the constraint values. The third constraint limits the structure of the searched network so that the searched network only selects one module at each module position. By using a mathematical solver to solve the set of equations, the network architecture with the highest accuracy that meets the delay and power consumption constraints in the search space can be obtained.

Images