Differentiable neural network architecture search method based on zeroth order approximation

Abstract

This invention provides a method for searching differentiable neural network architectures based on zero-order approximation, characterized by the following steps: Step S1, constructing an initial neural network including neurons and downsampling blocks; Steps S2 to S6, constructing a structural parameter matrix, an oscillating structural parameter matrix, a model parameter matrix, and an oscillating model parameter matrix; Step S7, updating the model parameter matrix and the oscillating model parameter matrix; Step S8, inputting the validation dataset into the initial neural network, and obtaining the updated structural parameter matrix through a second loss function and gradient descent; Step S9, determining whether the iteration is complete; Step S10, selecting the neural network operation corresponding to the maximum value of the structural parameters of each connection edge in the updated structural parameter matrix as the neural network operation for that connection edge, thus obtaining the complex neural network. In summary, this method can improve the efficiency and accuracy of searching complex neural networks.

Description

Technical Field

[0001] This invention belongs to the field of artificial neural network technology, specifically relating to a search method for differentiable neural network architectures based on zero-order approximation. Background Technology

[0002] Deep learning, as an end-to-end artificial intelligence technology, can completely delegate the feature extraction process to neural network models without using feature engineering or expert experience. Currently, methods in this field have achieved remarkable results in tasks such as image and speech processing. However, designing efficient and highly accurate neural network models requires extensive professional knowledge and manual parameter tuning, which greatly hinders the application of deep learning technology to a wide range of practical problems.

[0003] To address this problem, the field of artificial intelligence has begun to consider using Neural Architecture Search (NAS) methods. The DARTS algorithm is a representative differentiable structure search strategy within NAS. It assigns continuous weights to candidate operations and performs weighted mixing during the search, thereby making the search space continuous and forming a differentiable bilayer optimization problem. Gradient descent is then used for performance optimization. After optimization, DARTS selects the operation with the largest weight from the mixed operations, thus determining a high-performance neural network framework with a complex topology within the rich search space.

[0004] However, the existing DARTS automatic architecture search algorithm is not precise enough in solving optimization problems and requires a large time cost, and still suffers from inefficiency and low accuracy. Summary of the Invention

[0005] This invention is made to solve the above-mentioned problems, and aims to provide a search method for differentiable neural network architectures based on zero-order approximation.

[0006] This invention provides a method for searching differentiable neural network architectures based on zero-order approximation, used to obtain the corresponding complex neural network from a training dataset, a validation dataset, and a neural network operations. The method includes the following steps: Step S1, constructing an initial neural network comprising c neurons and d downsampling blocks, where each neuron includes k connection edges; Step S2, selecting b neural network operations from the a neural network operations as a set of neural network operations. Step S3: Assign each of the k connections of a nerve cell to a set of neural network operations. The process involves b neural operations to obtain k*b corresponding structural parameters. A structural parameter matrix α is constructed based on these k*b parameters, and the iteration number p is set to 1. In step S4, the minimum value μ and a random unit vector u are set based on the structural parameter matrix α. In step S5, the oscillating structural parameter matrix is ​​obtained based on the structural parameter matrix α, the minimum value μ, and the random unit vector u. Step S6: Set all weight parameter matrices of the initial neural network as the model parameter matrix w, and set the oscillation model parameter matrix according to the model parameter matrix w. Step S7, based on the training dataset, the structure parameter matrix α, and the oscillation structure parameters... By using the first loss function and gradient descent method, we obtain the updated model parameter matrix w' and the updated oscillation model parameter matrix. Step S8: Input the validation dataset into the initial neural network, and obtain the updated structural parameter matrix α' through the second loss function and gradient descent method; Step S9: Increment the iteration number p by 1, and then determine whether the iteration number p is less than the maximum iteration number q. If so, use the updated structural parameter matrix α' as the structural parameter matrix α and proceed to step S4. If not, proceed to step S10; Step S10: Select the neural network operation corresponding to the maximum value among the b structural parameters of each connection edge in the updated structural parameter matrix α' as the neural network operation of that connection edge, thus obtaining the complex neural network.

[0007] The differentiable neural network architecture search method based on zero-order approximation provided by this invention may also have the following feature: wherein the neural network operation is an operation that arbitrarily preserves the data dimension.

[0008] The differentiable neural network architecture search method based on zero-order approximation provided by this invention may also have the following features: wherein the neural network operations include zero operations, skip connection operations, 1×1 convolution operations, 3×3 convolution operations and 3×3 average pooling operations.

[0009] The differentiable neural network architecture search method based on zero-order approximation provided by this invention may also have the following feature: wherein the first loss function and the second loss function are both cross-entropy functions.

[0010] The search method for differentiable neural network architecture based on zero-order approximation provided by this invention may also have the following feature: wherein, in step S4, the minimum value μ = 0.005||α||2, and the expression for the random unit vector u is as follows: In the formula, u' is a random vector with k*b elements, each of which follows a normal distribution.

[0011] The differentiable neural network architecture search method based on zero-order approximation provided by this invention may also have the following feature: wherein, in step S5, the oscillation structure parameter matrix The expression is as follows: In the formula, u” is a matrix obtained by aligning the dimensions of the random unit vector u with the structure parameter matrix α.

[0012] The search method for differentiable neural network architectures based on zero-order approximation provided by this invention may also have the following features: Step S7 includes the following sub-steps: Step S7-1, setting the iteration round t = 1; Step S7-2, inputting the training dataset into the initial neural network, combining the structure parameter matrix α and the oscillating structure parameter matrix. And based on the first loss function and the learning rate η w Update the model parameter matrix w and the oscillation model parameter matrix. Step S7-3: Increment the iteration number t by 1. Determine if the iteration number t is less than the maximum iteration number TT. If yes, proceed to step S7-2. If no, when the iteration number t is the maximum iteration number TT, the updated model parameter matrix w is the updated model parameter matrix w'. The oscillation model parameter matrix... The update result is the updated oscillation model parameter matrix.

[0013] The differentiable neural network architecture search method based on zero-order approximation provided by this invention may also have the following feature: wherein, in step S7-2, the first loss function is a loss function. and loss function loss function The value is calculated based on the model parameter matrix w and the structural parameter matrix α using the training dataset; the loss function is... The value is based on the parameter matrix of the oscillation model from the training dataset. and oscillation structure parameter matrix The calculation results are based on the learning rate η. w and loss function gradient with respect to the model parameter matrix w The updated model parameter matrix w” is calculated using the gradient descent method and used as the model parameter matrix W at iteration t+1, based on the learning rate η. w and loss function For the parameter matrix of the oscillation model gradient The updated oscillation model parameter matrix is ​​calculated using the gradient descent method. The parameter matrix of the oscillation model at iteration t+1

[0014] The search method for differentiable neural network architectures based on zero-order approximation provided by this invention may also have the following feature: wherein, in step S8, the second loss function is a loss function. loss function The value is the approximate gradient of the structure parameter matrix α, calculated based on the validation dataset using the model parameter matrix w and the structure parameter matrix α. The calculation formula is: In the formula for The gradient of the model parameter matrix W is represented by W'(α), indicating that the update of the model parameter matrix W' is considered to be related to the structure parameter matrix α in the gradient calculation. for The gradient of the structure parameter matrix α is expressed as W'(), which indicates that the updated model parameter matrix w' is treated as a constant and is independent of the structure parameter matrix α in the gradient calculation. The superscript T indicates the transpose operation, based on the set learning rate η. α Approximate gradient of the structure parameter matrix α The updated structure parameter matrix α' is calculated using the gradient descent method.

[0015] The role and effect of invention

[0016] According to the zero-order approximation-based differentiable neural network architecture search method of the present invention, a neural network operation set is constructed by selecting multiple neural network operations, and a structure parameter matrix is ​​constructed based on the neural network operation set. The structure parameter matrix is ​​then updated using the gradient descent method, and the maximum value of the structure parameter corresponding to each connection edge in the updated structure parameter matrix is ​​selected as the neural network operation for the corresponding connection edge, thus obtaining a complex neural network. Therefore, the zero-order approximation-based differentiable neural network architecture search method of the present invention can improve the efficiency and accuracy of searching for complex neural networks. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the search method for a differentiable neural network architecture based on zero-order approximation in an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of the architecture of a complex neural network model for image recognition in an embodiment of the present invention;

[0019] Figure 3 This is a schematic diagram comparing the neural cells of the model obtained by the algorithm of this invention and the model obtained by the DARTS-V2 algorithm in an embodiment of this invention. Detailed Implementation

[0020] To make the technical means, creative features, objectives and effects of this invention easier to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the search method for differentiable neural network architectures based on zero-order approximation of this invention.

[0021] Figure 1 This is a flowchart illustrating the search method for a differentiable neural network architecture based on zero-order approximation in an embodiment of the present invention.

[0022] like Figure 1 As shown, the zero-order approximation-based differentiable neural network architecture search method in this embodiment is used to obtain the corresponding complex neural network through a training dataset, a validation dataset, and a neural network operations, and includes the following steps:

[0023] Step S1: Construct an initial neural network consisting of c neurons and d downsampling blocks. Each neuron has k connection edges.

[0024] Step S2: Select b neural network operations from a neural network operations to form a set of neural network operations.

[0025] Among them, neural network operations are operations that arbitrarily preserve the dimensions of the data.

[0026] The neural network operations include zero operations, skip connection operations, 1×1 convolution operations, 3×3 convolution operations, and 3×3 average pooling operations.

[0027] Step S3: Assign each of the k connections of a nerve cell to a set of neural network operations. b neural operations are performed to obtain k*b corresponding structural parameters. Based on the k*b structural parameters, a structural parameter matrix α is constructed, and the iteration round p is set to 1.

[0028] Step S4: Based on the structural parameter matrix α, set the minimum value μ and the random unit vector w.

[0029] Where the minimum value μ = 0.005||α||2, the expression for the random unit vector u is as follows:

[0030]

[0031] In the formula, u' is a random vector with k*b elements, each of which follows a normal distribution.

[0032] Step S5: Based on the structural parameter matrix α, the minimum value μ, and the random unit vector u, obtain the oscillating structural parameter matrix.

[0033] Among them, the oscillation structure parameter matrix The expression is as follows:

[0034]

[0035] In the formula, u” is a matrix obtained by aligning the dimensions of the random unit vector u with the structure parameter matrix α.

[0036] Step S6: Set all weight parameter matrices of the initial neural network as the model parameter matrix w, and set the oscillation model parameter matrix according to the model parameter matrix w.

[0037] Step S7, based on the training dataset, the structure parameter matrix α, and the oscillation structure parameter matrix... By using the first loss function and gradient descent method, we obtain the updated model parameter matrix w' and the updated oscillation model parameter matrix.

[0038] Step S7 includes the following sub-steps:

[0039] Step S7-1: Set the iteration round t = 1.

[0040] Step S7-2: Input the training dataset into the initial neural network, combining the structural parameter matrix α and the oscillating structural parameter matrix. And based on the first loss function and the learning rate η w Update the model parameter matrix w and the oscillation model parameter matrix.

[0041] Wherein, the first loss function is the loss function. and loss function loss function The value is calculated based on the model parameter matrix w and the structural parameter matrix α using the training dataset; the loss function is... The value is based on the parameter matrix of the oscillation model from the training dataset. and oscillation structure parameter matrix The calculation results show that the first loss function in this embodiment is the cross-entropy function.

[0042] Based on the learning rate η w and loss function gradient with respect to the model parameter matrix w The updated model parameter matrix w” is calculated using the gradient descent method and used as the model parameter matrix w at iteration t+1.

[0043] In this embodiment, the formula for calculating the updated model parameter matrix w” is:

[0044]

[0045] Based on the learning rate η w and loss function For the parameter matrix of the oscillation model gradient The updated oscillation model parameter matrix is ​​calculated using the gradient descent method. The parameter matrix of the oscillation model at iteration t+1

[0046] In this embodiment, the oscillation model parameter matrix is ​​updated. The calculation formula is:

[0047]

[0048] In this embodiment, the learning rate η w The calculation formula is:

[0049]

[0050] In the formula This is the minimum learning rate. This represents the maximum learning rate.

[0051] Step S7-3: Increment the iteration number t by 1. Determine if the iteration number t is less than the maximum iteration number TT. If yes, proceed to step S7-2. If no, when the iteration number t is the maximum iteration number TT, the updated model parameter matrix w is the updated model parameter matrix w'. The oscillation model parameter matrix... The update result is the updated oscillation model parameter matrix.

[0052] Step S8: Input the validation dataset into the initial neural network, and obtain the updated structural parameter matrix α' through the second loss function and gradient descent method.

[0053] Wherein, the second loss function is the loss function. loss function The value is calculated based on the model parameter matrix w and the structural parameter matrix α using the validation dataset. In this embodiment, the second loss function is the cross-entropy function.

[0054] Approximate gradient of the structure parameter matrix α The calculation formula is:

[0055]

[0056]

[0057] In the formula for The gradient of the model parameter matrix w is represented by w'(α), indicating that updating the model parameter matrix w' in this gradient calculation is considered to be related to the structure parameter matrix α. for The gradient of the structure parameter matrix α is expressed as w'(), which means that the updated model parameter matrix w' is treated as a constant and is independent of the structure parameter matrix α in the gradient calculation. The superscript T indicates the transpose operation.

[0058] Based on the set learning rate η α Approximate gradient of the structure parameter matrix α The updated structure parameter matrix α' is calculated using the gradient descent method. In this embodiment, the gradient descent method is an adaptive moment estimation algorithm based on given hyperparameters.

[0059] Step S9: Increment the iteration number p by 1, and then determine whether the iteration number p is less than the maximum iteration number q. If so, update the structure parameter matrix α' as the structure parameter matrix α and proceed to step S4. If not, proceed to step S10.

[0060] Step S10: Select the neural network operation corresponding to the maximum value among the b structural parameters of each connection edge in the updated structural parameter matrix α' as the neural network operation for that connection edge, and then obtain the complex neural network.

[0061] In this embodiment, targeting an image recognition problem scenario, 25,000 images are randomly selected from the CIFAR-10 dataset as the training dataset, and another 25,000 images are randomly selected from the remaining images in the CIFAR-10 dataset as the validation dataset. An initial neural network is constructed, comprising c = 15 neurons and d = 2 downsampling blocks. Each neuron includes 4 data nodes and k = 6 connection edges. The neural network operation set... There are 5 neural network operations, specifically zero operations, skip connection operations, 1×1 convolution operations, 3×3 convolution operations, and 3×3 average pooling operations. The zero operation involves multiplying the data by 0 and setting... TT=10, set the learning rate η α With q = 0.015 and q = 50, a complex neural network model for image recognition is obtained as the model of this invention.

[0062] In this process, data x from data node i is used as input, and after being processed through a hybrid operation, it is output to data node j. The calculation formula is as follows:

[0063]

[0064] In the formula, o and o' are both sets of neural network operations. One of the neural network operations, Let the edge connecting data node i to data node j be the structural parameter when performing neural network operation o. Let be the structural parameters of the connection edge from data node i to data node j when performing neural network operation o', exp() be an exponential function with base e, and o(x) be the result of input x after neural network operation o.

[0065] Figure 2 This is a schematic diagram of the architecture of a complex neural network model for image recognition in an embodiment of the present invention.

[0066] like Figure 2 As shown, the resulting complex neural network model for image recognition includes a convolutional network, neurons, downsampling blocks, and global average pooling. In this embodiment, the convolutional network organizes the dimensions of the input data, i.e., the image to be recognized, into dimensions that can be computed by the neurons. Global average pooling further processes the feature data calculated by the neurons and downsampling blocks to obtain the final result, i.e., the recognition result of the image to be recognized.

[0067] The following algorithms were used for model search: the zero-order approximation-based differentiable neural network architecture search method of this invention (i.e., the algorithm of this invention); the DARTS algorithm DARTS-V1, which uses a first-order expansion approximation for the inner layer optimization problem of two layers; the DARTS algorithm DARTS-V2, which uses a second-order expansion approximation for the inner layer optimization problem of two layers; the iDARTS structure search algorithm, which uses the hidden gradient method to solve the DARTS two-layer optimization problem; the PC-DARTS structure search algorithm, which is based on DARTS but reduces the computational cost of a single search; and the MileNAS structure search algorithm, which transforms the DARTS two-layer optimization problem into a single-layer optimization problem. The search time of each algorithm was compared. The image recognition accuracy of each model on the CIFAR-10 test dataset provided by the NAS-Bench-201 structure search dataset was obtained by querying the API of the NAS-Bench-201 structure search dataset. The image recognition accuracy and search time of the searched models were obtained by averaging three experiments. The search time and image recognition accuracy of each algorithm are shown in the table below.

[0068]

[0069] The first column of the table lists the names of each algorithm, the second column lists the search time for each algorithm, and the third column lists the image recognition accuracy of the model obtained by each algorithm. For example, the cell in the seventh row and third column indicates that the image recognition accuracy of the model obtained by this invention is 91.88 ± 0.46%, where 0.46 is the standard deviation. As shown in the table, the model obtained by the algorithm of this invention has a higher image recognition accuracy than the image recognition models obtained by the other five algorithms on the CIFAR-10 dataset, and the required search time is significantly reduced compared to other methods with higher accuracy, making it more efficient than the other five methods.

[0070] Figure 3 This is a schematic diagram comparing the neural cells of the model obtained by the algorithm of this invention and the model obtained by the DARTS-V2 algorithm in an embodiment of this invention.

[0071] like Figure 3 As shown in Figure (a), the neural network operations between data nodes in the neurons of the model obtained by searching using the DARTS-V2 algorithm (a second-order expansion approximation) for the two-layer optimization inner layer problem in image recognition are as follows: data node 0 to data node 1 is a skip connection operation, data node 0 to data node 2 is a skip connection operation, data node 0 to data node 3 is a skip connection operation, data node 1 to data node 2 is a skip connection operation, data node 1 to data node 3 is a skip connection operation, and data node 2 to data node 3 is a skip connection operation. For skip connection operations, Figure (b) shows the neural network operations between data nodes in the neurons of the model searched using the zero-order approximation-based differentiable neural network architecture search method of this invention for image recognition. Specifically, data node 0 to data node 1 is a 3×3 average pooling operation, data node 0 to data node 2 is a skip connection operation, data node 0 to data node 3 is a zero operation, data node 1 to data node 2 is a 1×1 convolution operation, data node 1 to data node 3 is a 1×1 convolution operation, and data node 2 to data node 3 is a 1×1 convolution operation. It can be seen that the neurons of the model searched by the algorithm of this invention contain only one skip connection operation, which retains sufficient parameters compared to the model searched by DARTS-V2. Therefore, the model searched by the algorithm of this invention has higher accuracy in image recognition than the model searched by DARTS-V2.

[0072] The role and effect of the embodiments

[0073] According to the zero-order approximation-based differentiable neural network architecture search method involved in this embodiment, a neural network operation set is constructed by selecting multiple neural network operations, and then a structure parameter matrix is ​​constructed based on the neural network operation set. The structure parameter matrix is ​​updated using the gradient descent method, and the maximum value of the structure parameter corresponding to each connection edge in the updated structure parameter matrix is ​​selected as the neural network operation for the corresponding connection edge. This method can quickly obtain complex neural networks, and by retaining more parameters, it improves the recognition accuracy of complex neural networks. In summary, this method can improve the efficiency and accuracy of searching for complex neural networks.

[0074] The above embodiments are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. A method for searching differentiable neural network architectures based on zero-order approximation, used to search through training datasets, validation datasets, and... A complex neural network is obtained through multiple neural network operations, thereby solving the image recognition problem. Multiple images are randomly selected from the dataset as the training dataset, and the same number of images are randomly selected from the remaining images in the dataset as the validation dataset. The characteristic of this method is that... Includes the following steps: Step S1, construct including One nerve cell and An initial neural network of downsampled blocks, the neurons comprising: One connecting edge; Step S2, from the Selecting from neural network operations The neural network operations are referred to as a set of neural network operations. ; Step S3, the nerve cells Each of the connecting edges corresponds to the neural network operation set. of The neural operations described above yield the corresponding... Each structural parameter, according to the Each structural parameter is used to construct a structural parameter matrix. Set the iteration rounds =1; Step S4, based on the structural parameter matrix Set minimum value and random unit vector ; Step S5, based on the structural parameter matrix The minimum value and the random unit vector The oscillation structure parameter matrix is ​​obtained. ; Step S6: Set all weight parameter matrices of the initial neural network as the model parameter matrix. According to the model parameter matrix Set the oscillation model parameter matrix ; Step S7, based on the training dataset and the structure parameter matrix and the oscillation structure parameter matrix The updated model parameter matrix is ​​obtained by using the first loss function and gradient descent method. and update the oscillation model parameter matrix ; Step S8: Input the validation dataset into the initial neural network, and obtain the updated structure parameter matrix through the second loss function and the gradient descent method. ; Step S9, the iteration rounds Add 1, then determine the iteration round. Is it less than the maximum number of iterations? If so, then update the structure parameter matrix. As the structural parameter matrix If not, proceed to step S4; otherwise, proceed to step S10. Step S10: Select the updated structural parameter matrix respectively. Each of the connecting edges in the middle corresponds to The neural network operation corresponding to the maximum value among the structural parameters is used as the neural network operation for the connecting edge, thus obtaining the complex neural network. In step S4, the minimum value , The random unit vector The expression is as follows: ， In the formula For having A random vector of n elements, each of which follows a normal distribution. In step S5, the oscillation structure parameter matrix The expression is as follows: ， In the formula For the random unit vector By using the structural parameter matrix The matrix obtained by aligning the dimensions.

2. The search method for differentiable neural network architectures based on zero-order approximation according to claim 1, characterized in that: in, The neural network operation is any operation that preserves the data dimensions.

3. The search method for differentiable neural network architectures based on zero-order approximation according to claim 2, characterized in that: in, The neural network operations include zero operations, skip connection operations, Convolution operation Convolution operations and Average pooling operation, The zero operation means multiplying the data by 0.

4. The search method for differentiable neural network architectures based on zero-order approximation according to claim 1, characterized in that: in, Both the first loss function and the second loss function are cross-entropy functions.

5. The search method for differentiable neural network architectures based on zero-order approximation according to claim 1, characterized in that: in, Step S7 includes the following sub-steps: Step S7-1, Set the iteration rounds ; Step S7-2: Input the training dataset into the initial neural network, and combine it with the structure parameter matrix. and the oscillation structure parameter matrix And based on the first loss function and learning rate Update the model parameter matrix and the oscillation model parameter matrix ; Step S7-3, the iteration rounds Increment by 1 to determine the iteration round. Is it less than the maximum number of iterations? If yes, proceed to step S7-2; otherwise, proceed to the next iteration. Maximum number of iterations At that time, the model parameter matrix The update result is the updated model parameter matrix. The oscillation model parameter matrix The updated result is the updated oscillation model parameter matrix. .

6. The search method for differentiable neural network architectures based on zero-order approximation according to claim 5, characterized in that: in, In step S7-2, the first loss function is a loss function. and loss function The loss function The value is based on the training dataset for the model parameter matrix. and the structural parameter matrix The calculation results, the loss function The value is based on the training dataset for the oscillation model parameter matrix. and the oscillation structure parameter matrix The calculation results According to the learning rate and the loss function For the model parameter matrix gradient The updated model parameter matrix is ​​calculated using the gradient descent method. As an iteration round The model parameter matrix at that time , According to the learning rate and the loss function For the oscillation model parameter matrix gradient The updated oscillation model parameter matrix is ​​calculated using the gradient descent method. As the iteration round The oscillation model parameter matrix at that time .

7. The search method for differentiable neural network architectures based on zero-order approximation according to claim 1, characterized in that: in, In step S8, the second loss function is a loss function. The loss function The value is based on the validation dataset for the model parameter matrix. and the structural parameter matrix The calculation results The structural parameter matrix approximate gradient The calculation formula is: ， ， In the formula for For the model parameter matrix gradient, This indicates that the updated model parameter matrix will be used in the gradient calculation. Considered as the structural parameter matrix related, for For the structural parameter matrix gradient, This indicates that the updated model parameter matrix will be used in the gradient calculation. Considered as a constant value and the structural parameter matrix Irrelevant, superscript For transpose operation, Based on the set learning rate and the structural parameter matrix approximate gradient The updated structure parameter matrix is ​​calculated using the gradient descent method. .

Images