A method for constructing binary neural networks based on neural network architecture search

Abstract

This invention discloses a method for constructing a binary neural network based on neural network architecture search, belonging to the field of object recognition technology. The method first provides a neural network search space template that is friendly to binary neural networks in image recognition tasks and proposes a non-decreasing channel number strategy to crop the search space. Next, a corresponding binary neural network supernet is generated based on the search space. This supernet contains numerous candidate binary subnets, and any subnet can obtain corresponding parameters from the supernet through weight inheritance. Next, three components are proposed: a binary teacher strategy, a binary conversion module, and a weight normalization module to optimize the training of the binary neural network supernet and enhance the accuracy of the final binary neural network obtained through the search in tasks such as image recognition. Finally, using an arbitrary search algorithm, the optimal binary neural network under corresponding constraints is obtained on the binary neural network supernet. This invention can meet image recognition needs in various deployment environments.

Description

Technical Field

[0001] The present invention belongs to the field of object recognition technology, relates to computer vision and deep learning technology, and in particular to a method for constructing a binary neural network based on neural network architecture search. Background Art

[0002] Binary Neural Networks (BNNs) use 1-bit weights and activation parameters. Compared to conventional full-precision networks, convolutional computations in BNNs are implemented using exclusive-or (XNOR) and bit-count operations, enabling at least 58x acceleration over conventional matrix multiplication on CPUs or embedded devices. Thanks to their rapid inference speed, BNNs have been widely applied to various computer vision tasks, including image classification, object detection, and instance segmentation, and have become a new paradigm for image recognition networks.

[0003] However, since strong expert knowledge is required when constructing binary neural networks for image recognition tasks (such as the additional introduction of Bi-Real modules in the neural network, etc.), and its training strategy is also significantly different from that of conventional neural networks (such as the need for two-stage training, etc.), manually constructing binary neural networks relies on a large amount of manpower and computing resources. Although some researchers have tried to introduce naive neural network architecture search into the process of building binary neural networks, they are limited by factors such as the small search space and simple search strategy, and have not been able to achieve breakthrough results. Overall, the existing methods of constructing binary neural networks have a high threshold for application in image recognition tasks. There are bottlenecks such as a heavy reliance on manpower, a huge demand for resources, and a lack of flexibility. It is impossible to quickly and automatically construct efficient binary neural networks for various image recognition tasks. Summary of the Invention

[0004] In order to overcome the shortcomings of the above-mentioned existing technologies, the present invention provides a method for constructing a binary neural network based on neural network architecture search, which can quickly obtain a binary neural network with optimal accuracy under given hardware constraints (such as theoretical computational load, etc.) to meet image recognition requirements in various deployment environments.

[0005] The technical solution of the present invention is:

[0006] A method for constructing a binary neural network based on neural network architecture search. First, a neural network search space template that is friendly to binary neural networks in image recognition tasks is given, and a non-decreasing channel number strategy is proposed to crop the search space. Then, based on the search space, a corresponding binary neural network supernet is generated. The supernet contains many candidate binary subnets (i.e., a certain binary neural network). Any subnet can obtain corresponding parameters from the supernet through weight inheritance. Then, three major components are proposed: a binary teacher strategy, a binary conversion module, and a weight normalization module to optimize the training of the binary neural network supernet and enhance the accuracy of the final binary neural network obtained by the search in tasks such as image recognition. Finally, an arbitrary search algorithm is used to obtain the optimal binary neural network under the corresponding restrictions on the binary neural network supernet. Specifically, the following steps are included:

[0007] 1) Provide a neural network search space template that is friendly to binary neural networks in image recognition tasks;

[0008] 2) Using the non-decreasing number of channels strategy to trim the neural network search space;

[0009] Assume that A is the search space before pruning, a∈A represents a subnet in the search space, L a is the total number of layers in the subnet, Represents the number of feature channels of the subnet at layer i, then the final pruned search space Expressed as

[0010]

[0011] 3) Based on search space Generate a corresponding binary neural network supernet; the supernet contains many candidate binary subnets, and any subnet obtains corresponding parameters from the binary neural network supernet through weight inheritance;

[0012] 4) Using the three components of binary teacher strategy, binary conversion module, and weight normalization module, the above binary neural network supernet is trained;

[0013] 5) Using any search algorithm, obtain the optimal binary neural network under the corresponding constraints on the binary neural network supernet.

[0014] The above-mentioned method of constructing a binary neural network based on neural network architecture search can be widely used in various image recognition tasks, and then applied to various application scenarios such as intelligent security, autonomous driving, and drone surveys, so as to quickly realize the effective deployment and application of the same model in various hardware requirements, reducing the consumption of manpower, material resources, and financial resources.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] The method provided by the present invention is a method for constructing a binary neural network based on neural network architecture search. Based on given hardware constraints (such as theoretical computing power, etc.), it can quickly obtain a binary neural network with optimal accuracy under corresponding constraints, thereby meeting image recognition requirements in various deployment environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of a specific embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present invention is further described below by way of examples, which however do not limit the scope of the present invention in any way.

[0019] This paper proposes a method for constructing a binary neural network based on a neural network architecture search algorithm. The overall process is as follows: Figure 1 The following is an introduction to the application of the present invention to an image classification task (ImageNet dataset) as a specific example.

[0020] 1) Utilize the neural network search space template that is friendly to binary neural networks in image recognition tasks.

[0021]

[0022] Table 1. Neural network search space templates that are friendly to binary neural networks

[0023] In order to enhance the versatility of the present invention, a neural network search space template that is friendly to binary neural networks in image recognition tasks is provided, as shown in Table 1. Input represents the resolution of the input feature map of the corresponding layer, Operator represents the category of the corresponding layer (including conventional convolution layer Conv2d, separable convolution layer Separable Conv2d, uniform pooling layer Avgpool2d, linear mapping layer Linear), Depth represents the number of candidate stackings of the corresponding layer, Channel width represents the number of candidate feature channels of the corresponding layer, Kernel size represents the size of the candidate convolution kernel of the corresponding layer, Groups represents the number of candidate convolution groups of the corresponding layer, and Stride represents the convolution step size of the corresponding layer. In this example, the search space template is directly used as the specific search space.

[0024] 2) Use the non-decreasing number of channels strategy to crop the search space.

[0025] Assume that A is the search space before pruning, a∈A represents a subnet in the search space, L a is the total number of layers in the subnet, Represents the number of feature channels of the subnet at layer i, then the final pruned search space It can be expressed as

[0026]

[0027] The above steps eliminate subnetworks in the search space with decreasing number of feature channels, effectively reducing the size of the search space and helping to accelerate the convergence of the binary neural network and the subsequent search stage.

[0028] 3) Based on search space Generate the corresponding binary neural network supernet.

[0029] Using the binary neural network architecture search space, a binary neural network supernet is directly generated, i.e., the largest binary neural network architecture that can be represented by the search space is constructed. Specifically, the three candidate operations (Depth, Channel width, and Kernel size) will be selected with the maximum value for each layer (since the larger the values ​​of these three operations, the larger the neural network), while the candidate operation (Groups) will be selected with the minimum value for each layer (since the smaller the value of this operation, the larger the neural network). In this example, the first layer of the binary neural network supernet has an operator of Conv2d, a depth of 1, a channel width of 48, a kernel size of 3, groups of 1, and a stride of 2; the second layer has an operator of Separable Conv2d, a depth of 3, a channel width of 96, a kernel size of 3, groups of 1, and a stride of 1; the third layer has an operator of Separable Conv2d, a depth of 3, a channel width of 192, a kernel size of 5, groups of 1, and a stride of 2; the fourth layer has an operator of Separable Conv2d, a depth of 3, a channel width of 384, a kernel size of 5, groups of 2, and a stride of 2; the fifth layer has an operator of Separable Conv2d, a depth of 9, a channel width of 768, a kernel size of 3 The size is 5, the Groups is 4, and the Stride is 2; the sixth layer Operator is SeparableConv2d, the Depth is 3, the Channel width is 1536, the Kernel size is 5, the Groups is 8, and the Stride is 2; the seventh layer Operator is Avgpool2d, the Depth is 1; the eighth layer Operator is Linear, the Depth is 1, and the Channel width is the number of target categories in the ImageNet dataset, which is 1000.

[0030] 4) Use the three major components of binary teacher strategy, binary conversion module, and weight normalization module to train the binary neural network supernet constructed above.

[0031] Binary teacher strategy. In the training phase, the present invention selects four binary neural network subnets for training in each iteration, namely the largest subnet, the smallest subnet and two random subnets. In the largest subnet, this solution chooses to retain its weights as full precision rather than binary, so as to accelerate the convergence of the overall weights through full-precision training. At the same time, the present invention uses the output information of the largest subnet as a supervisory signal to supervise the training of the remaining three subnets (the smallest subnet and the two random subnets). The overall loss function can be expressed as:

[0032]

[0033] Among them L CE and L KL Denote cross entropy loss and KL divergence loss respectively, Y is the true value label, is the predicted value of the largest subnet, y s and are the predicted values ​​of the smallest subnetwork and the ith random subnetwork.

[0034] Binary conversion module. During the training phase, the shared weights of the binary neural network supernet need to support the full-precision weights of the largest subnet, as well as the binary weights of the smallest subnet and two random subnets. In order to avoid the conflict between the gradients of the two numerical domains, the present invention introduces a binary conversion module. Assume that the shared weight W r The convolution kernel size is k, and the weight corresponding to the binary domain is W b , then the calculation formula of the binary conversion module is:

[0035] W b =ReShape k×k (Flatten(W r )·θ)

[0036] in It is the conversion matrix of the binary conversion module, which is initialized as a diagonal matrix. Reshape is a matrix shape setting operation, which configures the input matrix to the corresponding size and shape. Flatten is a matrix expansion operation, which flattens the input matrix into a one-dimensional vector.

[0037] Weight normalization module. Before the shared weights are binarized, the present invention proposes a weight normalization module to maximize the information entropy of the binary weights. Assume that the input weight is W and its mean is μ W , the standard deviation is σ W , then the weight after the weight normalization module is for:

[0038]

[0039] 4) Using any search algorithm, obtain the optimal binary neural network under the corresponding constraints on the binary neural network supernet.

[0040] Based on given constraints (such as theoretical computational effort), the proposed solution can apply any search algorithm to obtain the optimal binary neural network within the corresponding constraints on a binary neural network supernet. Specifically, the accuracy of each binary neural network subnet is obtained by directly inheriting the parameters of the binary neural network supernet and then testing it. Using this accuracy as a metric, this example employs a search algorithm based on an evolutionary algorithm. Specifically, N subnet populations that meet the constraints are randomly initialized and their accuracy is evaluated. Next, individuals in this population are subjected to "mutation" (i.e., random adjustment of a configuration) and "crossover" (i.e., random combination of two configurations) through random sampling to obtain the next generation of populations and evaluate them. Finally, the "mutation" and "crossover" processes are repeated to obtain the final population, and the binary neural network subnet with the highest accuracy within this population is output as the search result.

[0041] 5) Using an additional fine-tuning phase, the binary neural network obtained by the above search is trained again.

[0042] After searching for the optimal binary neural network subnet, its parameters can be directly obtained by inheriting the binary neural network supernet and directly applied to the image recognition task. This paper proposes an additional fine-tuning stage to further improve its accuracy by retraining the binary neural network on the corresponding image recognition task. The specific loss formula is:

[0043] L=L CE (y, Y).

[0044] Among them L CE represents the cross entropy loss, Y is the true value label, and y is the predicted value output by the binary neural network.

[0045] Compared with the existing binary neural network, the method of the present invention can obtain a series of binary neural networks with a computational load of 20M-200M on the ImageNet image classification dataset, and has significant advantages in a fair comparison of computational load. Specifically, the minimum binary neural network model obtained by the present invention achieved a top-1 accuracy of 61.15% with only 21M computing power, while the benchmark model Binary MobileNet-M3 had a computing power of 33M but only achieved a top-1 accuracy of 51.06%; the binary neural network with a theoretical computing power of 50M obtained by the present invention achieved a top-1 accuracy of 67.95%, while the benchmark model MobileNet-Mid had a computing power of 52M but only achieved a top-1 accuracy of 54.40%; the binary neural network with a theoretical computing power of 85M obtained by the present invention achieved a top-1 accuracy of 69.13%, while the benchmark model BATS had a computing power of 99M but only achieved a top-1 accuracy of 60.40%; the binary neural network with a theoretical computing power of 119M obtained by the present invention achieved a top-1 accuracy of 69.84%, while the benchmark model BCNN P=1 has a computational load of 131M but only achieves a top-1 accuracy of 69.00%; the binary neural network with a theoretical computational load of 162M obtained by the present invention achieves a top-1 accuracy of 70.19%, while the benchmark model ReActNet-B has a computational load of 163M but only achieves a top-1 accuracy of 70.10%.

[0046] It should be noted that the purpose of disclosing the embodiments is to facilitate a further understanding of the present invention. However, those skilled in the art will appreciate that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be determined by the scope defined in the claims.

Claims

1. A method for constructing a binary neural network based on neural network architecture search, comprising the following steps: 1) Provide a neural network search space template that is friendly to binary neural networks in image recognition tasks; 2) Using the non-decreasing number of channels strategy to trim the neural network search space; Assume that A is the search space before pruning, a∈A represents a subnet in the search space, L a is the total number of layers in the subnet, Represents the number of feature channels of the subnet at layer i, then the final pruned search space Expressed as 3) Based on search space Generate a corresponding binary neural network supernet, which contains many candidate binary subnets. Any subnet obtains corresponding parameters from the binary neural network supernet through weight inheritance. 4) Using the three components of binary teacher strategy, binary conversion module, and weight normalization module, the above binary neural network supernet is trained; 5) Using any search algorithm, obtain the optimal binary neural network under the corresponding constraints on the binary neural network supernet.

2. The method for constructing a binary neural network based on neural network architecture search according to claim 1, characterized in that: In step 3), the three candidate operations of Depth, Channel width, and Kernel size will select the maximum value for each layer; the candidate operation of Groups will select the minimum value for each layer.

3. The method for constructing a binary neural network based on neural network architecture search according to claim 1, characterized in that: The binary teacher strategy in step 4) specifically includes: in the training phase, four binary subnets are selected for training in each iteration, namely the largest subnet, the smallest subnet, and two random subnets. In the largest subnet, its weight is retained as full precision. At the same time, the output information of the largest subnet is used as a supervisory signal to supervise the training of the remaining three subnets. The overall loss function is expressed as: Among them L CE and L KL Denote cross entropy loss and KL divergence loss respectively, Y is the true value label, is the predicted value of the largest subnet, y s and are the predicted values ​​of the smallest subnetwork and the ith random subnetwork.

4. The method for constructing a binary neural network based on neural network architecture search according to claim 1, characterized in that: The binary conversion module in step 4) is used to realize the shared weight binarization, that is, while supporting the full-precision weight of the largest subnet, it also supports the binary weights of the smallest subnet and two random subnets. The specific shared weight binarization calculation formula is: W b =Resnapr k×k (Flat(Wr).θ). The shared weight W r The convolution kernel size is k, and the weight corresponding to the binary domain is W b , then the binary conversion module It is the conversion matrix of the binary conversion module, which is initialized as a diagonal matrix. Reshape is a matrix shape setting operation, which configures the input matrix to the corresponding size and shape. Flatten is a matrix expansion operation, which flattens the input matrix into a one-dimensional vector.

5. The method for constructing a binary neural network based on neural network architecture search according to claim 1, characterized in that: The weight normalization module in step 4) is used to obtain the information entropy of the maximized binary weight before the shared weight is binarized, that is, assuming that the input weight is W and its mean is μ W , the standard deviation is σ W , then the weight after the weight normalization module is for:

6. The method for constructing a binary neural network based on neural network architecture search according to claim 1, characterized in that: Step 5) An evolutionary algorithm is used as the basic search algorithm. The specific steps include: first, randomly initializing N subnet populations that meet the constraint requirements and evaluating their accuracy; then, "mutating" and "crossing" the individuals in the population by random sampling to obtain the next generation population and evaluate it; finally, repeatedly performing the "mutation" and "crossing" processes to obtain the final population, and outputting the binary neural network subnet with the highest accuracy in the population as the search result.

7. The method for constructing a binary neural network based on neural network architecture search according to claim 1, wherein: Step 5) After obtaining the optimal binary neural network, the accuracy of the binary neural network is further improved by retraining it on the corresponding image recognition task. The specific loss formula is: L=L CE (y,Y). Among them L CE represents the cross entropy loss, Y is the true value label, and y is the predicted value output by the binary neural network.

Images