Neural architecture search device, neural architecture search method, and program

Abstract

Provided is a time-efficient neural architecture search for backbone blocks of computer vision tasks. The neural architecture search device includes construction means (11) for constructing a super network. The target layer of the super network to be optimized is replaced with a plurality of candidate layers, and the super network includes a plurality of fully-connected layers. The neural architecture search device includes training means (12) for training the super network, wherein the plurality of candidate layers are trained in parts, and the plurality of fully-connected layers are trained corresponding to the parts of the plurality of candidate layers. The neural architecture search device includes selection means (13) for evaluating the trained super network and selecting a part corresponding to the part with the best performance among the plurality of fully-connected layers among the plurality of candidate layers.

Description

[Technical Field]

[0001] The present application relates to a neural architecture search device, a neural architecture search method, and a program. [Background technology]

[0002] Over the past few decades, convolutional neural network (CNN) models have become the state-of-the-art solution for computer vision tasks such as image classification, object detection, and semantic segmentation. The success of CNN models is primarily due to their ability to achieve high accuracy. In real-time applications, the time it takes for a CNN model to run, commonly referred to as its execution time, is also crucial.

[0003] CNN models that achieve high accuracy tend to have several CNN layers, while CNN models that achieve high speed (i.e., short execution time) tend to have fewer CNN layers. Therefore, there is a trade-off between accuracy and speed with respect to the number of CNN layers employed in a CNN model. Furthermore, there are several hyperparameters associated with CNN layers, such as kernel size, input channels, and output channels. Manually optimizing each hyperparameter for each layer is a time-consuming task and requires a lot of human expertise.

[0004] Recently, an efficient technique for such problems, called neural architecture search (NAS), has been developed. The NAS technique generally involves three steps. First, a network consisting of several candidate CNN layers is constructed, as shown in Figure 2. A large network with multiple candidate CNN layers is called a SuperNet. In the first step of NAS, the SuperNet is trained on a dataset. Then, in the second step, the SuperNet is intelligently pruned to produce a smaller network with fewer CNN layers, with the goal of minimizing accuracy degradation. The smaller network with fewer CNN layers is called a SubNet. Finally, in the third step, the SubNet is further trained on the dataset to recover accuracy.

[0005] A CNN model for object detection tasks consists of three main blocks: a backbone block, a neck block, and a head block. The main task of the backbone block is to extract shallow-level features from the input image, while the neck block extracts deeper features. The head block predicts labels based on the features extracted by the backbone and neck blocks. The NAS method can be applied to one or more blocks. NAS reduces the need for human expertise in designing CNN models. However, a concern with the NAS method is the time required for training the supernet and for searching and training the optimal subnet. To address this concern, Non-Patent Document 1 introduces special architecture parameters that are also trained during supernet training. Using these special architecture parameters allows for rapid pruning of the supernet, which speeds up the second step and accelerates the NAS method. However, the time required to train a large supernet is very long, resulting in a significant delay in obtaining the final subnet. [Prior art documents] [Non-patent literature]

[0006] Fast Neural Network Adaptation via Parameter Remapping and Architecture Search, Jiemin Fang, Yuzhu Sun, Kangjian Peng, Qian Zhang, Yuan Li, Wenyu Liu, Xinggang Wang, https: / / arxiv.org / abs / 2001.02525 Summary of the Invention [Problem to be solved by the invention]

[0007] The challenge of NAS is how to train a supernet model. A supernet model consists of multiple layers, and each layer has several candidates for the convolutional layer, which increases the size of the CNN model. Training such a large CNN model takes a long time.

[0008] Another challenge is the large gap between the structures of supernets and subnetworks: in supernets, multiple candidates in each layer are trained under the condition that there are multiple parallel layers before and after them, while in subnetworks, there are only one or fewer CNN layers before and after them.

[0009] One embodiment of the present invention has been made in view of these problems, and its purpose is to provide a time-efficient neural architecture search for the backbone blocks of computer vision tasks. [Means for solving the problem]

[0010] In order to achieve the above-mentioned object, the neural architecture search device of the present invention comprises: a construction means for constructing a supernetwork, wherein a target layer of optimization of the supernetwork is replaced by a plurality of candidate layers, and the supernetwork includes a plurality of fully-connected layers; a training means for training the supernetwork, wherein the plurality of candidate layers are trained in portions, and the plurality of fully-connected layers are trained corresponding to the portions of the plurality of candidate layers; and a selection means for evaluating the trained supernetwork and selecting a portion of the plurality of candidate layers corresponding to the best-performing portion of the plurality of fully-connected layers.

[0011] To achieve the above-mentioned object, the neural architecture search method of the present invention includes the steps of: constructing a supernetwork, in which a target layer of optimization of the supernetwork is replaced with a plurality of candidate layers, and the supernetwork includes a plurality of fully-connected layers; training the supernetwork, in which the plurality of candidate layers are trained in portions, and the plurality of fully-connected layers are trained corresponding to the portions of the plurality of candidate layers; and evaluating the trained supernetwork and selecting a portion of the plurality of candidate layers corresponding to the best-performing portion of the plurality of fully-connected layers.

[0012] In order to achieve the above object, a program causes a computer to function as a neural architecture search device, and the program causes the computer to function as a construction means, a training means, and a selection means. [Effects of the Invention]

[0013] Exemplary aspects of the present invention may provide a time-efficient neural architecture search for the backbone blocks of computer vision tasks. [Brief explanation of the drawings]

[0014] [Figure 1]FIG. 1 is a block diagram showing the configuration of a neural architecture search device according to the first embodiment. [Figure 2] FIG. 2 is a flowchart showing the flow of the neural architecture search method according to the first embodiment. [Figure 3] FIG. 3 is a flowchart showing the flow of the neural architecture search method according to the first embodiment. [Figure 4] FIG. 4 is a block diagram showing the configuration of a CNN model training system based on neural architecture search according to the second embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of a Supernet CNN model with a candidate search space. [Figure 6] FIG. 6 is a diagram showing a Supernet CNN model constructed by the Supernet Builder 300 according to the second embodiment. [Figure 7] FIG. 7 is a block diagram showing the internal structure of an FC block according to the second embodiment. [Figure 8] FIG. 8 is a flowchart showing the flow of the neural architecture search process executed by the system according to the second embodiment. [Figure 9] FIG. 9 is a flowchart for explaining the processing executed by the super network builder 300 according to the second embodiment. [Figure 10] FIG. 10 is a flowchart illustrating the process executed by the STOC according to the second embodiment. [Figure 11] FIG. 11 is a diagram illustrating the current training in step S403 of FIG. [Figure 12] FIG. 12 is a diagram illustrating the current training in step S405 of FIG. [Figure 13] FIG. 13 is another diagram showing the current training performed in step S403 of FIG. [Figure 14] FIG. 14 is another diagram showing the current training performed in step S404 of FIG. [Figure 15]FIG. 15 is a flowchart illustrating the processing executed by the neural architecture selector according to the second embodiment. [Figure 16] FIG. 16 is a diagram showing a Supernet CNN model constructed by the Supernet Builder according to the third embodiment. [Figure 17] FIG. 17 is a flowchart for explaining the processing executed by the super network builder 300 according to the third embodiment. [Figure 18] FIG. 18 is a flowchart illustrating the processing executed by the neural architecture selector 500 according to the third embodiment. [Figure 19] FIG. 19 is a diagram showing a Supernet CNN model constructed by the Supernet Builder according to the fourth embodiment. [Figure 20] FIG. 20 is a flowchart for explaining the processing executed by the super network builder according to the fourth embodiment. [Figure 21] FIG. 21 is a block diagram illustrating a hardware configuration according to an exemplary embodiment. DETAILED DESCRIPTION OF THE INVENTION First embodiment

[0015] In the following description, details of a first embodiment of the present invention will be described with reference to the drawings. Note that the first embodiment is the basis for the following embodiments.

[0016] In the first embodiment, a neural architecture search device and a neural architecture search method will be described with reference to FIGS.

[0017] (Configuration of a neural architecture search device) The configuration of a neural architecture search device 1 according to the first embodiment will be described below with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the neural architecture search device 1. As shown in Fig. 1, the neural network architecture search device 1 includes a construction unit 11, a training unit 12, and a selection unit 13.

[0018] The neural architecture search device 1 trains a large-scale network (called a supernetwork) that includes multiple candidate network layers that are candidates for optimization. After the training and optimization processes, the neural architecture search device 1 outputs a pruned network (also called a subnetwork), which is a network smaller than the supernetwork. The neural network search device 1 may also train a subnetwork.

[0019] The feature construction unit 11 is an example of construction means described in the claims. The training unit 12 is an example of training means described in the claims. The selection unit 13 is an example of selection means described in the claims.

[0020] The construction unit 11 constructs a super-network. Here, the super-network is a neural network including multiple candidate network layers, as described above. The candidate network layers are layers that are optimized in the optimization process by the neural architecture search device 1.

[0021] Figure 2 is a schematic diagram showing the configuration of a super-network SN. As shown in Figure 2, the super-network SN includes one or more target layers (TL) and one or more non-target layers (NTL1, NTL2, ...). As shown in Figure 2, the optimized target layer of the super-network SN is replaced with multiple candidate layers (CL1, CL2, CL3, ...).

[0022] 2, the super-network SN includes multiple fully connected layers (FCL1, FCL2, FCL3, ...), each of which may correspond to a candidate layer (CL1, CL2, CL3, ...).

[0023] When the super-network SN is trained, each of the candidate layers (CL1, CL2, CL3, ...) included in the target layer (TL) is trained together with the corresponding fully connected layer (FCL1, FCL2, FCL3, ...).

[0024] That is, the training unit 12 trains a super network, where multiple candidate layers (CL1, CL2, CL3, ...) are trained for each part, and multiple fully connected layers (FCL1, FCL2, FCL3, ...) are trained corresponding to the parts of the multiple candidate layers.

[0025] When the super-network SN is trained, the loss for the super-network SN is evaluated using a predefined loss function. Candidate layers (CL1, CL2, CL3, ...) and fully connected layers (FCL1, FCL2, FCL3, ...) are selected with reference to the evaluation.

[0026] That is, the selection unit 13 evaluates the trained super-network and selects, from among the multiple candidate layers (CL1, CL2, CL3, ...), the part that corresponds to the best-performing part of the multiple fully connected layers (FCL1, FCL2, FCL3, ...).

[0027] (Neural architecture search method flow) The flow of the neural architecture search method according to the first embodiment will be described below with reference to Fig. 3. Fig. 3 is a flowchart showing the flow of the neural architecture search method S1. As shown in Fig. 3, the flow of the neural architecture search method includes steps S11 to S13.

[0028] In step S11, a supernetwork SN is constructed by the construction unit 11 of the neural architecture search device 1. The supernetwork is a neural network including multiple candidate network layers. The candidate network layers are layers that are optimized in the optimization process by the neural network architecture search device 1. That is, the neural architecture search method S1 includes a step of constructing a supernetwork, in which the optimization target layer of the supernetwork is replaced with multiple candidate layers (CL1, CL2, CL3, ...), and the supernetwork includes multiple fully connected layers (FCL1, FCL2, FCL3, ...).

[0029] In step S12, the training unit 12 trains the super-network as described above. That is, the neural architecture search method S1 includes a super-network training step in which a plurality of candidate layers (CL1, CL2, CL3, ...) are trained for each part, and a plurality of fully connected layers (FCL1, FCL2, FCL3, ...) are trained in correspondence with the parts of the plurality of candidate layers (CL1, CL2, CL3, ...).

[0030] In step S13, the selection unit 13 evaluates the trained super-network as described above and selects a portion of the plurality of candidate layers (CL1, CL2, CL3, ...). That is, the neural architecture search means S1 includes a step of evaluating the trained super-network and selecting a portion of the plurality of candidate layers that corresponds to the best-performing portion of the plurality of fully connected layers.

[0031] (Advantageous Effects of the First Embodiment) As described above, according to the first embodiment, when training a supernetwork, multiple candidate layers are trained separately, and multiple fully connected layers are trained corresponding to the portions of the multiple candidate layers. This allows for training in a shorter time than training all layers simultaneously. This makes the training time of the supernet very efficient. In this way, a time-efficient neural architecture search for the backbone block of a computer vision task can be realized.

[0032] <Second embodiment> In the following description, the second embodiment of the present invention will be described in detail with reference to the drawings.

[0033] (Construction of a model training system based on neural architecture search) Hereinafter, the configuration of a neural architecture search-based CNN model training system 100 according to the second embodiment will be described with reference to FIG. 4. FIG. 4 is a block diagram showing the configuration of the neural architecture search-based model training system 100. The CNN model is a supernetwork. Hereinafter, the "supernetwork" may also be referred to as a "SuperNet." In the second embodiment, the neural architecture search-based model trained by the neural architecture search-based model training system 100 is used for at least one of an object detection task and an object classification task.

[0034] As shown in FIG. 4 , the model based on neural architecture search includes a training dataset 200 for object detection task, a supernet builder 300 , a supernet trainer 400 with object detection task and classification task, and a neural architecture selector 500 .

[0035] The training dataset 200 for the object detection task is a dataset provided for training the CNN model of the neural architecture search based model training system 100 used in the object detection task. The training dataset for the object detection task 200 includes images and labels. The images are the input, and the labels are the predicted values ​​that the supernet and subnet CNN models are intended to generate as output.

[0036] The SuperNet Builder 300 corresponds to the construction unit 11 in the first embodiment. The SuperNet Trainer 400 with object detection task and classification task corresponds to the training unit 12 in the first embodiment. The neural architecture selection unit 500 corresponds to the selection unit 13 in the first embodiment.

[0037] As described in the first embodiment, the Supernet Trainer 400 with object detection and classification tasks trains a supernet. Here, multiple candidate layers are trained in parts, and multiple fully connected layers are trained corresponding to the parts of the multiple candidate layers. More specifically, this embodiment describes the case where multiple candidate layers and corresponding fully connected layers are trained one by one.

[0038] The neural architecture search based model training system 100 further comprises a dataset transformer 600, a training dataset 700 for a classification task, and an optimized CNN model 800.

[0039] A training dataset 700 for a classification task includes images and labels. The images are the inputs, and the labels are the categories of objects present in each image. The labels are the predicted values ​​that a supernet under training based on the classification task is intended to produce as output.

[0040] The optimized CNN model 800 is obtained by training the supernet trainer 400 with object detection and classification tasks and selecting the neural architecture selector 500 .

[0041] (Data Transformer 600) The dataset transformer 600 is a functional block that functions as a conversion means for converting a training dataset for an object detection task into a training dataset for a classification task. The training dataset 700 for a classification task is a dataset for training a CNN model of the neural architecture search-based CNN model training system 100 used for the classification task.

[0042] The dataset transformer 600 comprises means for receiving an object detection dataset, means for transforming the object detection dataset into a classification dataset, and means for providing the classification dataset as an output. The means required to transform an object detection dataset into a classification dataset is a simple engineering task and will not be described herein.

[0043] Thus, the neural architecture search based model training system 100 includes a transformation means for transforming the object detection dataset into a classification dataset.

[0044] (SuperNet CNN model and SuperNet Builder 300) Figure 5 shows a schematic diagram of an example of a Supernet CNN model with a candidate search space. The Supernet CNN model 900a includes a backbone 901a, a neck 902a, and a head 903a. The details of the backbone 901a are illustrated as blocks (blocks 904a, 905a, ... 907a). For example, the backbone 901a includes N blocks.

[0045] Each block may be formed by various types of neural architectures such as Conv 3x3, SW 3x3, MAX, Skip, etc. In FIG. 5, only the details of block 905a are illustrated, but other blocks may also have this detailed structure. Conventionally, the training of a CNN model has been performed using various types of neural architectures for each block (blocks 904a, 905a,... 907a). However, such training is time-consuming and inefficient. In this embodiment, candidates for replacing each block are prepared.

[0046] FIG. 6 is a block diagram showing a supernet CNN model constructed by a supernet builder 300. The supernet builder 300 includes means for receiving a training dataset of an object detection task 200 and a supernet, and means for constructing the supernet. When the system 100 is executing an iteration other than the first iteration, it receives the supernet from a neural architecture selector 500. Then, the supernet builder 300 uses the pre-constructed supernet and modifies it. When the system 100 performs the first iteration, the supernet builder 300 constructs a supernet from scratch as shown in FIG. 6.

[0047] In FIG. 6, an image 2001 is input into the supernet CNN model 900.

[0048] F CNN layers are arranged in series and are called a fixed layer 904.

[0049] N CNN layers are arranged in series at the output of the last fixed layer 904. These N CNN layers in series are called B i In FIG. 6, i is an index where 0 < i ≤ N. In this case, layers from B1 to B N layers are arranged in series. At the output of the B N layer, M parallel FC (fully connected layer) blocks 9081 to 9083 of the same structure are arranged, and the construction of the backbone 901 of the supernet is completed.

[0050] FIG. 7 is a block diagram showing the internal structure of the FC block 9080. As shown in FIG. 7, the FC blocks 9081 to 9083 are basically a series arrangement of one or a number of fully-connected layers.

[0051] Finally, B N To the output of the B layer, as shown in FIG. 6, the neck 902 and the head 903 are arranged. The neck 902 and the head 903 are designed according to the requirements of the object detection task. Therefore, the output of the head 903 is provided to the object detection output 2002. On the other hand, the outputs of the FC blocks 9081 to 9083 are provided to the classification outputs 9091 to 9093, respectively.

[0052] In FIG. 6, the image 2001 is input to the super network CNN model 900. The image 2001 is first input to the fixed layer 904 of the backbone 901. The fixed layer 904 is connected to a plurality of sub-layers (B1 1 , B1 2 ,..., B1 M ). The plurality of sub-layers (B1 1 , B1 2 ,..., B1 M ) correspond to the plurality of candidate layers (CL1, CL2, CL3,...) in the first embodiment.

[0053] First, start with the optimization of the B1 layer. Replace the B1 layer with M parallel CNN layers also called sub-layers arranged as shown in FIG. 6. These M parallel CNN sub-layers are given as B i j (0 < j ≦ M). The M sub-layers are basically several variations of the CNN layer with the hyperparameters changed. One of the M sub-layers is selected as the winner during the selection executed by the neural architecture selector 500, and the remaining M - 1 sub-layers drop out.

[0054] As shown in Figure 6, the output from the final fixed layer 904 is input to all M parallel sub-layers. All M parallel outputs are combined by operations such as, but not limited to, concatenation and sum, and the output is provided to the B2 layer. N The layer has only one CNN layer for each iteration to train each of the M parallel sublayers. Training is described in more detail below.

[0055] The layer for optimization is also called the target layer. In this case, the B1 layer is the target layer. The target layer is the layer from B1 to B N As mentioned above, B N The output of the layer is connected to M parallel FC (fully-connected layer) blocks 9081 to 9083. In this way, multiple fully connected layers are connected to the output of the target layer or a layer deeper than the target layer.

[0056] B N The layers are connected to a neck 902, which is connected to a head 903. The output of the head 903 is shown as object detection output 2002. Thus, for object detection, the neck 902 and head 903, and the N layers of the backbone 901 are used for prediction.

[0057] Also, B N The layers are connected to M parallel FC blocks 9081 to 9083. The output of FC1 block 9081 is shown as classification output 9091. Similarly, the outputs of FC2 block and FC3 block 9081 are shown as classification output 9092 and classification output 9093. Thus, for object classification, the N layers of the FC blocks and backbone 901 are used for prediction.

[0058] (SuperNetTrainer 400 with object detection and classification tasks) The supernet trainer 400 with object detection task and classification task, also referred to as STOC400, comprises means for receiving the training data set of the object detection task 200, the training data set of the classification task 200, and the training data set 700 of the classification task, and means for receiving the supernet from the supernet builder 300. STOC400 also comprises means for executing the training of the supernet for the object detection task and the classification task. Finally, STOC400 also comprises means for outputting the trained supernet output.

[0059] The basic function of STOC40 is to train all M sub-layers of layer B together with other B k (0 < k ≤ N; i ≠ k;) layers, and to train all FC blocks 9080 in the classification task. i The training of all sub-layers of layer B is performed one by one. Similarly, the training of all FC blocks is performed one by one. Also, the training of sub-layers and FC blocks is performed alternately. That is, first, sub-layer 9051 is trained in the object detection task, and then FC1 block 9081 is trained in the classification task. Next, sub-layer 9052 of layer B is trained in the object detection task, and then FC2 block 9082 is trained in the classification task. Thus, STOC400 trains the supernetwork for at least one of the object detection task using the object detection data set and the classification task using the classification data set. i The training of all sub-layers of layer B is performed one by one. Similarly, the training of all FC blocks is performed one by one. Also, the training of sub-layers and FC blocks is performed alternately. That is, first, sub-layer 9051 is trained in the object detection task, and then FC1 block 9081 is trained in the classification task. Next, sub-layer 9052 of layer B is trained in the object detection task, and then FC2 block 9082 is trained in the classification task. Thus, STOC400 trains the supernetwork for at least one of the object detection task using the object detection data set and the classification task using the classification data set. i 1 sub-layer 9051 is trained in the object detection task, and then FC1 block 9081 is trained in the classification task. Next, i [[ID=​​​​​​​​

[0061] In this embodiment, the neural architecture search-based CNN model training system 100 performs a NAS (Neural Architecture Search) technique on the backbone 901. However, the neural architecture search-based CNN model training system 100 can be easily extended to the neck 902 and head 903 without much effort.

[0062] Hereinafter, the CNN model training system 100 based on neural architecture search will also be simply referred to as the system 100.

[0063] (Flow of neural architecture search process executed by system 100) FIG. 8 is a flowchart showing the flow of the neural architecture search process executed by the system 100 in the second embodiment.

[0064] In step S101, the system 100 builds a supernet using the supernet builder 300. The processing executed by the supernet builder 300 will be described later.

[0065] In step S102, system 100 trains a supernet using Supernet Trainer with Object Detection and Classification Tasks 400. The process performed by Supernet Trainer with Object Detection and Classification Tasks 400 is described in detail below.

[0066] In step S103, the system 100 performs candidate selection using the neural architecture selector 500. This step selects one sub-layer from the M parallel sub-layers, thereby optimizing the target layer. The process performed by the neural architecture selector 500 is described below.

[0067] In step S104, system 100 determines whether all N layers of the supernet have been optimized or covered. If not, the value of parameter i and the pruned supernet from neural architecture selector 500 are provided as input to supernet builder 300 for the next iteration. In this manner, system 100 performs the next iteration.

[0068] The processing of steps S101 to S104 is repeatedly executed until it is determined in S104 that all of the N layers of the supernet have been covered.

[0069] If it is determined in step S104 that all N layers of the supernet have been covered, the process proceeds to step S105, in which the system 100 outputs the optimized CNN model 800.

[0070] That is, the system 100 performs N such iterations to cover all N layers of the supernet. In each iteration, a target layer is set. For example, in the first iteration, the target layer is B1. In the second iteration, the target layer is B2. Finally, the target layer is B N At the end of the Nth iteration, where N is the final pruned supernet from the neural architecture selector 500, the final pruned supernet is given as output.

[0071] If desired, the pruned supernet may be trained for one or more epochs to improve accuracy using the STOC 400's means for training the supernet on the object detection task. The final pruned supernet, also called the optimal subnet or optimal CNN model, is the output of the system 100.

[0072] During supernet optimization, only one layer is targeted at a time, allowing for faster search compared to simultaneous optimization of all layers. Also, the target layer B at a particular iteration iExcept for the layer one before and the layer one after, all the other N - 2 layers including the sub - layer of B i are all trained with one layer as input and one layer as output. This is the architecture of a general output sub - network or output - optimized CNN model.

[0073] (Process by the super - network builder 300) When the system 100 is performing the first iteration of super - network (super - network) construction, the super - network is constructed from scratch as shown in FIG. 6. When the system 100 is performing an iteration other than the first iteration, it receives the super - network from the neural architecture selector 500. Then, the super - network builder 300 uses the pre - constructed super - network to modify it.

[0074] FIG. 9 is a flowchart for explaining the process executed by the super - network builder 300.

[0075] In step S301, the super - network builder 300 determines whether the current process is being executed as an initial super - network construction. The initial super - network construction corresponds to the first iteration. If the super - network builder 300 determines that the current process is being executed as an initial super - network construction, the process of step S302 is executed.

[0076] In step S302, the super - network builder 300 sets the value of parameter i to "1".

[0077] In step S303, the super - network builder 300 constructs a super - network having a backbone 901, a neck 902, and a head 903. Here, the backbone 901 has F fixed layers, N serial layers, M parallel sub - layers in the i - th layer (0 < i ≤ N), and M FC blocks connected to the output of the N - th layer. That is, the super - network builder 300 constructs a super - network CNN model as described with reference to FIG. 6.

[0078] In step S304, the Super Net Builder 300 initializes the weights for all layers and sub-layers.

[0079] There are several options for weight initialization, such as random initialization, parameter remapping, Xavier initialization, etc. The weight initialization task can be performed in STOC 400 as needed.

[0080] The constructed supernet with initialized weights is output in step S308 of FIG.

[0081] If, in step S301, the Supernet Builder 300 determines that the current process is not to be performed as an initial supernet construction, step S305 is executed, in which case the system 100 performs a second or subsequent iteration of the construction of the supernet.

[0082] In step S305, the Supernet Builder 300 receives a pruned supernet and a value for parameter i from the neural architecture selector 500. The supernet from the previous iteration is also called the pruned supernet. The pruned supernet is the output of the neural architecture selector 500. The supernet received from the neural architecture selector 500 has one layer for every N serial layers. The Supernet Builder 300 also receives from the neural architecture selector 500 the next B i Information about the layer is received as the value of the parameter i.

[0083] In step S306, the Supernet Builder 300 replaces the i-th layer with M parallel sublayers and connects M new FC blocks to the output of the N-th layer of the pruned supernet. i The layer is replaced with M parallel sublayers, and the remaining N-1 layers are kept as they are.

[0084] In step S307, the supernet builder 300 performs weight initialization only for the newly added sublayers in the i-th layer. B i Performs weight initialization for all M newly added sublayers arranged in the layer.

[0085] The modified supernet is given as the output of the supernet builder 300 in step S308 of FIG. 9. The output supernet is provided to the STOC400.

[0086] In this way, the processing of the supernet builder 300 is executed.

[0087] (Processing executed by the supernet trainer 400 with object detection task and classification task) FIG. 10 is a flowchart for explaining the processing executed by the STOC400. <000042​​​​​​​​​​​​​​​​​​​​​​​Only the sub-layer 9051 participates. In other words, among the M sub-layers of the B1 layer, only B1 1 sub-layer 9051 participates with the other B of the backbone 901 k (0 < k ≤ N; i ≠ k;) layers, the neck 902, and the head 903. The remaining M - 1 sub-layers of B1 do not participate during training.

[0092] Figure 11 shows the current training in step S403 of FIG. 10 when it is first executed. In this case, during the forward propagation in the training phase based on the object detection task of the supernet model in the first iteration, the sub-layer B1 1 [[ID=IO]]is shown. In FIG. 11, thick curved arrows are drawn to identify the blocks used in the current training. In FIG. 11, the thick curved arrows pass through the fixed layer 904, the B1 1 sub-layer 9051, the B2 layer 906 to the B N layer 907, the neck 902, the head 903, and the object detection output 2002.

[0093] In step S404, STOC400 freezes the weights of the entire supernet, while STOC400 unfreezes the weights of the FC1 block 9081. Then, the supernet is trained on the classification task. During training, among the M sub-layers of the B1 layer, the B1 1 sublayer 9051, the FC1 block 9081, and the other B k (0 < k ≤ N; i ≠ k;) layers only participate.

[0094] In step S405, STOC400 trains the supernet on the classification task. Here, the supernet freezes only the B i layer from the M sub-layers of the B i j and freezes the B k (0 < k ≤ N; i ≠ k;) layers, and unfreezes the FC j block.

[0095] 12 shows the current training in step S405 of FIG. 10, which is executed for the first time. In this case, the sublayer B1 of the forward propagation in the training phase based on the classification task of the FC block in the first iteration 1 In FIG. 12, thick curved arrows are drawn to identify the blocks used in the current training. In FIG. 12, thick curved arrows indicate the blocks in the fixed layer 904, B1 1 Sublayer 9051, B2 layer 906 to B N It passes through layer 907, FC1 block 9081, and classification output 9091. In this case, only the weights of FC1 block 9081 are updated during training.

[0096] In this way, the training described with reference to Figures 11 and 12 is performed for each sub-layer.

[0097] In step S406, STOC 400 determines whether all sub-layers of the i-th layer have been covered or whether the value of parameter j is equal to M. If STOC 400 determines that all sub-layers of the i-th layer have not been covered and the value of parameter j is not equal to M, step S407 is executed. In step S407, STOC 400 increments the value of parameter j by "1". Then, the processes of steps S402 to S406 are executed again.

[0098] 13 is a diagram showing the current training performed for the second time in step S403 of FIG. 10. In this case, the sublayer B1 of the forward propagation in the training phase based on the object detection task of the Supernet model in the first iteration 2 In FIG. 13, thick curved arrows are drawn to identify the blocks used in the current training. In FIG. 13, thick curved arrows indicate the fixed layer 904, B1 2 Sublayer 9052, B2 layer 906 to B N It passes through layer 907 , neck 902 , head 9903 and object detection output 2002 .

[0099] 14 shows the current training performed for the second time in step S404 of FIG. 10. In this case, the sublayer B1 of the forward propagation in the training phase based on the classification task of the FC block in the first iteration 2 In FIG. 14, thick curved arrows are drawn to identify the blocks used in the current training. In FIG. 14, thick curved arrows indicate the blocks in the fixed layer 904, B1 2 Sublayer 9052, B2 layer 906 to B N It passes through layer 907, FC2 block 9082, and classification output 9092. In this case, only the weights of FC2 block 9081 are updated during training.

[0100] The process of steps S402 to S406 is repeated until the STOC 400 determines that all sublayers of the i-th layer have been covered or the value of the parameter j is equal to M. That is, for an object detection task, B i The process of training the supernet by involving one sublayer from the layer and one of the corresponding FC blocks for classification tasks is repeated M-1 times for the other M-1 sublayers and M-1 FC blocks.

[0101] In this way, in the training process by STOC400, multiple candidate layers are trained one by one, and multiple fully connected layers are trained corresponding to each of the multiple candidate layers.

[0102] After M iterations of training the supernet on the object detection task and the classification task, the trained supernet and all FC blocks are output in step S408, completing step S102 of Figure 3, which was performed to train the supernet partly on the object detection task and partly on the classification task.

[0103] The trained supernet is input to the neural architecture selector 500, which performs the sub-layer selection described as step S103 in FIG.

[0104] In this way, the processing of STOC400 is executed.

[0105] (Processing executed by neural architecture selector 500) FIG. 15 is a flowchart for explaining the processing executed by neural architecture selector 500.

[0106] In step S501, neural architecture selector 500 receives the trained supernet from STOC400. Also, neural architecture selector 500 receives the training dataset of classification task 700 from dataset transformer 600.

[0107] In step S502, neural architecture selector 500 selects, as the winner of the i-th layer, the sublayer in which the corresponding FC block has the minimum loss / maximum accuracy in the i-th layer.

[0108] First, using the output of FC1 block 9081 and the ground truth label from the training dataset of classification task 700, the loss is evaluated using a pre-defined loss function. During the evaluation of the loss in output FC1 block 9081, the other B k (0 < k ≦ N; i ≠ k;) layers together with FC1 block 9081 and only the B i 1 sub-layers participate.

[0109] The definition of the loss function may vary depending on the purpose of NAS. If the only purpose is to achieve high accuracy, the definition of the loss function may consist only of the classification loss as represented by the following equation (1).

[0110]

Equation

[0111]

number

[0112] In this way, the neural architecture selector 500 selects one candidate layer from among the multiple candidate layers that corresponds to the best performing one of the multiple fully connected layers.

[0113] The more efficient the feature extraction of a sub-layer, the more likely the corresponding FC block can accurately classify the input image. Therefore, compared with the other M-1 FC block sub-layers, the sub-layer corresponding to the best-performing FC block can be reliably said to be the best choice among the M sub-layers.

[0114] In step S503, the neural architecture selector 500 keeps only the winning sublayer and removes the remaining M-1 sublayers from the supernet. The remaining M-1 sublayers are pruned, so that at the end of step S503, layer B1 is optimized. The pruned supernet, with a backbone 901, one layer out of all N layers, plus another fixed layer 904, and a neck 902 and head 903, is provided as output.

[0115] In step S504, the neural architecture selector 500 determines whether or not the search has been completed for all layers in the supernet. If the neural architecture selector 500 determines that the search has not been completed for all layers in the supernet, the process proceeds to step S505.

[0116] In step S505, the neural architecture selector 500 outputs the current supernet, which is the pruned supernet. That is, the neural architecture selector 500 includes an output means for outputting the pruned supernet as a result of the selection process of the neural architecture selector 500.

[0117] By the processing of step S505, the neural architecture selector 500 completes the candidate selection process for the B1 layer in the backbone 901, as shown in step 3, S103, of Figure 8. The neural architecture selector 500 also increments the value of the parameter i to optimize the next layer. The output supernet is provided to the supernet builder 300. The processing of step S305 is performed on the pruned supernet.

[0118] If the neural architecture selector 500 determines in step S504 that the search for all layers of the supernet has been completed, the process proceeds to step S506. In step S506, the neural architecture selector 500 outputs the current supernet. Since the search for all layers in the supernet has been completed, it is determined in step S104 of FIG. 8 that all N layers in the supernet have been covered. Then, in step S105 of FIG. 8, the supernet output from the neural architecture selector 500 is output as an optimized CNN model.

[0119] In this manner, the processing of the neural architecture selector 500 is performed.

[0120] (Advantageous Effects of the Second Embodiment) According to the second embodiment, only one target layer is optimized at a time. Compared with optimizing all layers simultaneously, the search for the neural architecture can be performed in a short time. Therefore, the time efficiency of training the supernet is very good.

[0121] Also, except for the layer before and the layer after the target layer B in a specific iteration, the other N - 2 layers including the sub - layer B i are all trained with one layer as the input and one layer as the output. This is the architecture of a general output sub - net or the optimal CNN model to be output. Therefore, by optimizing one layer in one iteration, the gap in the architecture between the supernet and the sub - net can be significantly reduced. i Furthermore, the advantage of such a narrow gap may also shorten the training time of the sub - net, thereby further shortening the training time required by the NAS - based CNN model training system 100.

[0122] <The Third Embodiment>

[0123] <The third embodiment will be described below with reference to the drawings. In the following description, only the differences between the system 100 and the neural architecture search process according to the second embodiment and the system 100 and the neural architecture search process according to the third embodiment will be described.

[0124] Elements having the same functions as those described in the second embodiment are given the same reference numerals, and their descriptions are omitted as appropriate.

[0125] In the second embodiment, the optimization of the Bi layer is performed in the order of the B1 layer, the B2 layer, ···, the B N layer. However, in order to select the target layer, the order may be randomly changed (random traverse) so that 0 < i ≤ N.​

[0126] In this embodiment, when performing the first iteration, B i Any layer of the layer can be optimized. The only constraint is that in N iterations, all B i The thing is that you have to cover the layers.

[0127] Fig. 16 shows the Supernet CNN model constructed by the Supernet Builder 300 in this embodiment. The Supernet CNN model shown in Fig. 16 is constructed in the first iteration of step S101 in Fig. 8. In this case, the random traverse is written as "N", which means that the initial value i of the parameter is "N".

[0128] In the example shown in Figure 16, the optimization of backbone 901 is B N It starts from the layer B. N layer with M parallel CNN layers (sublayers). Therefore, B N In the layer, multiple sub-layers (B N 1 , B N 2 , ..., B N M ) is shown. M FC blocks 9081 to 9083 are connected to sub-layers 9071 to 9073, respectively.

[0129] On the other hand, in the second embodiment, as shown in FIG. 6, first, the B1 layer is replaced by M parallel CNN layers (sub-layers).

[0130] FIG. 17 is a flowchart for explaining the processing executed by the super network builder 300 in this embodiment.

[0131] In step S301 of Figure 17, the Supernet Builder 300 determines whether the current process is to be executed as an initial supernet construction. The initial supernet construction corresponds to the first iteration. When the Supernet Builder 300 determines that the current process is to be executed as an initial supernet construction, the process of step S302 is executed.

[0132] In step S302 of FIG. 17, the Super Network Builder 300 sets the value of the parameter i to "N".

[0133] On the other hand, in the second embodiment, the super network builder 300 sets the value of the parameter i to "1" as shown in step S302 of FIG.

[0134] FIG. 18 is a flowchart for explaining the processing executed by the neural architecture selector 500 in this embodiment.

[0135] 18, the neural architecture selector 500 determines whether or not the search has been completed for all layers in the supernet. If the neural architecture selector 500 determines that the search has not been completed for all layers in the supernet, the process proceeds to step S505.

[0136] 18, the neural architecture selector 500 outputs the current supernet, which is the pruned supernet. By the processing of step S505, the neural architecture selector 500 selects the current B in the backbone 901. i The layer candidate selection procedure is completed as shown in step 3, S103, of Figure 8. The output supernet is provided to the Supernet Builder 300. The process of S305 is then performed using the pruned supernet.

[0137] The neural architecture selector 500 also sets the value of parameter i to optimize the next layer, which is also selected by random traversal. Suppose the system 100 performs an initial iteration in which the value of parameter i is set to "N," and then in step S505 of FIG. 18, a value between 1 and N-1 is randomly selected.

[0138] On the other hand, in the second embodiment, the super network builder 300 sets the value of the parameter i to "i+1" as shown in step S505 of FIG.

[0139] (Advantageous Effects of the Second Embodiment) As explained above, target group B i By randomly selecting, the degree of freedom of optimization can be increased.

[0140] <Fourth embodiment> The fourth embodiment will be described below with reference to the drawings. Note that the following description will focus only on the differences between the system 100 and neural architecture search process according to the second and third embodiments and the system 100 and neural architecture search process according to the fourth embodiment.

[0141] It should be noted that elements having the same functions as those described in the second and third embodiments are given the same reference numerals, and their description will be omitted as appropriate.

[0142] In the second and third embodiments, the M FC blocks 9081 to 9083 are N However, the M FC blocks are connected to the output of layer 907. later In other words, in this embodiment, the M FC blocks are connected to the output of the B i It may be connected to either layer.

[0143] FIG. 19 shows the supernet CNN model constructed by the supernet builder 300 in this embodiment. The supernet CNN model shown in FIG. 19 is constructed in the first iteration of step S101 in FIG. 8, and the target layer is layer B1.

[0144] Therefore, a plurality of sub-layers (B1 1 、B1 2 、...、B1 M ) are illustrated. The M FC blocks 9081 to 9083 are respectively connected to the sub-layers 9051 to 9053. In this case, the M FC blocks are respectively connected to the output of the target layer. This means that the value of the parameter later is equal to the value of the parameter i.

[0145] On the other hand, in the second and third embodiments, as shown in FIGS. 6 and 16, the M FC blocks are respectively connected to the output of layer B N regardless of the layer currently selected as the target layer.

[0146] FIG. 20 is a flowchart for explaining the process executed by the supernet builder 300 according to this embodiment.

[0147] In step S303 of FIG. 20, the supernet builder 300 constructs a supernet having a backbone 901, a neck 902, and a head 903. Here, the backbone 901 has F fixed layers, N serial layers, M parallel sub-layers in the i-th layer (0 < i ≤ N), and M FC blocks connected to the output of the later layer (i ≤ later).

[0148] On the other hand, in the second and third embodiments, as shown in step 303 of FIG. 9 and step 303 of FIG. 17, the supernet builder 300 constructs a supernet having a backbone 901, a neck 902, and a head 903, where the backbone 901 has F fixed layers, N serial layers, M parallel sub-layers in the i-th layer (0 < i ≤ N), and M FC blocks connected at the output of the N-th layer.

[0149] (Advantageous Effects of the Fourth Embodiment) As described above, by connecting the M FC blocks to any of the B layers equal to or deeper than the target layer, the degree of freedom in optimization can be improved. i

[0150] (Configuration Example by Software) One or more of all the functions of the system 100 can be realized by hardware such as an integrated circuit (IC chip) or, as an alternative, by software.

[0151] In the latter case, the system 100 is realized by a computer that executes program instructions of a program, which is software that realizes the aforementioned functions. FIG. 21 shows an example of such a computer (hereinafter referred to as "computer C"). Computer C includes at least one processor C1 and at least one memory C2. In memory C2, a program P for causing computer C to function as system 100 is stored. In computer C, the functions of system 100 are realized when processor C1 reads and executes program P from memory C2.

[0152] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0153] The computer C may further include a random access memory (RAM) into which the program P is loaded when the program P is executed and in which various data is temporarily stored. The computer C may further include a communication interface for transmitting and receiving data to and from other devices. The computer C may further include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0154] The program P can be stored in a non-transitory tangible storage medium M that can be read by the computer C. Examples of the storage medium M include a tape, a disk, a card, a semiconductor memory, and a programmable logic circuit. The computer C can acquire the program P through the storage medium M. The program P can be transmitted through a transmission medium. Examples of the transmission medium include a communication network and broadcast waves. The computer C can also acquire the program P through such a transmission medium.

[0155] [Appendix 1] The present invention is not limited to the above-described embodiments, and various modifications may be made by those skilled in the art within the scope of the claims. For example, the present invention also encompasses, within its technical scope, any embodiment derived by appropriately combining the technical means disclosed in the above-described embodiments.

[0156] [Appendix 2] All or part of the embodiments disclosed above can be described as follows. However, it should be noted that the present invention is not limited to the aspects of the following embodiments.

[0157] [supplementary explanation] Aspects of the present invention can also be expressed as follows: (Aspect 1) construction means for constructing a super-network, wherein a target layer of optimization of the super-network is replaced by a plurality of candidate layers, and the super-network includes a plurality of fully connected layers; training means for training the super-network, wherein the plurality of candidate layers are trained portion-by-portion, and the plurality of fully-connected layers are trained corresponding to the portions of the plurality of candidate layers; and selection means for evaluating the trained super-network and selecting a portion of the plurality of candidate layers that corresponds to the best performing portion of the plurality of fully connected layers; A neural architecture search device comprising:

[0158] (Aspect 2) In the training process by the training means, the plurality of candidate layers are trained one by one, and the plurality of fully connected layers are trained corresponding to the one of the plurality of candidate layers; 2. The neural architecture search device according to claim 1, wherein the selection means selects one of the plurality of candidate layers corresponding to the best-performing layer of the plurality of fully connected layers.

[0159] (Aspect 3) The neural architecture search device according to aspect 1, wherein the plurality of fully connected layers are connected to the output of the target layer or a layer deeper than the target layer.

[0160] (Aspect 4) The training means includes: Object detection tasks using object detection datasets, and a classification task using a classification dataset.

[0161] (Aspect 5) 5. The neural architecture search apparatus according to aspect 4, further comprising a conversion means for converting the object detection dataset into the classification dataset.

[0162] (Aspect 6) The super network includes a backbone block, a neck block, and a head block; The backbone block includes a plurality of serially arranged CNN layers and the plurality of fully connected layers; 2. The neural architecture search apparatus according to claim 1, wherein the target layer is selected from the plurality of serially arranged CNN layers.

[0163] (Aspect 7) The neural architecture search device according to aspect 1 further comprises an output means for outputting the super-network pruned by the selection process of the selection means.

[0164] (Aspect 8) constructing a super-network, replacing a target layer of optimization of the super-network with multiple candidate layers, the super-network including multiple fully connected layers; training the super-network by training the plurality of candidate layers in portions and training the plurality of fully connected layers corresponding to the portions of the plurality of candidate layers; and evaluating the trained super-network and selecting a portion of the plurality of candidate layers that corresponds to the best-performing portion of the plurality of fully connected layers; A neural architecture search method comprising:

[0165] (Aspect 9) A program for causing a computer to function as the neural architecture search device of claim 1, the program causing the computer to function as the construction means, the training means, and the selection means. [Explanation of symbols]

[0166] 1 Neural Architecture Search Device 11 Construction Department 12 Training Department 13 Selection section 100 Model Training System Based on Neural Architecture Search 200 training datasets for object detection tasks 300 Super Net Builder SuperNet Trainer with 400 Object Detection and Classification Tasks 500 Neural Architecture Selector 600 Dataset Transformers 700 training datasets for classification tasks 800 optimized CNN models 900 Supernet 901 Backbone 902 Neck 903 Head

Claims

1. construction means for constructing a super-network, wherein a target layer of optimization of the super-network is replaced by a plurality of candidate layers, and the super-network includes a plurality of fully connected layers; training means for training the super-network, wherein the plurality of candidate layers are trained portion-by-portion, and the plurality of fully-connected layers are trained corresponding to the portions of the plurality of candidate layers; and a selection means for evaluating the trained super-network and selecting one candidate layer from the plurality of candidate layers corresponding to a fully connected layer determined to be the best based on a predefined performance evaluation function; A neural architecture search device comprising:

2. In the training process by the training means, the plurality of candidate layers are trained one by one, and the plurality of fully connected layers are trained corresponding to the one of the plurality of candidate layers; the selection means selects one candidate layer from the plurality of candidate layers, the candidate layer corresponding to a fully connected layer determined to be the best based on a predefined performance evaluation function.

2. The neural architecture search apparatus according to claim 1.

3. The neural architecture search device according to claim 1 , wherein the plurality of fully connected layers are connected to the output of the target layer or a layer deeper than the target layer.

4. The training means includes: Object detection tasks using object detection datasets, and 10. The neural architecture search apparatus of claim 1, wherein the super-network is trained for at least one of: a classification task using a classification dataset;

5. The neural architecture search apparatus of claim 4 , further comprising a conversion means for converting the object detection dataset into the classification dataset.

6. The super network includes a backbone block, a neck block, and a head block; The backbone block includes a plurality of serially arranged CNN layers and the plurality of fully connected layers; The neural architecture search device according to claim 1 , wherein the target layer is selected from the plurality of serially arranged CNN layers.

7. 2. The neural architecture search apparatus according to claim 1, further comprising: an output means for outputting the super-network pruned by the selection process of said selection means.

8. The performance evaluation function is defined by a classification loss.

2. The neural architecture search apparatus according to claim 1.

9. A computer comprising: constructing a super-network, replacing a target layer of optimization of the super-network with multiple candidate layers, the super-network including multiple fully connected layers; training the super-network by training the plurality of candidate layers in portions and training the plurality of fully connected layers corresponding to the portions of the plurality of candidate layers; and evaluating the trained super-network and selecting one candidate layer from the plurality of candidate layers corresponding to the best fully connected layer determined based on a predefined performance evaluation function; A neural architecture search method including:

10. 2. A program for causing a computer to function as the neural architecture search device of claim 1, the program causing the computer to function as the construction means, the training means, and the selection means.

Images