Automatic machine learning development system, automatic machine learning development method, and program

Abstract

[Solution] The automated machine learning development system includes a neural architecture search block that outputs a trained small CNN model including the trained α parameter values ​​and a trained derived backbone of the large CNN model; an α analyzer block that outputs a reduced candidate search space including a reduced number of candidate sublayers to a model compression block; a model compression block that compresses the trained small CNN model into a lightweight CNN model by replacing sublayers within each layer of the trained derived backbone of the smaller CNN model with the remaining candidate sublayers in the reduced candidate search space; and a model deployment block that deploys the lightweight CNN model on target hardware.

Description

[Technical Field]

[0001] The present invention relates to an automated machine learning development system, an automated machine learning development method, and a program. [Background technology]

[0002] In recent years, convolutional neural network (CNN) models have emerged as the state-of-the-art solution for high-precision computer vision tasks. In real-time systems, CNN models are not only designed to achieve high accuracy, but also compressed to achieve real-time speed.

[0003] Designing a CNN model to achieve high accuracy requires optimizing several hyperparameters. At the same time, designing a CNN model involves several candidates for each layer. Manually selecting the optimal candidate for each layer and optimizing the hyperparameters is time-consuming and requires human expertise.

[0004] In recent years, a time-efficient technique, namely, Neural Architecture Search (NAS), has been developed. There are several types of NAS techniques that have been developed, such as one-shot NAS, Bayesian Optimization NAS, and reinforcement learning-based NAS. The main goal of NAS is to find an optimal CNN model from a large search space. Since exploring the entire search space is computationally very challenging, NAS techniques attempt to efficiently find a CNN model with the desired accuracy.

[0005] A generated CNN model with a desired accuracy generally does not have sufficient execution time. Therefore, the generated model is compressed to have similar accuracy and reduced execution time. During compression, each layer of the CNN model is replaced with a lightweight layer, assuming that the accuracy is roughly maintained. However, for each layer of the CNN model, there may be several lightweight layers. It is time-consuming to exhaustively try all combinations of lightweight candidate layers for the entire CNN model.

[0006] There are several documents related to designing neural network architecture models: Patent document (PTL) 1 relates to an information processing method that makes it possible to find an inference model that suppresses losses due to quantization.

[0007] Patent Document 2 relates to complex model scaling for neural networks to determine neural network architecture.

[0008] Patent Document 3 relates to a method for deriving information about hyperparameters related to learning model performance.

[0009] Patent Document 4 relates to a method for searching for a neural network structure model. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] WO2022 / 044465A1 [Patent Document 2] Special Publication No. 2022-523666 [Patent Document 3] Japanese Patent Publication No. 2022-065599 [Patent Document 4] Patent Publication No. 2021-064342 Summary of the Invention [Problem to be solved by the invention]

[0011] The following analysis is given by the inventor.

[0012] The first problem with automated machine learning is that it takes time to compress a model. The first reason is that for each layer of a CNN model, there are several lightweight candidate layers. Trying each candidate exhaustively takes time.

[0013] Therefore, one of the aims of the present invention is to contribute to making the overall automated machine learning development system faster. [Means for solving the problem]

[0014] The automated machine learning development system of the present invention uses an alpha analyzer block that analyzes training information during high-precision CNN model generation, thereby removing some candidates from the model compression search space. With a reduced number of candidate layers, model compression can be performed faster.

[0015] According to a first aspect of the present invention, there is provided an automated machine learning development system including a neural architecture search block, an alpha analyzer block, a model compression block, and a model deployment block, The neural architecture search block includes: -receive the dataset, -Build large CNN models, The large CNN model includes at least a backbone, a neck, and a head; the backbone includes a series of connected layers, each of which includes a plurality of candidate sublayers connected in parallel; an α parameter is assigned to the output of each of the candidate sublayers of the backbone, and the output of each of the candidate sublayers is multiplied by a respective trained α parameter value; - training the backbone of the large CNN model by training weights and α parameter values ​​of all candidate sublayers of the backbone of the large CNN model using the dataset; - selecting a best candidate sublayer for each of the layers based on the highest trained α parameter value, and constructing a small CNN model including at least a derived backbone including the selected best candidate sublayer for each of the layers, the neck, and the head; training the derived backbone of the small CNN model with the dataset; - outputting a trained small CNN model including the trained α parameter value and the trained derived backbone of the large CNN model; The alpha analyzer block receiving trained α parameter values ​​from the neural architecture search block; - ranking all the candidate sublayers of each of the layers based on the trained α parameter values ​​and comparing the trained α parameter values ​​with the sizes of the candidate sublayers; - removing weak candidate sublayers within each of the layers from the candidate search space, the candidate sublayers having a size larger than the candidate sublayer with the highest α parameter value or having an α parameter value smaller than the highest α parameter value by a predetermined threshold, thereby reducing the candidate search space for the model compression; - outputting a reduced candidate search space comprising a reduced number of said candidate sublayers to said model compression block, The model compression block receiving the trained small CNN model from the neural architecture search block and the reduced candidate search space from the alpha analyzer block; compressing the trained small CNN model into a lightweight CNN model by replacing sublayers in each layer of the trained derived backbone of the smaller CNN model with the remaining candidate sublayers in the reduced candidate search space; The model deployment block includes: Deploying the lightweight CNN model on target hardware. We can provide an automated machine learning development system.

[0016] According to a second aspect of the present invention, a computer receiving a dataset; - Building a large CNN model, The large CNN model includes at least a backbone, a neck, and a head; the backbone includes a series of connected layers, each of which includes a plurality of candidate sublayers connected in parallel; an α parameter is assigned to the output of each of the candidate sublayers of the backbone, and the output of each of the candidate sublayers is multiplied by a respective trained α parameter value; The computer - training the backbone of the large CNN model by training weights and α parameter values ​​of all candidate sublayers of the backbone of the large CNN model using the dataset; - selecting a best candidate sublayer for each of the layers based on the highest trained α parameter value, and constructing a small CNN model including at least a derived backbone including the selected best candidate sublayer for each of the layers, the neck, and the head; - training the derived backbone of the small CNN model with the dataset; - outputting a trained small CNN model including the trained α parameter value of the large CNN model and a trained derived backbone; The computer receiving a trained α parameter value; - ranking all the candidate sublayers of each of the layers based on the trained α parameter value and comparing the trained α parameter value with the size of the candidate sublayers; - removing weak candidate sublayers within each of the layers from the candidate search space, the candidate sublayers having a size larger than the candidate sublayer with the highest α parameter value or having an α parameter value smaller than the highest α parameter value by a predetermined threshold, thereby reducing the candidate search space for the model compression; - outputting a reduced candidate search space comprising a reduced number of said candidate sublayers, The computer - receiving the trained small CNN model from the neural architecture search block and the reduced candidate search space from the alpha analyzer block; - compressing the trained small CNN model into a lightweight CNN model by replacing sublayers in each layer of the trained derived backbone of the smaller CNN model with the remaining candidate sublayers in the reduced candidate search space; The computer - deploying the lightweight CNN model on target hardware; The present invention provides an automated machine learning development method, which includes the steps of: (a) generating a machine learning model based on a plurality of inputs;

[0017] According to a third aspect of the present invention, a computer is provided with: receiving a data set; -Run the process of building a large CNN model, The large CNN model includes at least a backbone, a neck, and a head; the backbone includes a series of connected layers, each of which includes a plurality of candidate sublayers connected in parallel; an α parameter is assigned to the output of each of the candidate sublayers of the backbone, and the output of each of the candidate sublayers is multiplied by a respective trained α parameter value; The computer, - training the backbone of the large CNN model by using the dataset to train weights and α parameter values ​​of all candidate sublayers of the backbone of the large CNN model; - selecting a best candidate sublayer for each of the layers based on the highest trained α parameter value, and constructing a small CNN model including a derived backbone including at least the selected best candidate sublayer for each of the layers, the neck, and the head; - training the derived backbone of the small CNN model using the dataset; - executing a process of outputting a trained small CNN model including the trained α parameter value and the trained derived backbone of the large CNN model; The computer, receiving trained α parameter values; - ranking all the candidate sublayers of each of the layers based on the trained α parameter value and comparing the trained α parameter value with the size of the candidate sublayers; - removing weak candidate sublayers in each of the layers from the candidate search space, the candidate sublayers having a size larger than the candidate sublayer with the highest α parameter value or having an α parameter value smaller than the highest α parameter value by a predetermined threshold, thereby reducing the candidate search space for the model compression; - performing a process that outputs a reduced candidate search space that includes a reduced number of said candidate sublayers; The computer, - receiving the trained small CNN model from the neural architecture search block and the reduced candidate search space from the alpha analyzer block; - compressing the trained small CNN model into a lightweight CNN model by replacing sublayers in each layer of the trained derived backbone of the smaller CNN model with the remaining candidate sublayers in the reduced candidate search space; The computer, Deploying the lightweight CNN model on target hardware; It is possible to provide a computer program that causes the above-mentioned program to be executed. Note that these programs can be recorded on a computer-readable storage medium. The storage medium can be a non-transitory medium such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. The present invention can also be embodied as a computer program product. [Effects of the Invention]

[0018] The present invention allows for a faster overall automated machine learning development system. [Brief explanation of the drawings]

[0019] [Figure 1] 1 shows an example of a block diagram of an automated machine learning development system 100 according to an embodiment of the present invention. [Figure 2] 1 shows a flowchart of an automated machine learning development system 100 according to an embodiment of the present invention. [Figure 3] 2 shows a flow chart of the neural architecture search block 200 of one embodiment of the present invention. [Figure 4] 3 shows a flow chart of the alpha analyzer block 300 of one embodiment of the present invention. [Figure 5] 4 shows a flowchart of the model compression block 400 of one embodiment of the present invention. [Figure 6]1 shows an example of a supernet architecture constructed by the neural architecture search block 200 of one embodiment of the present invention. [Figure 7] 2 shows an example block diagram of a subnet 2001 architecture constructed by a neural architecture search block 200 of one embodiment of the present invention. [Figure 8] 2 shows an example block diagram of a subnet 2001B architecture compressed by the model compression block 400 of one embodiment of the present invention. [Figure 9] 3 shows an example of a tabular representation of the ranking of candidate sublayers made by the alpha analyzer block 300 of one embodiment of the present invention. [Figure 10] 1 shows an example of a table representation of a reduced candidate search space where some candidate sublayers have been removed from the candidate search space input to the model compression block 400 by the alpha analyzer block 300 of one embodiment of the present invention. [Figure 11] 1 shows the configuration of a computer that constitutes an automated machine learning development system of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0020] [One embodiment] An automated machine learning development system according to an embodiment of the present invention will be described in detail with reference to the drawings.

[0021] Note that the reference numerals in the drawings attached to this summary are used for convenience to assist in understanding and are not intended to limit the present invention. Furthermore, connecting lines between blocks in the drawings and the like include both bidirectional and unidirectional lines. Unidirectional arrows are used to schematically indicate the flow of the main signal (data) and do not exclude bidirectionality. Furthermore, although not shown, input and output ports exist at the input and output terminals of each connecting line in the circuit diagrams, block diagrams, internal configuration diagrams, connection diagrams, and the like shown in this disclosure. The same applies to input / output interfaces.

[0022] 1 shows an example of a block diagram of an automated machine learning development system 100 according to an embodiment of the present invention. The automated machine learning development system 100 includes at least a dataset 101 used in an object detection task, a neural architecture search block 200, an alpha analyzer block 300, a model compression block 400, and a model deployment block 500.

[0023] 2 shows a flowchart of the automated machine learning development system 100 according to one embodiment of the present invention. The flowchart starts at step S101. Next, the neural architecture search block 200 executes the process of step S102. Next, the alpha analyzer block 300 executes the process of step S103. Next, the model compression block 400 executes the process of step S104. Finally, the model deployment block 500 executes the process of step S105. The flowchart then ends at step S106.

[0024] The dataset 101 for the object detection task includes images and labels. The images are input to the supernet 2000 and subnet 2001, and the labels are the predictions that the supernet 2000 and subnet 2001 are intended to produce as output.

[0025] The operation of the neural architecture search block 200 will now be described.

[0026] First, a high-precision CNN model is searched by the neural architecture search block 200.

[0027] The neural architecture search block 200 receives the dataset 101, constructs a large CNN model called the supernet 2000 that includes at least a backbone, a neck, and a head, trains the backbone of the supernet 2000, extracts and constructs a small CNN model called the subnet 2001 that includes the backbone, the neck, and the head, trains the backbone of the subnet 2001, and outputs the trained α parameter value of the supernet 2000 and the subnet 2001 model. FIG. 3 shows a flowchart of the neural architecture search block 200 according to an embodiment of the present invention. FIG. 6 shows an example of a supernet architecture constructed by the neural architecture search block 200 according to an embodiment of the present invention. FIG. 7 shows an example of a block diagram of the subnet 2001 architecture constructed by the neural architecture search block 200 according to an embodiment of the present invention. The flowchart starts at step S201.

[0028] As shown in step S202 of FIG. 3, first, the neural architecture search block 200 constructs a large CNN model called the supernet 2000 from scratch. In the construction of the supernet 2000, first, F CNN layers called fixed layers are arranged continuously. Next, M candidate sublayers (also called candidates) are arranged in parallel and, at the output of the last fixed layer (fixed block), form one large layer. As shown in FIG. 6, layers 211 to 21N each include M candidate sublayers arranged in parallel and are arranged continuously N times. As shown in FIG. 6, the indexing of the CNN layers is given by B i j where 0 < i ≦ n and 0 < j ≦ M. All layers B iInstead of having a fixed number of M candidate sub - layers, it is also possible to have a variable number of candidate sub - layers. Successively, a plurality of such sub - layers connected in parallel is the backbone (also called the backbone part) 201 of the super - net 2000. At the output of each candidate sub - layer of the backbone 201, each α - parameter value is assigned. As shown in FIG. 6, the indexing of the α - parameter values is given by α i j where \(0 < i\leq n\) and \(0 < j\leq M\). During forward propagation, the output of each candidate layer is multiplied by the \(i\) - th respective α - parameter value. The parameter values of each layer may be normalized such that the sum of the α - parameter values within each layer is 1.

[0029] As shown in FIG. 6, at the output of the last layer 21N of the backbone 201, the neck 202 layer and the head 203 layer are successively connected. The neck 202 layer and the head 203 layer are designed according to the requirements of the object detection task.

[0030] After the super - net 2000 is constructed, the neural architecture search block 200 efficiently trains the weights of all candidate sub - layers of the backbone 201 of the super - net 2000 and the α - parameter values of the backbone 201 of the super - net 2000 by using the dataset 101, as shown in step S203 of FIG. 3, thereby training the backbone 201 of the super - net 2000.

[0031] After completing training of the backbone 201 of the supernet 2000, among the M candidate sublayers in each of the layers 211 to 21N of the backbone 201, only one candidate sublayer (best candidate sublayer) having the highest trained α parameter value (hereinafter also referred to as the α parameter value of the winning sublayer) is selected as the winning sublayer for each of the layers 211 to 21N, and the rest are discarded by the neural architecture search block 200. Thus, as shown in FIG. 7, only one candidate sublayer (best candidate sublayer) remains in each of the layers 211 to 21N, forming the derived backbone 2011. As shown in step S204 of FIG. 3, the neural architecture search block 200 constructs a new small CNN model called subnetwork 2001 using the derived backbone 2011 and the existing neck 202 and head 203 layers. The neural architecture search block 200 then trains the derived backbone 2011 of the subnet 2001 using the dataset 101, as shown in step S205 of Figure 3. After training the backbone 2011 of the subnet 2001, the neural architecture search block 200 outputs the trained α parameter value from training the backbone 201 of the supernet 2000 and the trained subnet 2001, including the trained derived backbone 2011. The flowchart ends at step S206.

[0032] Next, the operation of the α analyzer block 300 will be described below.

[0033] The alpha analyzer block 300 receives the trained alpha parameter values ​​from the neural architecture search block, performs the analysis, and outputs a reduced number of candidates to the model compression block 400.

[0034] Figure 4 shows a flowchart of the alpha analyzer block 300 of one embodiment of the present invention. Figure 9 shows an example of a table representation of the ranking of candidate sublayers made by the alpha analyzer block 300 of one embodiment of the present invention. Figure 10 shows an example of a table representation of a reduced candidate search space, where some candidate sublayers have been removed from the candidate search space, input to the model compression block 400 by the alpha analyzer block 300 of one embodiment of the present invention. The flowchart begins at step S301.

[0035] First, the α analyzer block 300 ranks all candidate sublayers for each layer based on the α parameter value, as shown by step S302. The candidate sublayer with the highest α parameter value for each layer is selected as the winning sublayer, as shown in the "Winner" column of FIG. 9. The candidate sublayer with the second highest α parameter value is selected as the second-highest, as shown in the "Second Place" column of FIG. 9. Similarly, the ranking continues to the third-highest, as shown in FIG. 9. Next, one layer I is selected from the table of FIG. 9, as shown in step S303 of FIG. 4.

[0036] Next, the sizes of the candidate sublayers of all rankings other than the winner are compared with the size of the winning sublayer. The definition of size varies depending on the use case. Some definitions of size include, but are not limited to, the number of multiply-accumulate operations, the number of weight parameters, or execution time. The α parameter values ​​of all rankings other than the winner are compared with the α parameter value of the winning sublayer. As shown in step S304 of FIG. 4, weak candidate sublayers within each layer, i.e., candidate sublayers with a size larger than the size of the winning sublayer or candidate sublayers with an α parameter value smaller than the α parameter value of the winning sublayer by a predetermined threshold (Δ), are removed from the candidate search space, which lists candidate sublayers to be replaced by each layer of the backbone of the smaller CNN model, thereby reducing the candidate search space. In step S305 of FIG. 4, a process similar to the removal of candidate sublayers is performed on the remaining layers. An example of the reduced candidate search space is shown in the table of FIG. 10. The reduced candidate search space, including the reduced number of candidate sublayers, is output to the analyzer block 300. The flowchart ends at step S306.

[0037] The operation of the model compression block 400 will now be described.

[0038] The model compression block 400 receives the trained subnet 2001 from the neural architecture search block 200 and the reduced candidate search space from the alpha analyzer block 300. The model compression block 400 then compresses the subnet 2001 by replacing each layer of the backbone 2011 with a lightweight sublayer, as shown below, so that accuracy is roughly maintained.

[0039] Figure 5 shows a flowchart of the model compression block 400 of one embodiment of the present invention. Figure 8 shows an example block diagram of a subnet 2001B architecture compressed by the model compression block 400 of one embodiment of the present invention. The flowchart starts at step S401.

[0040] As shown in step S402 of FIG. 5, the model compression block 400 starts with (or selects) one layer i for compression. Next, the model compression block 400 replaces the winning sublayer in each layer within the reduced candidate search space, successively trying all remaining candidate sublayers, as shown in step S403 of FIG. 5, to compress the subnet 2001 into a lightweight subnet 2001B as shown in FIG. 8. As shown in step S404 of FIG. 5, the candidate sublayer that approximately maintains the minimum size and accuracy (the candidate sublayer with the best evaluation result) is selected, and the original winning sublayer is replaced with the candidate sublayer with the best evaluation result (which is considered the new winner). As shown in step S405 of FIG. 5, a step of evaluating all lightweight candidate sublayers against the original winning sublayer is performed for all remaining layers. The flowchart ends in step S406.

[0041] Finally, as shown in Figure 8, the lightweight subnet 2001B including the backbone 2011B compressed by the model compression block 400 is provided at the output by the model compression block 400, where X, Y, and Z denote examples of candidate permuted sublayers in each of layers 1, 2, and N, respectively.

[0042] The model deployment block 500 receives the lightweight subnet 2001B from the model compression block 400 and deploys it efficiently onto the target hardware. The internal workings of the model deployment block 500 are outside the scope of this invention.

[0043] It is generally recognized that fully training the backbone 201 of a supernet is difficult due to the large CNN structure. Therefore, the weights of all candidate sublayers are not fully trained. This makes it difficult to find the optimal backbone 2011 of the subnetwork 2001 model from the entire search space. Due to the insufficiently trained weights of the candidate sublayers, there is a high possibility that their respective α parameter values ​​are not sufficiently trained and are very close to the winning sublayer. However, candidate sublayers with too low α parameter values ​​are quite weak candidates and should no longer be present in the search space for model compression. Therefore, removing such weak candidates from the search space of the model compression block 400 will result in faster execution of the model compression block 400. Faster execution of the model compression block 400 ultimately leads to a faster automatic machine learning development system 100.

[0044] In the automated machine learning development system 100 of one embodiment of the present invention, the neural architecture search block 200 can also search for the backbone 2011 of the optimal subnet 2001 for the neck 202 layer and the head 203 layer. Therefore, the alpha analyzer block 300 and the model compression block 400 can operate on the neck 202 layer and the head 203 layer, respectively.

[0045] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and further modifications, substitutions, and adjustments can be made without departing from the basic technical concept of the present invention. For example, the network configurations, element configurations, and message representation formats shown in the drawings are examples intended to aid in understanding the present invention. The present invention is not limited to the configurations shown in these drawings. Furthermore, "A and / or B" in the above description means at least either A or B. Even if not shown, ports or interfaces exist at the connection points between inputs and outputs in individual blocks in the drawings.

[0046] Furthermore, the procedure described in one embodiment can be realized by a program that causes a computer (9000 in FIG. 11) functioning as an automated machine learning development system to realize the functions of the automated machine learning development system. Such a computer is exemplified by a configuration including a CPU (Central Processing Unit) 9010, a communication interface 9020, a memory 9030, and an auxiliary storage device 9040 in FIG. 11. That is, the CPU 9010 in FIG. 11 may be configured to execute a processing program as the automated machine learning development system 100 and to perform an update process for each calculation parameter stored in the auxiliary storage device 9040 or the like.

[0047] In other words, each part (processing means, function) of the automated machine learning development system 100 shown in the above-described embodiment can be realized by a computer program that causes a processor installed in the automated machine learning development system 100 to execute each of the above-described processes using its hardware.

[0048] Finally, preferred embodiments of the present invention will be summarized. [First form] (See the first perspective on automated machine learning development systems above) [Second form] In the first form of the automated machine learning development system, the size of the candidate sublayer is preferably the number of multiply-accumulate operations, the number of weight parameters, or the execution time of the candidate sublayer. [Third form] In the first and second forms of the automated machine learning development system, the dataset includes images and labels; Preferably, the images are input to the large CNN model and the small CNN model, and the labels are predictions that the large CNN model and the small CNN model are intended to produce as output. [Fourth form] In any one of the first to third forms of the automated machine learning development system, the candidate search space preferably lists candidate sublayers to be replaced with each layer of the backbone of the smaller CNN model. [Fifth form] In the automated machine learning development system of any one of the first to fourth aspects, it is preferable that the α parameter value of each layer is normalized so that the sum of the α parameter values ​​within each layer is 1. [6th form] In the automated machine learning development system of any one of the first to fifth aspects, it is preferable that the number of candidate sublayers within a layer varies from layer to layer. [7th form] (See the second perspective on automated machine learning development methods above.) [8th form] In the seventh form of the automated machine learning development method, it is preferable that the size of the candidate sublayer is the number of product-sum operations, the number of weight parameters, or the execution time of the candidate sublayer. [9th Form] (See the third aspect of computer programs above.) [10th Form] In the computer program of the ninth aspect, it is preferable that the size of the candidate sub-layer is the number of multiply-accumulate operations, the number of weight parameters, or the execution time of the candidate sub-layer.

[0049] The disclosures of the above-mentioned patent documents are incorporated herein by reference. Modifications and adjustments of the embodiments and examples are possible within the scope of the entire disclosure of the present invention (including the scope of the claims), and further based on the basic technical concept thereof. Furthermore, various combinations and selections (including partial deletions) of various disclosed elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible within the scope of the disclosure of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible by a person skilled in the art in accordance with the entire disclosure and technical concept, including the scope of the claims. In particular, with regard to the numerical ranges described herein, any numerical value or subrange included within the range should be construed as being specifically described, even if not otherwise specified. [Explanation of symbols]

[0050] 100 Automated Machine Learning Development System 101 datasets 200 Neural Architecture Search Blocks 201 Neural Architecture Search Blocks Constructed as the Backbone of a Supernet 2000 Supernet 2001 Subnet 2001B lightweight subnet 2011 Derived backbone part of a subnet constructed by neural architecture search block 2011B Model Compression Block Compressed Lightweight Subnet Backbone 202 Neural Architecture: The Neck Layer of a Supernet Constructed by Search Blocks 203 Head layer of a supernet constructed with neural architecture search blocks 300 α analyzer block 400 Model Compression Blocks 500 Model Deployment Blocks

Claims

1. An automated machine learning development system including a neural architecture search block, an alpha analyzer block, a model compression block, and a model deployment block, The neural architecture search block includes: - receiving a data set, - Build a large CNN model, The large CNN model includes at least a backbone, a neck, and a head; the backbone includes a series of connected layers, each of which includes a plurality of candidate sublayers connected in parallel; an α parameter is assigned to the output of each of the candidate sublayers of the backbone, and the output of each of the candidate sublayers is multiplied by a respective trained α parameter value; - training the backbone of the large CNN model by using the dataset to train weights and α parameter values ​​of all candidate sublayers of the backbone of the large CNN model; - selecting a best candidate sublayer for each of the layers based on the highest trained α parameter value, and constructing a small CNN model including at least a derived backbone including the selected best candidate sublayer for each of the layers, the neck, and the head; - training the derived backbone of the small CNN model with the dataset; - outputting a trained small CNN model including the trained α parameter values ​​and the trained derived backbone of the large CNN model; The alpha analyzer block - receiving trained α parameter values ​​from the neural architecture search block; - ranking all the candidate sublayers of each of the layers based on the trained α parameter values ​​and comparing the trained α parameter values ​​with the sizes of the candidate sublayers; - removing weak candidate sublayers within each of the layers from the candidate search space, the candidate sublayers having a size larger than the candidate sublayer with the highest α parameter value or having an α parameter value smaller than the highest α parameter value by a predetermined threshold, thereby reducing the candidate search space for the model compression; - outputting a reduced candidate search space containing a reduced number of said candidate sub-layers to said model compression block; The model compression block - receiving the trained small CNN model from the neural architecture search block and the reduced candidate search space from the alpha analyzer block; compressing the trained small CNN model into a lightweight CNN model by replacing sublayers in each layer of the trained derived backbone of the smaller CNN model with the remaining candidate sublayers in the reduced candidate search space; The model deployment block includes: Deploying the lightweight CNN model on target hardware. Automated machine learning development system.

2. The automated machine learning development system of claim 1 , wherein the size of the candidate sublayer is the number of multiply-accumulate operations, the number of weight parameters, or the execution time of the candidate sublayer.

3. The dataset contains images and labels.

2. The automated machine learning development system of claim 1, wherein the images are input to the large CNN model and the small CNN model, and the labels are predictions that the large CNN model and the small CNN model are intended to produce as output.

4. 2. The automated machine learning development system of claim 1, wherein the candidate search space lists candidate sublayers to be replaced in each layer of the backbone of the smaller CNN model.

5. The automated machine learning development system of claim 1 , wherein the α parameter values ​​of each layer are normalized so that the sum of the α parameter values ​​within each layer is 1.

6. The automated machine learning development system of claim 1 , wherein the number of candidate sublayers within the layer varies from layer to layer.

7. The computer - receiving a data set; - Building a large CNN model, The large CNN model includes at least a backbone, a neck, and a head; the backbone includes a series of connected layers, each of which includes a plurality of candidate sublayers connected in parallel; an α parameter is assigned to the output of each of the candidate sublayers of the backbone, and the output of each of the candidate sublayers is multiplied by a respective trained α parameter value; The computer - training the backbone of the large CNN model by training the weights and α parameter values ​​of all candidate sublayers of the backbone of the large CNN model using the dataset; - selecting a best candidate sublayer for each of the layers based on the highest trained α parameter value, and constructing a small CNN model including a derived backbone including at least the selected best candidate sublayer for each of the layers, the neck, and the head; - training the derived backbone of the small CNN model with the dataset; outputting a trained small CNN model including the trained α parameter values ​​of the large CNN model and a trained derived backbone; The computer - receiving trained α parameter values; - ranking all the candidate sublayers of each of the layers based on the trained α parameter value and comparing the trained α parameter value with the size of the candidate sublayers; - removing weak candidate sublayers within each of the layers from the candidate search space, the candidate sublayers having a size larger than the candidate sublayer with the highest α parameter value or having an α parameter value smaller than the highest α parameter value by a predetermined threshold, thereby reducing the candidate search space for the model compression; - outputting a reduced candidate search space comprising a reduced number of said candidate sublayers, The computer receiving the trained small CNN model from the neural architecture search block and the reduced candidate search space from the alpha analyzer block; compressing the trained small CNN model into a lightweight CNN model by replacing sublayers in each layer of the trained derived backbone of the smaller CNN model with the remaining candidate sublayers in the reduced candidate search space; The computer - Deploying the lightweight CNN model on target hardware; Automated machine learning development methods, including:

8. The automated machine learning development method according to claim 7 , wherein the size of the candidate sublayer is the number of multiply-accumulate operations, the number of weight parameters, or the execution time of the candidate sublayer.

9. On the computer, - receiving a data set; - Running a process to build a large CNN model, The large CNN model includes at least a backbone, a neck, and a head; the backbone includes a series of connected layers, each of which includes a plurality of candidate sublayers connected in parallel; an α parameter is assigned to the output of each of the candidate sublayers of the backbone, and the output of each of the candidate sublayers is multiplied by a respective trained α parameter value; The computer, - training the backbone of the large CNN model by using the dataset to train the weights and α parameter values ​​of all candidate sublayers of the backbone of the large CNN model; - selecting a best candidate sublayer for each of the layers based on the highest trained α parameter value, and constructing a small CNN model including a derived backbone including at least the selected best candidate sublayer for each of the layers, the neck, and the head; - training the derived backbone of the small CNN model using the dataset; - performing a process to output a trained small CNN model including the trained α parameter values ​​and the trained derived backbone of the large CNN model; The computer, - receiving trained α parameter values; - ranking all the candidate sublayers of each of the layers based on the trained α parameter value and comparing the trained α parameter value with the size of the candidate sublayers; - removing weak candidate sublayers within each of the layers from the candidate search space, the candidate sublayers having a size larger than the candidate sublayer with the highest α parameter value or having an α parameter value smaller than the highest α parameter value by a predetermined threshold, thereby reducing the candidate search space for the model compression; - performing a process that outputs a reduced candidate search space that includes a reduced number of said candidate sublayers; The computer, - receiving the trained small CNN model from the neural architecture search block and the reduced candidate search space from the alpha analyzer block; - compressing the trained small CNN model into a lightweight CNN model by replacing sublayers in each layer of the trained derived backbone of the smaller CNN model with the remaining candidate sublayers in the reduced candidate search space; The computer, - Deploying the lightweight CNN model on target hardware; A computer program that executes

10. The computer program product of claim 9 , wherein the size of a candidate sublayer is the number of multiply-accumulate operations, the number of weight parameters, or the execution time of the candidate sublayer.

Images