System and method for machine learning evaluation pipeline

Abstract

To provide a method, or the like, for a neural architecture search (NAS) pipeline.SOLUTION: Provided are a method, system, and device for a neural architecture search (NAS) pipeline for performing an optimized neural architecture search (NAS). The method may include: obtaining a first search space comprising a plurality of candidate layers for a neural network architecture; performing a training-free NAS in the first search space to obtain a first set of architectures; performing the training-free NAS in the first search space to obtain a second set of architectures; performing a gradient-based search in the second search space to obtain a second set of architectures; performing a sampling method search utilizing the second set of architectures as an initial sample; and obtaining an output architecture as an output of the sampling method search.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] Systems and methods according to example embodiments of the present disclosure relate to providing a pipeline for evaluating machine learning models. [Background technology]

[0002] In the related art, neural networks can generally be characterized by two main parameters: (1) the architecture of the neural network, and (2) the weights applied to the inputs transmitted between neurons. Typically, the architecture is designed manually, i.e., by a user, and the weights are optimized by training the network with a training set and a neural network algorithm. Thus, to optimize the performance of a neural network, the architecture design is an important consideration, especially since neural networks are generally static after they are deployed for use. Figure 1 is a diagram showing an example of a neural network architecture in the related art.

[0003] 1, an example of a neural network architecture in the related art is defined by the connection, the type (or operation) of layers, the number of layers, and the number of channels in each layer. The layer types may include, for example, convolution (in the case of a Convolutional Neural Network (CNN)), activation (e.g., ReLU (Rectified Linear Unit)), pooling, fully connected, batch normalization, dropout, etc., and the architecture design of the network is embodied by a combination of layers corresponding to at least some of these layer types.

[0004] In the related art, there is a technique called Neural Architecture Search (NAS) for automatically designing the architecture of a neural network. Related art methods for performing NAS may include training-free (zero-shot) NAS, supernet-based NAS, and simple iterative search.

[0005] In the related art, no-training (zero-shot) NAS predicts network performance without training network parameters. This method is fast but may not have good accuracy. For example, this method can be done by evaluating metrics or architectures in a search space without actually training.

[0006] Fig. 2 is a diagram showing a supernet-based (gradient-based) NAS according to the related art, Fig. 3 shows an example of a superblock in the supernet-based NAS, and Fig. 4 shows an example of a flow of the supernet-based NAS method according to the related art.

[0007] Supernet-based (gradient-based) NAS may use a supernet. A supernet is a network of all candidate architectures. With reference to FIG. 2, a supernet-based search searches neural network architectures through a process of training multiple connected superblocks once, where each superblock corresponds to one layer (and / or set of layers) and includes many choices of candidate layers (and / or set of layers). For example, choice 1 may be 3×3 CNN, choice 2 may be 5×5 CNN, choice 3 may be max pooling or residual block, etc. In a supernet-based search, the best layer is searched and selected in each superblock. As shown in FIG. 3 and FIG. 4, each candidate has a θ value (or architecture parameter), which is updated through the training process. That is, once a candidate is determined to have contributed significantly to the loss reduction, the θ value of the candidate (representing the importance of the candidate layer) is increased through back propagation, and vice versa. After the architecture parameter θ is updated in one iteration of the supernet-based search, the weight parameters are updated using training images (e.g., car images). This process is repeated until a certain network stability is reached (i.e., a certain latency (inference time) or loss is reached) or a predetermined number of iterations are completed. Once training is complete, the best layer (i.e., the layer with the highest θ value) is selected for each superblock to generate the final (optimal) network architecture.

[0008] Fig. 5 is a functional block diagram of a sampling method according to the related art. Referring to Fig. 5, the sampling method is an iterative process by which a controller generates sample architecture candidates (e.g., from a search space consisting of a set of layers). The sample architectures are optimized through training and evaluated to obtain metric values ​​(e.g., accuracy, latency, model size, etc.). The controller then generates the next sample architecture candidate based on the evaluation results of the previous sample using methods such as reinforcement learning, Bayesian optimization, and evolutionary algorithms. This process is repeated until a target performance is reached (e.g., a target accuracy, a target latency, a target mode size, etc.).

[0009] The sampling method illustrated in FIG. 5 may yield the most accurate / optimal architecture, but it involves extensive training and is therefore very time consuming to run across a large number of candidates.

[0010] While doing NAS manually (as is common in related art systems) may certainly result in a higher quality architecture, it is very time consuming and burdensome to the user, so automating NAS is desirable.

[0011] However, the above-mentioned existing methods for automating NAS in the related art can be time-consuming and may not result in an optimal network architecture. In particular, if the search space is large, it takes too much resource and time to complete. On the other hand, if the search space is too small, it is very likely to result in a suboptimal architecture with poor performance. Therefore, there is a need for a method for NAS that can find a network with optimal performance in a minimum amount of time. Summary of the Invention

[0012] According to an embodiment, a method, system, and device of a neural architecture search (NAS) pipeline for performing an optimized neural architecture search (NAS) is provided. Specifically, the apparatus and method according to the exemplary embodiment may sequentially perform a training-free NAS, a supernet / gradient-based search, and a sampling search to reduce the number of architectures that need to be trained at the end of the sampling search while ensuring high quality of the architectures provided at the end of the sampling search. Thus, an overall high quality / optimal architecture can be found in a minimum amount of time.

[0013] According to an embodiment, a method for performing neural architecture search (NAS) may be provided, which includes obtaining a first search space including a plurality of candidate layers for neural network architectures, performing training-free NAS in the first search space to obtain a first set of architectures, obtaining a second search space based on the first set of architectures, performing a gradient-based search in the second search space to obtain the second set of architectures, performing a sampling search using the second set of architectures as initial samples, and obtaining an output architecture as an output of the sampling search.

[0014] According to some embodiments, the first search space may be obtained based on a set of architecture parameters, and the second search space may be obtained based on a subspace that includes one or more architectures in the first set of architectures.

[0015] According to some embodiments, the sampling search may be performed iteratively. The sampling search may include an evolutionary search algorithm. The number of iterations of the sampling search may be based on a predefined threshold. The training-free NAS may be performed based on one or more metrics.

[0016] Additional aspects will be set forth in part in the description that follows, and in part will be apparent from the description, or may be learned by practicing the presented embodiments of the present disclosure. [Brief description of the drawings]

[0017] The features, advantages, and importance of exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which like reference numbers refer to like elements, and in which:

[0018] [Figure 1] FIG. 1 is a diagram showing an example of a neural network architecture according to the related art.

[0019] [Diagram 2] FIG. 2 is a functional block diagram of a related art supernet-based (gradient-based) search method.

[0020] [Diagram 3] FIG. 3 is a diagram showing an example of a superblock in a supernet-based searching method according to the related art.

[0021] [Figure 4] FIG. 4 is a flow diagram of a supernet-based searching method according to the related art.

[0022] [Diagram 5] FIG. 5 is a functional block diagram showing a sampling method according to the related art.

[0023] [Figure 6] FIG. 6 illustrates an example Neural Architecture Search (NAS) pipeline according to one or more embodiments.

[0024] [Figure 7] FIG. 7 is a flow diagram of an exemplary method for performing an optimized NAS process according to one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] The following detailed description of exemplary embodiments refers to the accompanying drawings. The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations. Moreover, one or more features or components of one embodiment may be incorporated or combined with another embodiment (or one or more features of another embodiment). In addition, it is understood that in the flowcharts and operational descriptions provided below, one or more operations may be omitted, one or more operations may be added, one or more operations may be performed (at least partially) simultaneously, and one or more operations may be reordered.

[0026] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code. As such, it will be understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.

[0027] Although particular combinations of features are recited in the claims and / or disclosed herein, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features can be combined in ways not specifically recited in the claims and / or specifically disclosed herein. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes the combination of each dependent claim with every other claim in the claims.

[0028] No element, act, or instruction used herein should be construed as critical or essential unless expressly stated otherwise. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." Where only one item is intended, the term "one" or similar words are used. Also, as used herein, the terms "has," "have," "having," "include," "including," and the like are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless expressly stated otherwise. Furthermore, phrases such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood as including only A, only B, or both A and B.

[0029] Throughout this specification, reference to "one embodiment," "an embodiment," "a non-limiting exemplary embodiment," or similar language means that a particular feature, structure, or characteristic described in connection with the illustrated embodiment is included in at least one embodiment of the solution. Thus, the phrases "in one embodiment," "in an embodiment," "in one non-limiting exemplary embodiment," and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment.

[0030] Furthermore, the features, advantages, and characteristics described in the present disclosure may be combined in any suitable manner in one or more embodiments. In light of the description herein, one skilled in the art will recognize that the present disclosure may be practiced without one or more of the specific features or advantages of a particular embodiment. In other cases, additional features and advantages may be recognized in a particular embodiment, but may not be present in all embodiments of the present disclosure.

[0031] Exemplary embodiments of the present disclosure provide a method and system that sequentially performs training-free NAS, supernet / gradient-based search, and sampling search to reduce the number of architectures that need to be trained at the end of the sampling search while ensuring high quality of the architectures provided at the end of the sampling search, thereby finding an overall high quality / optimal architecture in the least amount of time.

[0032] Figure 6 illustrates an example of a neural architecture search (NAS) pipeline 600. With reference to Figure 6, a baseline model and / or a target latency for a desired architecture may be provided (e.g., input by a user) to the NAS pipeline 600. This is useful for providing an initial search space, particularly since an initial search space can be generated based on the baseline model and the target latency can also be used to filter out architectures that do not match the target latency.

[0033] 6, in operation S601, an initial first search space may be obtained (e.g., constructed). The initial first search space may include multiple candidate layers for a neural network architecture. According to an embodiment, the initial first search space may be provided by a user or may be obtained based on a set of architecture parameters.

[0034] Referring to FIG. 6, in step S602, the initial first search space may be refined to obtain a second search space. According to an embodiment, the refinement may be performed by first performing a training-free NAS on the first search space to obtain a first architecture set. It should be understood that the training-free NAS may use a variety of methods as long as it does not involve training. For example, good architectures may be evaluated based on simple metrics. Thus, the first architecture set may be obtained. Since no training is involved in operation S602, the speed of completion should be relatively fast.

[0035] The first set of architectures may then be utilized to obtain a second search space. The second search space may also be considered a "reduced" or "contracted" search space compared to the first search space. According to an embodiment, subspaces including and / or surrounding each architecture in the first set of architectures may be used to determine the second search space. For example, if the first set of architectures includes architectures 1, 2, and 3, subspaces 1, 2, and 3 may correspond to search spaces (subspaces) surrounding architectures 1, 2, and 3, respectively. Thus, in one example, the second search space may be the union of subspaces 1, 2, and 3.

[0036] Referring again to Figure 6, in step S603, a supernet-based NAS (gradient-based search) may be performed using the second (reduced) search space. This method results in a second set of architectures based on the second search space. Gradient-based search should also be relatively quick since it involves only one training and the second search space is already a reduced search space.

[0037] Referring again to FIG. 6, in operation S604, a sampling method may be performed using each architecture in the second architecture set as an initial sample (full training search). According to an embodiment, this may be an iterative full training search. Since the second architecture set is already relatively narrowed down compared to the first architecture set, even if full training is performed, the time is still relatively short. Rather, since full training is performed, the accuracy can be considered optimal / high. Thus, the NAS pipeline 600 may result in an optimal architecture being obtained.

[0038] According to one embodiment, the sampling search may be an evolutionary search algorithm. The number of iterations may also be based on a predefined threshold. Specifically, the evolutionary search may be iterative and may include using performance metrics or fitness / validation scores to obtain an optimized architecture based on the second search space. According to one embodiment, in each iteration of the evolutionary search, a candidate architecture is mutated (e.g., changing a 5×5 convolutional layer to a 3×3 convolutional layer) using the entire search space (e.g., any possible layer, layer type, number of channels, number of layers), trained (e.g., for several epochs), and evaluated. The lowest scoring architecture in the candidate pool is then replaced with a new (mutated) architecture that is determined to perform better. This is repeated over multiple iterations until a predefined number of iterations have been performed or until some predefined threshold (e.g., a predefined optimal performance) is reached, and an optimized architecture in the second space is output. Nevertheless, it should be understood that the threshold for selecting the optimized architecture may use several possible criteria. The simplest method would be to select the architecture with the highest score, but if multiple metrics (e.g., accuracy and latency (runtime speed)) are considered, one exemplary process would output all architectures in the Pareto front set. Another possible method would be to use crossover during the evolutionary search process. Specifically, an architecture with a low score could be replaced by a crossover of two architectures with higher scores.

[0039] It will be appreciated that while the embodiment described above utilizes evolutionary search, one or more other embodiments are not so limited, and other embodiments may utilize other sampling techniques or algorithms (e.g., reinforcement learning).

[0040] 7 illustrates a flow diagram of an example method 700 for performing an optimized NAS process. Referring to FIG. 7, in operation S710, a first search space having candidate layers for a neural network architecture is obtained. This may be similar to operation S601 described above with respect to FIG. 6.

[0041] At operation S720, a training-free NAS may be performed on the first search space to obtain a first set of architectures, which may be similar to portions of operation S602 described above with respect to FIG.

[0042] At operation S730, a second (reduced) search space may be obtained based on the first set of architectures. According to some embodiments, this may be obtained based on a subspace that includes and / or surrounds each architecture in the first set of architectures to determine the second search space. This may be similar to portions of operation S602 described above with respect to FIG. 6.

[0043] At act S740, a supernet-based (gradient-based) search may be performed in the second search space obtained at act S730 to output a second set of architectures, which may be similar to portions of act S603 described above with respect to FIG.

[0044] In act S750, a sampling method may be performed using the second set of architectures obtained from act S740 as an initial sample. According to one embodiment, the sampling method may be an evolutionary search algorithm. The evolutionary search algorithm may be repeated for a number of iterations, which may be based on a predefined threshold. This may be similar to act S604 described above with respect to FIG. 6.

[0045] At operation S760, the optimal architecture is output as a result of the sampling method performed at operation S750. It should also be appreciated that although the present embodiment is described as outputting one optimal architecture, some embodiments may output multiple optimal architectures.

[0046] In view of the above, exemplary embodiments of the present disclosure provide a method and system for optimized neural architecture search (NAS) by setting a coarse initial search space and then narrowing the search space to substantially increase the speed of the search, and the narrowed search space can be used in combination with supernet NAS and iterative sampling search to yield architectures with optimal performance. Thus, by using these search methods in combination in sequence, optimal performance architectures can be found with substantially shorter search and computation times.

[0047] It should be understood that the particular order or hierarchy of blocks in the processes / flowcharts disclosed herein is illustrative of an example methodology. Based on design preferences, it should be understood that the particular order or hierarchy of blocks in the processes / flowcharts may be rearranged. Further, some blocks may be combined or omitted. The accompanying method claims present elements of the various blocks in an example order and are not intended to be limited to the particular order or hierarchy presented.

[0048] Some embodiments may relate to systems, methods, and / or computer-readable media at any conceivable level of technical detail. Furthermore, one or more of the above components described above may be implemented as instructions executable by at least one processor stored on a computer-readable medium (and / or may include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium (or media) having computer-readable program instructions for causing a processor to perform operations.

[0049] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memories (RAM), read only memories (ROM), erasable programmable read only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read only memories (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanically encoded punch cards, or raised structures in grooves with instructions recorded thereon, and any suitable combination thereof. Computer-readable storage media, as used herein, should not be construed as being themselves transitory signals such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0050] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to the respective computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0051] The computer readable program code / instructions for performing the operations may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state configuration data, configuration data of an integrated circuit, or source or object code, which may be written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may run entirely on the user's computer as a stand-alone software package, or partially on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuit to perform an aspect or operation.

[0052] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to generate a machine such that the instructions, executed via the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may be stored on a computer readable storage medium capable of directing a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0053] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or another device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0054] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or part of an instruction, including one or more executable instructions for implementing the specified logical function(s). The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks than those shown in the figures. In some alternative implementations, the functions described in the blocks may occur in a different order than that described in the figures. For example, two blocks shown in succession may in fact be executed simultaneously or substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, as well as combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0055] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, the operation and behavior of the systems and / or methods have been described herein without reference to any specific software code. It will be understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.

Claims

1. 1. A method for conducting neural architecture search (NAS), comprising: Obtaining a first search space including a plurality of candidate layers for a neural network architecture; performing training-free NAS within the first search space to obtain a first set of architectures; obtaining a second search space based on the first architecture set; performing a gradient based search within the second search space to obtain a second set of architectures; performing a sampling search using the second set of architectures as an initial sample; obtaining an output architecture as an output of said sampling search; The method includes:

2. The method of claim 1 , wherein the first search space is obtained based on a set of architectural parameters.

3. The method of claim 1 or 2, wherein the second search space is obtained based on a subspace that includes one or more architectures in the first set of architectures.

4. The method of claim 1 or 2, wherein the sampling search is performed iteratively.

5. The method of claim 4 , wherein the sampling search comprises an evolutionary search algorithm.

6. The method of claim 5 , wherein the number of iterations of the sampling search is based on a predetermined threshold.

7. The method of claim 1 or 2, wherein the training-free NAS is based on one or more metrics.

8. 1. An apparatus for performing neural architecture search (NAS), comprising: at least one memory storing computer executable instructions; At least one processor; The processor executes the computer-executable instructions to: Obtaining a first search space including a plurality of candidate layers for a neural network architecture; performing training-free NAS within the first search space to obtain a first set of architectures; obtaining a second search space based on the first set of architectures; performing a gradient-based search within the second search space to obtain a second set of architectures; performing a sampling search using the second set of architectures as an initial sample; obtaining an output architecture as an output of the sampling method search. Device.

9. The apparatus of claim 8 , wherein the first search space is obtained based on a set of architectural parameters.

10. The apparatus according to claim 8 or 9, wherein the second search space is obtained based on a subspace that includes one or more architectures in the first set of architectures.

11. 10. The apparatus of claim 8 or 9, wherein the sampling search is performed iteratively.

12. The apparatus of claim 11 , wherein the sampling search comprises an evolutionary search algorithm.

13. The apparatus of claim 12 , wherein the number of iterations of the sampling search is based on a predetermined threshold.

14. The apparatus of claim 8 or 9, wherein the training-free NAS is based on one or more metrics.

15. A non-transitory computer-readable recording medium having instructions executable by at least one processor, the instructions causing the at least one processor to: Obtaining a second search space based on the first set of architectures; performing a gradient based search within the second search space to obtain a second set of architectures; performing a sampling search using the second set of architectures as an initial sample; obtaining an output architecture as an output of said sampling search; A non-transitory computer readable medium for carrying out a method, comprising:

16. The non-transitory computer-readable medium of claim 15 , wherein the first search space is obtained based on a set of architectural parameters.

17. The non-transitory computer-readable storage medium of claim 15 or 16, wherein the second search space is obtained based on a subspace that includes one or more architectures in the first set of architectures.

18. The non-transitory computer-readable storage medium of claim 15 or 16, wherein the sampling search is performed iteratively.

19. 20. The non-transitory computer-readable medium of claim 18, wherein the sampling search comprises an evolutionary search algorithm.

20. 20. The non-transitory computer-readable medium of claim 19, wherein the number of iterations of the sampling search is based on a predetermined threshold.

Images