Apparatus and method for integrated search of artificial neural network and computation accelerator structure considering hard constraints

Abstract

To satisfy hard constraints while performing an integrated search of an artificial neural network and a computation accelerator.SOLUTION: An integrated search device for an artificial neural network and a computation accelerator structure with consideration on hard constraints includes: a Neural Architecture Search module that determines a neural network architecture; a High-constrained Differentiable Co-Exploration evaluation module that determines an accelerator architecture based on the determined neural network architecture and predicts the loss of the determined neural network architecture and hardware metrics related to the determined accelerator architecture; and a gradient adjustment module that adjusts the gradients of a loss function and a constraint function so that the hardware metrics predicted based on a gradient descent method move in a direction that satisfies designated constraints.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an integrated search technique for an artificial neural network and a dedicated hardware accelerator, and more particularly to an integrated search device and method for an artificial neural network and an arithmetic accelerator structure that considers hard constraints and can satisfy hard constraints such as throughput per second and energy consumption while performing an integrated search of the artificial neural network and the arithmetic accelerator. [Background technology]

[0002] While the primary concern of most DNN (Deep Neural Network) research has been application performance (i.e., accuracy), it has also led to a rapid growth in network size that requires huge computational resources to run. In recent years, numerous schemes have emerged to mitigate resource issues, most of which fall into one of the network and hardware optimization categories. Network optimization refers to improving neural network architectures to reduce computation while maintaining similar accuracy, while hardware optimization involves improving the efficiency of DNN execution using optimized hardware called accelerators. Unfortunately, one effort often hampers the benefits of the other. As an example, it is known that MobileNet, which was developed to achieve high speeds, performs slower than general neural networks when run on a well-known accelerator, the TPU, because the characteristics of MobileNet, which reduces the number of channels, directly contradict the TPU's acceleration strategy.

[0003] For this reason, it is necessary to design the accelerator and the artificial neural network simultaneously so that they can synergize with each other, when the work to be done by the accelerator is already known.

[0004] Existing integrated design methodologies can be broadly divided into two types. The first type is based on reinforcement learning (RL) or evolutionary algorithms, and is considered to be close to traditional integrated design methods. In this method, a solution with an arbitrary accelerator structure and neural network structure is derived, then evaluated, and a method is selected to find a better solution based on the evaluation results. Traditionally, quantized approximation algorithms, genetic algorithms, etc. are used to find a better solution, and reinforcement learning has been used since the popularization of deep learning.

[0005] However, this method has the drawback that the process of evaluating each solution takes a very long time. In the case of an artificial neural network, a learning process is required to evaluate its accuracy, but it is known that it usually takes several weeks to learn one neural network, and even if parallelization is performed using a large number of servers, it is not easy to handle the enormous amount of calculations.

[0006] To overcome this, a backpropagation-based methodology has been recently presented, which is based on a neural network search method, in which the solution space of the neural network and accelerator is designed to be differentiable, and then a learnable multiplication factor is placed on each differentiable path. Once the learning is completed, the path with the factor with the largest value becomes the final solution, and the design of the neural network and accelerator is derived based on this. In the case of a neural network, the accuracy is evaluated by learning the structure created in this way once.

[0007] This approach shares most deep learning-based methodologies and their shortcomings due to the fact that backpropagation relies on a single loss function, which is not a problem when there is only one desired objective, but when there are multiple, it is difficult to achieve the desired balance between these objectives, especially when a particular objective has hard constraints, which is practically impossible to reflect. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Korean Patent Publication No. 10-2019-0101677 Summary of the Invention [Problem to be solved by the invention]

[0009] The present invention has been made in consideration of the above-mentioned problems in the conventional technology, and an object of the present invention is to provide an apparatus and method for integrated search of an artificial neural network and a computation accelerator structure, taking into account hard constraints, which can satisfy hard constraints while performing an integrated search of an artificial neural network and a computation accelerator. [Means for solving the problem]

[0010] In order to achieve the above object, according to one aspect of the present invention, an integrated exploration device for an artificial neural network and a computation accelerator structure taking into account hard constraints includes a Neural Architecture Search (NAS) module that determines a neural network architecture, a High-constrained Differentiable Co-Exploration (HDX) evaluation module that determines an accelerator architecture based on the determined neural network architecture and predicts a loss of the determined neural network architecture and a hardware metric related to the determined accelerator architecture, and a gradient adjustment module that adjusts the gradients of a loss function (Loss) and a constraint function (Const) based on a gradient descent method so that the predicted hardware metric moves in a direction that satisfies a specified constraint.

[0011] The NAS module can use backpropagation to find paths in a SuperNet to generate the neural network architecture that is ultimately searched, and calculate the cross entropy loss (LossCE). The HDX evaluation module may include a hardware generation network that is constructed through pre-learning, searches for optimal hardware for the determined neural network architecture as the accelerator architecture, and determines at least one of a PE (Processing Element) array configuration (PEx, PEy), a register file (RF) configuration, and a dataflow (DF) configuration for the accelerator architecture, and a cost estimation network that predicts the hardware metrics based on parameters for the neural network architecture and a configuration for the accelerator architecture. The cost estimation network can be constructed of a multi-layer perceptron with residual connections, through which delay, area, and energy consumption can be determined and the hardware metrics can be predicted. The cost estimation network can predict the hardware metric by calculating a balanced weighted sum over the delay, area, and energy consumption as a cost function. The gradient adjustment module can perform gradient adjustment of the loss function if the predicted hardware metrics do not satisfy a specified constraint, to select the neural network architecture toward satisfying the constraint. The gradient adjustment module may calculate an adjustment direction through a pseudo-inverse such that an angle between the gradient of the loss function and the gradient of the constraint function is 90 degrees or less. The gradient adjustment module can perform a gradient adjustment of a hardware cost function when the predicted hardware metric does not satisfy a specified constraint, to select the accelerator architecture in a direction toward satisfying the constraint.

[0012] In order to achieve the above object, one aspect of the present invention provides an integrated search method for an artificial neural network and a computation accelerator architecture taking into account hard constraints, the method comprising: executing a NAS module that determines a neural network architecture; executing a HDX evaluation module that determines an accelerator architecture based on the determined neural network architecture and predicts a loss of the determined neural network architecture and a hardware metric related to the determined accelerator architecture; and executing a gradient adjustment module that adjusts the gradients of a loss function (Loss) and a constraint function (Const) based on a gradient descent method so that the predicted hardware metric moves in a direction that satisfies a specified constraint.

[0013] The NAS module execution step can use backpropagation to find paths within a SuperNet to generate the neural network architecture that is ultimately searched, and calculate the cross entropy loss (LossCE). The HDX evaluation module execution step may include a hardware generation network execution step of searching for optimal hardware based on the determined neural network architecture as the accelerator architecture and determining at least one of a PE (Processing Element) array configuration (PEx, PEy), a register file (RF) configuration, and a data flow (DF) configuration for the accelerator architecture, constructed through pre-learning, and a cost estimation network execution step of predicting the hardware metrics based on parameters for the neural network architecture and a configuration for the accelerator architecture. The cost estimation network implementation step may comprise a multi-layer perceptron with residual connections, through which delay time, area, and energy consumption are determined and the hardware metrics are predicted. The gradient adjustment module execution step can perform gradient adjustment of the loss function to select the neural network architecture in a direction that satisfies a specified constraint if the predicted hardware metric does not satisfy the constraint. The gradient adjustment module execution step may perform a gradient adjustment of a hardware cost function to select the accelerator architecture in a direction that satisfies a specified constraint if the predicted hardware metric does not satisfy the constraint. Effect of the Invention

[0014] The technology disclosed in this specification has the following effects. However, this does not mean that a specific embodiment must include all of the following effects or that it must include only the following effects, and therefore the scope of the disclosed technology should not be understood to be limited thereby.

[0015] According to the device and method for integrated search of an artificial neural network and a computation accelerator structure taking into account hard constraints of the present invention, it is possible to satisfy hard constraints such as throughput or energy consumption per second while performing an integrated search of an artificial neural network and a computation accelerator.

[0016] In addition, by using gradient descent to perform the search, an artificial neural network that represents the entire search space can be trained once to complete the search, enabling very fast search, and optimizing direct hardware metrics such as latency and energy consumption in a differentiable manner that reflects hard constraints.

[0017] Furthermore, by using hard constraints to guide the search in a direction that satisfies the hard constraints through gradient adjustment, a high-quality solution can be obtained without compromising the design goal. [Brief description of the drawings]

[0018] [Figure 1] FIG. 2 is a diagram illustrating a functional configuration of an integrated search device according to the present invention. [Diagram 2] 1 is a flow chart illustrating an example of a method for integrating an artificial neural network and a computation accelerator architecture that takes into account hard constraints according to the present invention. [Diagram 3] 13 shows experimental results of the loss function adjustment method considering hard constraints. [Figure 4] FIG. 2 is a diagram illustrating a method for jointly searching an artificial neural network and a computation accelerator structure taking into account hard constraints according to the present invention. [Diagram 5] 11A to 11C are diagrams illustrating experimental results according to the present invention. [Figure 6] 11A to 11C are diagrams illustrating experimental results according to the present invention. [Figure 7] FIG. 1 illustrates an example of a search network and accelerator design according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] The present invention was filed with the support of the following research project: [National research and development project that supported this invention] [Project unique number] 1711193986 [Project Number] 2020-0-01361-004 [Ministry name] Ministry of Science, ICT and Technology [Name of issue management (specialized) organization] Information and Communications Planning and Evaluation Agency [Research Project Name] Information, Communications, and Broadcasting Innovation Human Resource Development [Research title] Artificial Intelligence Graduate School Support (Yonsei University) [Contribution rate] 1 / 1 [Name of project implementation organization] Yonsei University Industry-Academic Cooperation Group [Research period] 2023.01.01~2023.12.31

[0020] The description of the present invention is merely an embodiment for structural or functional description, and therefore the scope of the present invention should not be construed as being limited by the embodiment described in the present specification. In other words, since the embodiment can be modified in various ways and can have various forms, the scope of the present invention should be understood to include equivalents that can realize the technical idea. In addition, the objectives or effects presented in the present invention do not mean that a specific embodiment should include all of them or should include only such effects, and therefore the scope of the present invention should not be understood as being limited thereby.

[0021] Meanwhile, the meanings of the terms used in this specification should be understood as follows.

[0022] The terms "first", "second", etc. are used to distinguish one component from another component, and should not be used to limit the scope of the rights. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0023] When referring to an element being "connected" to another element, it should be understood that it may be directly connected to the other element, but there may also be other elements in between. Conversely, when referring to an element being "directly connected" to another element, it should be understood that there are no other elements in between. Meanwhile, other expressions describing the relationship between elements, such as "between" and "immediately between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0024] Singular expressions should be understood to include plural expressions unless the context clearly dictates otherwise, and terms such as "comprise" or "have" should be understood to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof that are performed, but not to preclude the presence or additional possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0025] In each step, the identification numbers (e.g., a, b, c, etc.) are used for convenience of description, and the identification numbers do not describe the order of each step, and each step may occur in a different order than specified unless the context clearly dictates a specific order. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the opposite order.

[0026] The present invention is embodied as computer readable codes in a computer readable recording medium, and the computer readable recording medium includes all kinds of recording devices in which data read by a computer system is stored. Examples of the computer readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Also, the computer readable recording medium may be distributed among computer systems connected by a network, so that the computer readable codes may be stored and executed in a distributed manner.

[0027] All terms used herein, unless otherwise defined, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention pertains. Terms defined in commonly used dictionaries must be interpreted to be consistent with the meaning they have in the context of the relevant art, and cannot be interpreted as having an ideal or overly formal meaning unless expressly defined in this specification.

[0028] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, specific examples of the embodiments for carrying out the present invention will be described in detail with reference to the drawings.

[0029] FIG. 1 is a diagram illustrating the functional configuration of an integrated search device according to the present invention.

[0030] 1, the integrated search device 100 completes the search by training an artificial neural network once to represent the entire search space, enabling very fast search and optimizing direct hardware metrics such as latency and energy consumption in a differentiable manner that takes into account hard constraints. To achieve this, the integrated search device 100 is realized by including a Neural Architecture Search (NAS) module 110, a Hard-constrained Differentiable Co-Exploration (HDX) evaluation module 130, and a gradient adjustment module 150.

[0031] The NAS module 110 performs an operation of determining a neural network architecture, the HDX evaluation module 130 performs an operation of determining an accelerator architecture corresponding to the neural network architecture determined by the NAS module 110 and predicting hardware metrics related to the neural network architecture and accelerator architecture, and the gradient adjustment module 150 performs an operation of optimizing to satisfy hard constraints during the integrated search process of the NAS module 110 and the HDX evaluation module 130.

[0032] More specifically, the NAS module 110 uses backpropagation to find paths in the SuperNet to generate the final searched neural network architecture and computes the cross entropy loss (LossCE) for it.

[0033] In one embodiment, the HDX evaluation module 130 is constructed through pre-training and configured to include two networks: a hardware generation network and a cost estimation network.

[0034] First, the hardware generation network searches for an optimal hardware based on the neural network architecture determined by the NAS module 110 as an accelerator architecture, and performs an operation of determining at least one of a PE (Processing Element) array configuration (PEx, PEy), a register file (RF, Register File) configuration, and a data flow configuration for the accelerator architecture. That is, the hardware generation network is pre-trained to search for an optimal hardware architecture, and generates an optimal configuration for the optimal hardware architecture as a parameter. For example, the hardware generation network generates PE array features (PEx, PEy), a register file RF, a data flow DF, etc. for the optimal hardware architecture as an output.

[0035] In one embodiment, the hardware generative network uses the neural network architecture parameters as input and outputs an optimal hardware realization. The hardware generative network is independent of a particular cost function and is jointly trained during joint search to adapt to constraints. The hardware generative network determines an optimal accelerator structure from the neural network architecture, where the optimal accelerator structure varies depending on the target loss function and hard constraints. The hardware generative network is implemented to connect its output to the input of the cost estimation network and trains in the direction of optimizing the loss function.

[0036] The cost estimation network operates to predict hardware metrics based on the configuration for the accelerator architecture. In one embodiment, the cost estimation network predicts hardware metrics using the hardware generation network and the output of the NAS module 110 as inputs. The cost estimation network is pre-trained with the network and accelerator search space, for which MAESTRO, Timeloop, Accelergy, etc. are used. After pre-training, the cost estimation network predicts hardware costs (Cost HW In one embodiment, the cost estimation network consists of a multi-layer perceptron with residual connections.

[0037] In one embodiment, the cost estimation network predicts hardware metrics by determining latency, area, and energy consumption through a multi-layer perceptron, where the cost estimation network predicts hardware metrics by calculating a balanced weighted sum of the latency, area, and energy consumption using a cost function.

[0038] The gradient adjustment module 150 adjusts the gradients of the loss function (Loss) and the constraint function (Const) so that the hardware metric predicted based on the gradient descent method moves in a direction that satisfies a specified constraint. In one embodiment, if the predicted hardware metric does not satisfy a specified constraint, the gradient adjustment module 150 performs gradient adjustment of the loss function so as to select a neural network architecture in a direction that satisfies the constraint. The gradient adjustment module 150 calculates the adjustment direction via a pseudo-inverse matrix so that the angle between the gradient of the loss function and the gradient of the constraint function is 90 degrees or less.

[0039] In one embodiment, the gradient adjustment module 150 performs gradient adjustment of a hardware cost function instead of a loss function to select an accelerator architecture in the direction of satisfying a specified constraint if the predicted hardware metrics do not satisfy the constraint.

[0040] FIG. 2 is a flow chart illustrating an example of a method for jointly exploring artificial neural networks and computation accelerator architectures taking into account hard constraints according to the present invention.

[0041] 2, the integrated search device 100 determines a neural network architecture via the NAS module 110 (step S210). The integrated search device 100 determines an accelerator architecture based on the neural network architecture via the HDX evaluation module 130 (step S230). The integrated search device 100 predicts the loss of the neural network architecture and hardware metrics related to the accelerator architecture via the HDX evaluation module 130 (step S250). The integrated search device 100 adjusts the gradients of the loss function (Loss) and the constraint function (Const) via the gradient adjustment module 150 so that the hardware metrics predicted based on the gradient descent method move in a direction that satisfies the specified constraint (step S270).

[0042] Hereinafter, the method for jointly searching for an artificial neural network and a computation accelerator structure taking into account hard constraints according to the present invention will be described in more detail with reference to FIGS.

[0043] Neural Network Architecture Search (NAS) automates the design of DNN architectures to address the increasing network size and corresponding reactive design efforts. In neural network architecture search, reinforcement learning (RL) or evolutionary algorithms (EA) have been adopted in the early days to generate networks.

[0044] However, such algorithms can be very expensive to search, taking up to thousands of GPU-days due to the full training required for all candidates. Differentiable neural network architecture search creates a SuperNet in a way that alleviates this cost and finds paths within it. That is, differentiable neural network architecture search can find a network with state-of-the-art performance in many times less time. Nevertheless, optimizing only on network performance can be insufficient because it does not consider hardware efficiency. Some recent work to solve this problem attempts to reduce the size of the network for a delay time constraint by adding loss terms to consider hardware costs, and some use simple delay time models. However, they cannot be used for integrated search because only the design of the network is considered and the relationship between the network accuracy and the hardware structure is not reflected in the model.

[0045] Early work on integrated search leverages a variant of reinforcement learning or evolutionary algorithms to exploit simplicity. Each candidate network is trained for evaluation, and the accelerator design is analyzed for hardware efficiency. Such values ​​generate a compensation that the agent uses to generate the next candidate solution. However, all of these inherit the same problems as RL-based NAS methods, which require fairly costly training to evaluate each candidate solution. Also, integrated search requires a much larger network / hardware search space than searching networks alone.

[0046] In this regard, differentiable access methods have been adopted for the integrated search. Auto-NBA uses a differentiable accelerator search engine to build an integrated search pipeline, and DANCE trains an auxiliary neural network for hardware search and cost evaluation. However, they do not adequately address the problem of hard constraints. This invention proposes a holistic method to handle hard constraints for differentiable integrated search.

[0047] A very simple way to consider hard constraints in a differentiable joint search is to adjust the relative weighting of the hardware cost. For example, if we want to simultaneously consider accuracy (cross-entropy), latency, and energy, we can set the loss function as shown in Equation 1 below.

[0048]

number

[0049] Here, λ Cost By adjusting λ, it is possible to adjust which part of the function the solution is more concentrated on. Cost Increasing indirectly directs the search process to take the hardware metric into account more, but giving a larger penalty does not directly translate into a decrease in the constraint metric value.

[0050] Figure 3 clearly illustrates this problem. Cost The results are shown by adjusting λ from 0.001 to 0.010 and searching three times from each point. The average value of the three searches is shown by the thick circle. Cost Even if a slight trend is observed by, the discrepancy in both the direction and variance of the trajectories is more prevalent. Consider a scenario where a designer is trying to design a neural network-accelerator architecture pair with some constraints (e.g., 33.3 ms) on the latency using an existing joint search method, and the initial λ Cost However, we can try to search with λ and adjust the value in multiple search steps.Cost Such discrepancies and delays between the,constraints make it difficult to obtain a good solution, not to mention the,enormous time cost of making several searches.,It is necessary to design an effective strategy to solve,the hard constraint integration search problem.

[0051] The differentiable joint search with hard constraints is defined in Equation 2 below.

[0052]

number

[0053] where t denotes the current value of the constrained metric, such as delay or energy, and T denotes the target value (e.g., 33.3 ms for delay). α and β denote the neural network architecture parameters and the hardware accelerator configuration, respectively. w is the weight of the NAS supernet, and net(α) is the selected neural network architecture. eval(α,β) denotes the hardware metric evaluated for α and β. The goal of the joint search is to find the best neural architecture loss (Loss) defined by the user. NAS ) and hardware costs HW ) are expressed using two evaluation metrics:

[0054] Referring to Fig. 4, an integrated search method executed by the integrated search device 100 according to the present invention is illustrated. That is, the integrated search method according to the present invention corresponds to a differentiable integrated search method that takes into account hard constraints, called HDX (Hard-constrained Differentiable co-eXploration). The left side of Fig. 4(a) corresponds to a network search module (i.e., corresponding to the NAS module 110 in Fig. 1), which uses backpropagation to find a path in a supernet and generates a network to be finally searched.

[0055] The right part of FIG. 4(a) corresponds to an evaluator (i.e., corresponds to the HDX evaluation module 130 in FIG. 1) that searches for optimal hardware accelerator designs using the architecture parameters obtained from the network search module and evaluates the cost metric. The evaluator network eval() is the core of the differentiable joint search that considers the relationship between the hardware accelerators and allows the gradient to flow to the supernet. The evaluator network is composed of a hardware generation network gen() and an estimation network est(). The hardware generation network takes the neural architecture parameters as input and outputs the optimal hardware realization (i.e., corresponds to β in Equation 2). The hardware generation network is jointly trained during the joint search to adapt to the constraints without relying on a specific cost function.

[0056] The estimation network takes the output of the network search module and the generation network, and outputs hardware-related metrics. The estimation network is pre-trained by the network and accelerator search space. For this, traditional (non-differentiable) cost estimation frameworks such as MAESRTO, Timeloop, and Accelergy are used as ground truth. After pre-training, the estimation network is frozen during the search and is only used to infer the hardware cost given the network architecture. This converts the above Equation 2 into the following Equation 3.

[0057]

number

[0058] where v is the weight for the hardware generative network.

[0059] The present invention applies gradient adjustment to a differentiable joint search method as an effective solution to hard constraints. Gradient adjustment is often used to solve many goals, such as continuous functions or differential equations. Here, we apply gradient adjustment to a joint search to satisfy hard constraints. Referring to diagrams (b) and (c) of FIG. 4, the gradient adjustment method artificially generates a force that can pull the gradient in a direction that agrees with the constraint. The condition under which the method is applied to calculate a new gradient g is defined as Equation 4 below.

[0060]

number

[0061] Here, g Loss is the original gradient of the global loss function defined as in Equation 5 below, as shown in Equation 3 above.

[0062]

number

[0063] g Const is the gradient of the constraint loss, defined as Const=max(tT,0). Since t is a function of α, it is backpropagated to find the gradient with respect to α. In the ideal case where t≦T, the constraint is already satisfied, so we do nothing. If the constraint is still not satisfied, we compute the dot product of the two gradients to determine the match in that direction. g Loss ·g Const ≥ 0 (i.e., the angle between the two gradients is less than or equal to 90 degrees), it means that the gradient descent update contributes to satisfying the constraint. Therefore, it is interpreted as a match in direction and the same g Loss is used as is (corresponding to Fig. 4(b)). However, as shown in Fig. 4(c), in the case of mismatch (i.e., g Loss ·g Const <0), with a gradient of m α By forcing it to move in the direction of (m α +g Loss )·g Const≧0, and guarantees a reduction in the target cost after gradient descent. This is α ·g Const +g Loss ·g Const = δ, where δ ≥ 0 is a small value to ensure gradual movement in the direction that satisfies the constraint. To update α and w, we calculate the optimal m for α, the parameter of the network architecture. α Solve.

[0064] m α is parallel to the gradient of the hard constraints and its size is minimized to limit its impact on the gradient of the loss function. This can be easily computed via a pseudo-inverse and expressed as:

[0065]

number

[0066] To control the size of the pull, we use a small multiplication factor p>0 on δ. The policy for updating δ with p is:

[0067] There is some initial value δ0 for δ. If the target metric does not satisfy the constraints, δ is multiplied by 1+p to strengthen the pull (δ′=(1+p)δ), otherwise δ is reset to its initial value (δ′=δ0). Gradient descent is also used to train the weights v of the hardware generative network. Thus, m is trained in the same way. v * However, for updating the hardware generation network, Loss Instead of g CostHW This can also be applied to the case where there are many hard constraints, and in this case, it is expressed by the following Equation 7.

[0068]

number

[0069] Essentially, the hardware cost function is a balanced weighted sum of three hardware cost metrics: HW and is expressed as the following Equation 8.

[0070]

number

[0071] Controlling CE, CL, and CA sets conditions on how to measure the balance between each cost metric. To match the scale of these hyperparameters, we use mJ, ms, and μm for each cost. 2 The units used are:

[0072] The estimation and generation networks are modeled with a 5-layer perceptron with residual connections. To train the estimation network, we first construct a dataset by randomly sampling a set of 10.8M network accelerators in the search space (2.95e-9% of the total search space) and evaluate them on hardware metrics using Timeloop and Accelergy. Using this dataset, the estimation network is trained for 200 epochs with a batch size of 256. Weight updates are performed using an Adam optimizer with a learning rate of 1e-4. The accuracy of the estimation network is over 99% in all metrics, making it powerful enough to be used as an engine for joint search. The generation network is randomly initialized and jointly trained with the NAS supernet. As gradients adjusted with hard constraints are backpropagated, the generation network learns how to generate accelerators that satisfy the constraints of a given neural network architecture.

[0073] The search space is constructed with ProxylessNAS as the backbone. It consists of various configurations of MBConv tasks with kernel sizes {3,5,7} and expansion ratios {3,6}. The total number of layers is 18 and 21 for CIFAR-10 and ImageNet datasets, respectively. The invention is orthogonal to the NAS realization and is selected among differentiable NAS algorithms such as DARTS or OFA. Eyeriss is used as the backbone architecture of the accelerator. It consists of a two-dimensional PE array with each PE having a MAC (Multiply-Accumulate) unit connected to a register file. Thus, the design space of the hardware accelerator includes PE array sizes from 12x8 to 20x24, and register file sizes per PE from 16B to 256B. The search space also includes WS (Weight-Stationary), OS (Output-Stationary), and RS (Row-Stationary) data flows.

[0074] The results of experiments relating to the present invention will now be described.

[0075] <<EXPERIMENTS>>

[0076] <Experimental Environment>

[0077] Experiments are performed using the CIFAR-10 and ImageNet datasets. For all reported hardware metrics (latency, energy, and chip area), we use direct evaluation on the hardware designed with a time loop and accelerators instead of the values ​​output by the cost estimation network to prevent as many errors as possible in the trained model. Network accuracy is evaluated by training the final network architecture from scratch for 300 epochs using a batch size of 64. We use the SGD optimizer with Nesterov momentum, cosine learning rate scheduling with an initial value of 0.008, and weight decay terms and momentum of 1e-3 and 0.9, respectively. Augmentations on the training data are taken from AutoAugment for all of the training data on both datasets.

[0078] <Comparison with Existing Methods>

[0079] Table 1 below compares existing differentiable integrated search methods with HDX.

[0080] [Table 1]

[0081] The methods compared here are as follows:

[0082] NAS → HW: Only differentiable general NAS, followed by an exhaustive hardware search using a Timeloop.

[0083] Auto-NBA: A differentiable integrated search method that uses gradient descent to directly search for hardware parameters, and represents the relationship between the hardware and the DNN as a lookup table.

[0084] DANCE: A hard-constrained, differentiable integrated search method in which the hardware generation network and the cost estimation network are modeled as neural networks.

[0085] DANCE+Soft Constraint: To show a rational access method that considers constraints in existing solutions, we use the term λ for soft constraints. Soft Added max(t / T-1,0).

[0086] This is the proposed method for HDX:p=1e-2.

[0087] As shown in Table 1 above, HDX is the only method that considers hard constraints along with a differentiable joint search. For quantitative comparison, we devised an algorithm that finds constrained solutions using joint search without hard constraints. Since a constrained metric (e.g., latency) that is too low often means an inefficient solution (in terms of other metrics (e.g., accuracy and energy)), we set a criterion of having solutions that are 50%-100% of the target constraints. A rough sketch of the algorithm, similar to a binary search, is as follows:

[0088] (1) A control parameter (λ) that indirectly affects the metric through constraints (e.g., delay time). Soft and λ Cost ).

[0089] (2) A search is performed using basic control parameter values.

[0090] (3) The control parameters are doubled at each step and the search is continued until the metric falls within the constraint.

[0091] (4) If the metric is less than 50% of the target value, reduce the control parameters using a binary search method.

[0092] This problem cannot be solved exactly by binary search due to the large search-to-search variation and nonlinear relationships. In the actual implementation, more features were added to handle corner cases and avoid falling into erroneous parameter regions (caused by search-to-search variation). We ran the algorithm 100 times and reported the average, and also reported the average error as a proxy for solution quality.

[0093] As shown in Table 1 above, all baselines require multiple searches to find a solution that satisfies the constraints. In the case of DANCE, it takes 6.6 searches with an average of 12.2 GPU time, and even with soft constraints, it takes 4.9 searches and 9.1 GPU time. In contrast, HDX always searches for a solution that satisfies the constraints in a single search. Also, the number of baseline searches varies greatly depending on the selection of the base values ​​of the control parameters. It is difficult to guess the correct initial parameters, which incurs a larger iteration cost. This justifies the need for HDX.

[0094] <Co-exploration Results>

[0095] Figure 5 shows the experimental results of the joint search of various techniques on the CIFAR-10 dataset in a two-dimensional plot. The horizontal axis is the accuracy of the artificial neural network, and the vertical axis shows the delay time on the left and middle axes, and all the hardware factors of the loss function on the right axis. We conducted experiments with two delay times: 30 frames (33.3 ms) and 60 frames (16.6 ms). For all joint search methods, λ Cost We varied from 0.001 to 0.005 and obtained five solutions to provide a fair comparison of the various design points obtained from each access method.

[0096] NAS→HW refers to a methodology (individual design) in which an artificial neural network is designed first, rather than an integrated design, and the accelerator structure is derived from this. DANCE and Auto-NBA are backpropagation-based methodologies that do not consider existing hard constraints and utilize loss function techniques. Finally, HDX refers to the technique proposed in this invention. In the case of NAS→HW, λ Cost Since cannot be directly applied, we obtain 10 solutions with diverse ranges as reference. For DANCE and Auto-NBA, the black markers are unconstrained and the color markers are obtained with soft constraints on the hue.

[0097] In all experiments, we use CE=2.9, CL=6.2, and CA=1.0 in Equation 8 to make the difference scale of each metric almost equal for a fair comparison. HW was used in the hardware design step.

[0098] The left and middle panels of Fig. 5 show the relationship between error and latency, with the colored bars indicating the two latency targets. Cost It is easy to see that regardless of the value of , all solutions found by HDX meet the given hard constraints. Also, all solutions have latency just below the constraints, indicating that the solutions are not over-optimized for the limited metric (latency). DANCE and Auto-NBA exploit a compromise between hardware metric and accuracy, but do not have control over satisfying the constraints. Even using soft constraints, they mostly fail to obtain solutions within the constraints.

[0099] In the case of individual designs, due to the excessive emphasis on accuracy in practice, hardware-related goals were not considered. For both the 30-frame and 60-frame cases using the loss-function-based methodology, many hard constraint goals could not be achieved. In particular, in the case of 30 frames, although the accidental goals were achieved in many cases, in the case of difficult constraints such as 60 frames, the goals could not be achieved despite the modification of the loss function. This is because the loss function only provides a slight guide for optimization and cannot enforce substantial hard constraints.

[0100] The right plot shows the optimization quality of the loss function by dividing it into two parts: accuracy and hardware numerical value. The lower the value on the left side, the better the result. As can be easily confirmed from the figure, the proposed technique is not lagging behind in the quality of the loss function compared to the existing techniques, and in fact, there are cases where it is even better, indicating that this method does not achieve hard constraints at the expense of quality.

[0101] <HDXが見つけた解の品質(Solution Quality Found by HDX)>

[0102] The HDX method proposed in the present invention shows that 1) it can handle all constraint conditions of three metrics (delay time, energy, and chip area), 2) it can handle various constraint conditions, and 3) generally, solutions of excellent quality can be obtained. In the case on the right side of Figure 5, Cost HW and error are both displayed to evaluate the quality of the solution in terms of Pareto optimization. In the cases on the left and middle of Figure 5, since other metrics are overlooked, Cost HW must be compared fairly by comparing them together.

[0103] The plot shows that the quality of the solutions from HDX is better than the NAS→HW method and does not degrade compared to existing integrated search methods. In fact, the tightly constrained (16.6ms) solution is able to find better solutions in terms of Pareto optimization than existing solutions. To study the quality of the solutions found by HDX in more detail, we performed another set of experiments. The solutions obtained by the existing integrated design method, the DANCE method, are called "Anchor" and are listed in Table 2 below.

[0104] [Table 2]

[0105] Table 2 above shows that the proposed method can satisfy hard constraints and perform integrated design in various settings. Using the CIFAR-10 dataset, hard constraints were designed based on the solution of "Anchor" obtained by the existing integrated design method. Among them, one or all three hardware metrics to be fixed as hard constraints were selected, and integrated search was performed using HDX. For example, in the case of A, the delay time is 69 ms and the energy consumption is 37 mJ. By setting the hard constraint for the delay time to the same 69 ms and then performing integrated design using the proposed method, it is possible to determine 1) whether the hard constraints can be satisfied, and 2) whether a solution of the same / similar quality can be derived as the solution designed without the hard constraint. In fact, when designed with a delay time of 69 ms, a solution of 43.99 ms was obtained, and the overall loss was also lower, indicating that the quality was not inferior. In addition, when hard constraints were applied to all of the delay time, energy, and chip area, it was observed that an excellent solution was obtained while successfully satisfying all constraints.

[0106] <Results from ImageNet Dataset>

[0107] Table 3 below shows the results of two integrated designs for ImageNet under the constraint of 125 ms.

[0108] [Table 3]

[0109] As shown in Table 3, the HDX method proposed in the present invention is the only one that satisfies all the constraints in both cases, and it can be confirmed from the loss value that the quality of the solution is the best among the solutions that satisfy the constraints. In addition, the Top-1 error and global loss show that the quality of the solution found by HDX is not at all impaired compared to DANCE or its variants.

[0110] <Sensitivity Study on Pulling Magnitude>

[0111] Referring to FIG. 6, the response of the proposed HDX method to the only adjustable factor p is shown, indicating that the performance of the proposed technique is not the result of arbitrarily adjusting p.

[0112] Figure 6 (a)-(c) show how the global loss and delay time change in a search with a limited delay time (33.3 ms) for various p, 1e-2, 7e-3, and 4e-3, respectively. Although there are differences in that the constraint is satisfied a little later when p is small and a little earlier when p is large, it also shows that the constraint is ultimately satisfied in all cases and the delay time is not significantly lower than the constraint.

[0113] That is, the curves for the constrained values ​​show similar trends regardless of the value of p. Initially, the global loss is primarily optimized while the delay time is kept constant. At this step, the pulling magnitude δ (see Equation 6) is still increasing and is not strong enough to make meaningful changes. At a certain point, δ becomes strong enough to pull the solution in the direction of decreasing the delay time. Once the delay time meets the constraint, the global loss begins to decrease while maintaining the delay time. There is no significant difference between the global loss and the final solution for the delay time, which indicates that the HDX method proposed in this invention is insensitive to the adjustable factor p.

[0114] <Analysis on the Searched Solutions>

[0115] Referring to Fig. 7, (a) and (b) show example neural network and accelerator structures obtained under delay time constraints of 60 and 30 frames, respectively, where (m,n) denotes an MBConv block with m × m kernel and n expansion ratio.

[0116] In the case of 60 frames in Figure 7(a), since the latency constraint is strong, the neural network mostly has small kernels and instead increases the number of layers to ensure a trade-off between latency and accuracy. To meet the strict constraints while maintaining accuracy, the network mainly has small 3x3 kernels, and using smaller kernels can reduce the number of multiplications quadratically. Therefore, reducing the kernel size and increasing the number of layers is a good choice to reduce inference latency. Accelerators pursue latency reduction by utilizing a weight stationary (WS) structure, which is known to be fast, along with a large number of PEs. Looking at the accelerator design, to achieve low latency, a relatively large PE array (16x16) is used and WS data flow, which is known to have low latency, is required. In addition, the network has some kernels with a high channel expansion ratio. WS has an advantage over found networks because it utilizes parallelism for fast execution.

[0117] In contrast, for the neural network obtained under the constraint of 30 frames in Figure 7(b), since there is more time, many large kernels are selected, which are advantageous for increasing accuracy, and the accelerator uses a row stationary (RS) structure, which is known to be energy efficient. In the design for 30 fps, the design settles on a solution that optimizes energy consumption while satisfying the constraints. This design uses larger kernels in the network and RS dataflow in the accelerator. RS is known to be energy efficient and exploits parallelism in the spatial dimension and activations of the kernel. Therefore, using larger kernels has an advantage over the RS dataflow. To reduce energy consumption, the design has fewer PEs (12 x 8), a larger RF to save off-chip access energy, and fewer layers in the network.

[0118] We propose HDX, an integrated search method that is differentiable under hard constraints, and can stably satisfy strict constraints with high-quality solutions by conditionally applying gradient operations that move the solution in a direction that satisfies the constraints.The integrated search method of the present invention can significantly reduce the burden of computational load in the integrated design of reinforcement learning-based neural network / accelerator that takes hard constraints into account.

[0119] Although the embodiments of the present invention have been described in detail above with reference to the drawings, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical spirit of the present invention. [Explanation of symbols]

[0120] 100 Integrated Search Device 110 NAS Module 130 HDX Evaluation Module 150 Gradient Adjustment Module

Claims

1. A Neural Architecture Search (NAS) module that determines a neural network architecture; a High-constrained Differentiable Co-Exploration (HDX) evaluation module that determines an accelerator architecture according to the determined neural network architecture and predicts losses of the determined neural network architecture and hardware metrics for the determined accelerator architecture; A gradient adjustment module that adjusts the gradient of a loss function (Loss) and a constraint function (Const) so that the predicted hardware metric moves in a direction that satisfies a specified constraint based on a gradient descent method; The gradient adjustment module includes: If the predicted hardware metrics do not satisfy a specified constraint, adjusting the gradient of the loss function to select the neural network architecture in a direction that satisfies the constraint; The integrated search device of the artificial neural network and the calculation accelerator structure considering the hard constraints is characterized in that the adjustment direction is calculated through a pseudo-inverse matrix so that the angle between the gradient of the loss function and the gradient of the constraint function is 90 degrees or less.

2. The integrated search device for artificial neural networks and computation accelerator structures taking into account hard constraints as claimed in claim 1, wherein the NAS module uses backpropagation to find a path within a supernet to generate the neural network architecture that is ultimately searched, and calculates a cross entropy loss (LossCE).

3. The HDX evaluation module includes: a hardware generation network that is constructed through pre-learning, searches for an optimal hardware according to the determined neural network architecture as the accelerator architecture, and determines at least one of a PE (Processing Element) array configuration (PEx, PEy), a register file (RF) configuration, and a data flow (DF) configuration for the accelerator architecture; and a cost estimation network that predicts the hardware metrics based on parameters related to the neural network architecture and configurations related to the accelerator architecture.

4. 4. The integrated search device for artificial neural networks and computation accelerator structures taking into account hard constraints as claimed in claim 3, wherein the cost estimation network is composed of a multilayer perceptron with residual connections, and the delay time, area, and energy consumption are determined through the multilayer perceptron to predict the hardware metrics.

5. The integrated search device for artificial neural networks and computation accelerator structures taking into account hard constraints as claimed in claim 4, wherein the cost estimation network predicts the hardware metric by calculating a balanced weighted sum of the delay time, area, and energy consumption as a cost function.

6. executing a NAS module to determine a neural network architecture; executing an HDX evaluation module to determine an accelerator architecture according to the determined neural network architecture and to predict losses of the determined neural network architecture and hardware metrics for the determined accelerator architecture; A gradient adjustment module execution step for adjusting the gradient of a loss function (Loss) and a constraint function (Const) so that the predicted hardware metric moves in a direction that satisfies a specified constraint based on a gradient descent method; The gradient adjustment module execution step includes: If the predicted hardware metrics do not satisfy a specified constraint, adjusting the gradient of the loss function to select the neural network architecture in a direction that satisfies the constraint; The method for integrated search of an artificial neural network and a computation accelerator structure taking into account hard constraints is characterized in that an adjustment direction is calculated through a pseudo-inverse matrix so that an angle between the gradient of the loss function and the gradient of the constraint function is 90 degrees or less.

7. 7. The method of claim 6, wherein the NAS module execution step uses backpropagation to find a path in a supernet to generate the neural network architecture that is ultimately searched, and calculates a cross entropy loss (LossCE).

8. The HDX evaluation module execution step includes: A hardware generation network execution step of searching for an optimal hardware based on the determined neural network architecture constructed through pre-learning as the accelerator architecture and determining at least one of a PE (Processing Element) array configuration (PEx, PEy), a register file (RF) configuration, and a data flow (DF) configuration for the accelerator architecture; and executing a cost estimation network to predict the hardware metrics based on parameters related to the neural network architecture and configurations related to the accelerator architecture.

9. 9. The method for integrated exploration of an artificial neural network and a computation accelerator structure taking into account hard constraints as claimed in claim 8, wherein the cost estimation network execution step is configured with a multilayer perceptron with residual connections, and the delay time, area, and energy consumption are determined through the multilayer perceptron, and the hardware metrics are predicted.

Images