Artificial neural network and computational accelerator structure co-exploration apparatus and method with hard constraints

Abstract

The present invention relates to an apparatus and method for integrated search of an artificial neural network and a computational accelerator structure considering rigidity constraints. The apparatus comprises: a NAS (Neural Architecture Search) module for determining a neural network architecture; an HDX (High-constrained Differentiable Co-Exploration) evaluation module for determining an accelerator architecture based on the determined neural network architecture and predicting the loss of the determined neural network architecture and hardware metrics related to the determined accelerator architecture; and a gradient control module for adjusting the gradients of a loss function (Loss) and a constraint function (Const) based on gradient descent so that the predicted hardware metrics proceed in a direction that satisfies specified constraints. Accordingly, the present invention enables very fast search by completing the search by training an artificial neural network representing the entire search space only once, by conducting the search using gradient descent, and allows for the optimization of direct hardware metrics such as latency or energy consumption in a differentiable manner with rigidity constraints applied.

Description

Technology Field

[0001] The present invention relates to an integrated search technology for artificial neural networks and dedicated hardware accelerators, and more specifically, to an integrated search apparatus and method for an artificial neural network and a computational accelerator structure that considers rigidity constraints, such as throughput per second or energy consumption, while integrating the search of an artificial neural network and a computational accelerator. Background Technology

[0003] While the primary focus of most Deep Neural Network (DNN) research has been application performance (i.e., accuracy), this has also led to a rapid growth in network size, requiring massive computational resources for execution. In recent years, numerous frameworks have emerged to mitigate resource issues, most falling under the categories of network and hardware optimization. Network optimization refers to improving neural network architectures to reduce computation while maintaining similar accuracy, while hardware optimization involves improving DNN execution efficiency by utilizing optimized hardware known as accelerators. Unfortunately, efforts on one side often hinder the benefits of the other. For example, it is known that running MobileNet, which was developed for high speed, on the famous accelerator TPU results in slower performance than running a standard neural network; this is because MobileNet's characteristic of reducing the number of channels directly contradicts the TPU's acceleration strategy.

[0004] For this reason, when the task to be performed on the accelerator is already known, it is necessary to design the accelerator and the artificial neural network simultaneously so that they can create synergy.

[0005] Existing integrated design methodologies can be broadly divided into two categories. The first is based on reinforcement learning (RL) or evolutionary algorithms and can be considered close to traditional integrated design methods. This method derives a solution with an arbitrary accelerator or neural network structure, evaluates it, and then seeks a better solution based on the evaluation results. Traditionally, quantization approximation algorithms and genetic algorithms were used to find better solutions, but since the popularization of deep learning, reinforcement learning has been used.

[0006] However, this method has the disadvantage that the process of evaluating a solution takes a very long time whenever a solution is derived. In the case of artificial neural networks, a training process is required to evaluate their accuracy; it is known that training a single neural network usually takes several weeks, and even with parallelization using a large number of servers, it is not easy to handle the massive amount of computation.

[0007] To overcome this, a backpropagation-based methodology has recently been proposed. Based on neural network search methods, this approach involves designing the solution space of the neural network and accelerator to be differentiable and then placing learnable multiplicative factors on each differentiable path. Once training is complete, the path containing the factor with the largest value becomes the final solution, and the design of the neural network and accelerator is derived based on this. In the case of the neural network, the accuracy is evaluated by training the structure created in this way once.

[0008] This approach shares the drawbacks of most deep learning-based methodologies, which stem from the fact that backpropagation relies on a single loss function. While this poses no significant problem when there is only one desired objective, it is difficult to strike the desired balance when there are multiple objectives; in particular, it is virtually impossible to incorporate them when there are strong constraints on specific objectives. Prior art literature

[0010] Korean Patent Publication No. 10-2019-0101677 (2019.09.02) The problem to be solved

[0011] One embodiment of the present invention aims to provide an integrated search device and method for an artificial neural network and a computational accelerator structure that considers rigidity constraints, such as throughput per second or energy consumption, while integrating the search of an artificial neural network and a computational accelerator.

[0012] One embodiment of the present invention aims to provide an integrated search device and method for an artificial neural network and computational accelerator structure that considers rigidity constraints, which enables very fast search by completing the search by training an artificial neural network representing the entire search space only once by performing the search using gradient descent, and can optimize direct hardware metrics such as latency or energy consumption in a differentiable manner that reflects rigidity constraints.

[0013] One embodiment of the present invention aims to provide an integrated search device and method for an artificial neural network and a computational accelerator structure that considers stiffness constraints, thereby enabling high-quality solutions to be obtained without compromising design goals by inducing the search to proceed in a direction that satisfies the stiffness constraints through slope adjustment according to the stiffness constraints. means of solving the problem

[0015] Among the embodiments, the artificial neural network and computational accelerator architecture integrated search device considering rigidity constraints includes: a NAS (Neural Architecture Search) module for determining a neural network architecture; an HDX (High-constrained Differentiable Co-Exploration) evaluation module for determining an accelerator architecture according to the determined neural network architecture and predicting the loss of the determined neural network architecture and hardware metrics regarding the determined accelerator architecture; and a gradient control module for adjusting the gradients of a loss function (Loss) and a constraint function (Const) based on gradient descent so that the predicted hardware metrics proceed in a direction that satisfies specified constraints.

[0016] The above NAS module uses backpropagation to find paths within the SuperNet, generates the neural network architecture that is ultimately searched, and cross-entropy loss (Loss CE ) can be produced.

[0017] The above HDX evaluation module may include a hardware generation network that is built through prior learning and searches for optimal hardware according to the determined neural network architecture as the accelerator architecture, and determines at least one of a PE (Processing Element) array configuration (PEx, PEy), a register file (RF) configuration, and a data flow (DF) configuration regarding the accelerator architecture; and a cost estimation network that predicts the hardware metric based on parameters regarding the neural network architecture and configurations regarding the accelerator architecture.

[0018] The above cost estimation network is composed of a multilayer perceptron with residual connections, and the hardware metric can be predicted by determining the latency, area, and energy consumption through the multilayer perceptron.

[0019] The above cost estimation network can predict the hardware metric by calculating a balanced weighted sum regarding the latency, area, and energy consumption as a cost function.

[0020] The gradient adjustment module can perform gradient adjustment of the loss function to select the neural network architecture in a direction that satisfies the constraint when the predicted hardware metric does not satisfy the specified constraint.

[0021] The slope control module can calculate the control direction through a pseudo-inverse so that the angle between the slope of the loss function and the slope of the constraint function is 90 degrees or less.

[0022] The gradient control module can perform gradient control of the hardware cost function to select the accelerator architecture in a direction that satisfies the constraint when the predicted hardware metric does not satisfy the specified constraint.

[0023] Among the embodiments, the method for integrated search of an artificial neural network and a computational accelerator structure considering rigidity constraints comprises: a NAS module execution step for determining a neural network architecture; an HDX evaluation module execution step for determining an accelerator architecture according to the determined neural network architecture and predicting the loss of the determined neural network architecture and hardware metrics regarding the determined accelerator architecture; and a gradient adjustment module execution step for adjusting the gradients of a loss function (Loss) and a constraint function (Const) based on gradient descent so that the predicted hardware metrics proceed in a direction that satisfies specified constraints.

[0024] The above NAS module execution step uses backpropagation to find paths within the SuperNet, generates the neural network architecture that is finally searched, and cross-entropy loss (Loss CE ) can be produced.

[0025] The above HDX evaluation module execution step may include a hardware generation network execution step that searches for optimal hardware according to the neural network architecture determined through prior learning as the accelerator architecture and determines at least one of the PE (Processing Element) array configuration (PEx, PEy), register file (RF) configuration and data flow (DF) configuration regarding the accelerator architecture; and a cost estimation network execution step that predicts the hardware metric based on parameters regarding the neural network architecture and configurations regarding the accelerator architecture.

[0026] The above cost estimation network execution step is composed of a multilayer perceptron with residual connections, and the hardware metric can be predicted by determining the latency, area, and energy consumption through the multilayer perceptron.

[0027] The gradient adjustment module execution step described above can perform gradient adjustment of the loss function to select the neural network architecture in a direction that satisfies the constraint when the predicted hardware metric does not satisfy the specified constraint.

[0028] The above gradient adjustment module execution step can perform gradient adjustment of the hardware cost function to select the accelerator architecture in a direction that satisfies the constraint when the above predicted hardware metric does not satisfy the specified constraint. Effects of the invention

[0030] The disclosed technology may have the following effects. However, this does not mean that a specific embodiment must include all of the following effects or only the following effects; therefore, the scope of the rights of the disclosed technology should not be understood as being limited by this.

[0031] The integrated search device and method for an artificial neural network and a computational accelerator structure considering stiffness constraints according to the present invention can satisfy stiffness constraints, such as throughput per second or energy consumption, while integratedly searching the artificial neural network and the computational accelerator.

[0032] An integrated search device and method for an artificial neural network and a computational accelerator structure considering stiffness constraints according to one embodiment of the present invention enables very fast search by completing the search by training an artificial neural network representing the entire search space only once by performing the search using gradient descent, and can optimize direct hardware metrics such as latency or energy consumption in a differentiable manner that reflects stiffness constraints.

[0033] An integrated search device and method for an artificial neural network and a computational accelerator structure considering stiffness constraints according to one embodiment of the present invention can obtain a high-quality solution without compromising the design goal by inducing the search to proceed in a direction that satisfies the stiffness constraints through slope adjustment according to the stiffness constraints. Brief explanation of the drawing

[0035] FIG. 1 is a diagram illustrating the functional configuration of an integrated search device according to the present invention. FIG. 2 is a flowchart illustrating an embodiment of a method for integrating an artificial neural network and a computational accelerator structure considering stiffness constraints according to the present invention. Figure 3 is a diagram showing the experimental results of a loss function control method considering stiffness constraints. FIG. 4 is a diagram illustrating an integrated search method for an artificial neural network and a computational accelerator structure considering stiffness constraints according to the present invention. Figures 5 and 6 are drawings illustrating experimental results according to the present invention. FIG. 7 is a drawing illustrating an embodiment of a search network and accelerator design according to the present invention. Specific details for implementing the invention

[0036] The description of the present invention is merely an example for structural or functional explanation, and therefore the scope of the present invention should not be interpreted as being limited by the examples described in the text. That is, since the examples are subject to various modifications and may take various forms, the scope of the present invention should be understood to include equivalents capable of realizing the technical concept. Furthermore, the objectives or effects presented in the present invention do not imply that a specific example must include all of them or only such effects; therefore, the scope of the present invention should not be understood as being limited by them.

[0037] Meanwhile, the meaning of the terms described in this application should be understood as follows.

[0038] Terms such as "first," "second," etc., are intended to distinguish one component from another, and the scope of rights shall not be limited by these terms. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0039] When it is stated that one component is "connected" to another component, it should be understood that it may be directly connected to that other component, or that there may be other components in between. Conversely, when it is stated that one component is "directly connected" to another component, it should be understood that there are no other components in between. Meanwhile, other expressions describing the relationships between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0040] A singular expression should be understood to include a plural expression unless the context clearly indicates otherwise, and terms such as "include" or "have" are intended to specify the existence of the implemented features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood not to preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0041] In each step, identifiers (e.g., a, b, c, etc.) are used for convenience of explanation and do not describe the order of the steps; the steps may occur differently from the specified order unless a specific order is clearly indicated in the context. That is, the steps may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

[0042] The present invention may be implemented as computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Additionally, the computer-readable recording medium may be distributed across networked computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0043] Unless otherwise defined, all terms used herein have the same meaning as generally understood by those skilled in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having meanings consistent with the context of the relevant technology and should not be interpreted as having an ideal or overly formal meaning unless explicitly defined in this application.

[0045] FIG. 1 is a diagram illustrating the functional configuration of an integrated search device according to the present invention.

[0046] Referring to FIG. 1, the integrated search device (100) can complete the search by training an artificial neural network representing the entire search space once, thereby enabling very fast search and optimizing direct hardware metrics such as latency or energy consumption in a differentiable manner that considers rigidity constraints. As a configuration for this, the integrated search device (100) may be implemented by including a NAS (Neural Architecture Search) module (110), an HDX (Hard-constrained Differentiable Co-Exploration) evaluation module (130), and a gradient control module (150).

[0047] The NAS module (110) can perform an operation to determine a neural network architecture, the HDX evaluation module (130) can perform an operation to determine an accelerator architecture corresponding to the neural network architecture determined by the NAS module (110) and to predict hardware metrics regarding the neural network architecture and the accelerator architecture, and the gradient control module (150) can perform an operation to optimize to satisfy rigidity constraints during the integrated search process of the NAS module (110) and the HDX evaluation module (130).

[0048] More specifically, the NAS module (110) can use backpropagation to find a path within the SuperNet and generate a neural network architecture that is ultimately searched, and the cross-entropy loss (Loss) regarding this CE ) can be produced.

[0049] In one embodiment, the HDX evaluation module (130) can be built through prior learning and can be configured to include two networks. That is, the HDX evaluation module (130) can include a hardware generation network and a cost estimation network.

[0050] First, the hardware generation network can search for an optimal hardware as an accelerator architecture according to the neural network architecture determined by the NAS module (110), and perform an operation to determine at least one of the PE (Processing Element) array configuration (PEx, PEy), register file (RF, Register File) configuration, and data flow configuration regarding the accelerator architecture. That is, the hardware generation network can be pre-trained to search for an optimal hardware architecture and can generate optimal configurations regarding the optimal hardware architecture as parameters. For example, the hardware generation network (131) can generate features of the PE array PEx, PEy and the register file RF, data flow DF, etc. regarding the optimal hardware architecture as outputs.

[0051] In one embodiment, the hardware generative network can output an optimal hardware implementation using neural network architecture parameters as input. The hardware generative network can be jointly trained during integrated search to adapt to constraints without relying on a specific cost function. The hardware generative network can determine the optimal accelerator structure from the neural network architecture, wherein the optimal accelerator structure may vary depending on the target loss function and stiffness constraints. The hardware generative network is implemented to connect its output as the input to a cost estimation network so that it can be trained in a direction that optimizes the loss function.

[0052] The cost estimation network can perform the operation of predicting hardware metrics based on configurations regarding the accelerator architecture. In one embodiment, the cost estimation network can predict hardware metrics by taking the output of the hardware generation network and the NAS module (110) as inputs. The cost estimation network can be pre-trained according to the network and the accelerator search space, and for this purpose, MAESTRO, Timeloop, and Accelergy, etc., may be used. After pre-training, the cost estimation network, given the hardware cost (Cost HW ) can be estimated. In one embodiment, the cost estimation network may be composed of a multilayer perceptron with residual connections.

[0053] In one embodiment, the cost estimation network can predict hardware metrics by determining latency, area, and energy consumption through a multilayer perceptron. In this case, the cost estimation network can predict hardware metrics by calculating a balanced weighted sum of latency, area, and energy consumption as a cost function.

[0054] The gradient control module (150) can control the gradients of the loss function (Loss) and the constraint function (Const) so that the hardware metric predicted based on gradient descent proceeds in a direction that satisfies the specified constraint. In one embodiment, the gradient control module (150) can control the gradient of the loss function to select a neural network architecture that satisfies the constraint when the predicted hardware metric does not satisfy the specified constraint. The gradient control module (150) can calculate the control direction through a pseudo-inverse so that the angle between the gradient of the loss function and the gradient of the constraint function is 90 degrees or less.

[0055] In one embodiment, the gradient control module (150) may perform gradient control of the hardware cost function instead of the loss function to select an accelerator architecture that satisfies the constraint when the predicted hardware metric does not satisfy the specified constraint.

[0057] FIG. 2 is a flowchart illustrating an embodiment of an integrated search method for an artificial neural network and a computational accelerator structure considering stiffness constraints according to the present invention.

[0058] Referring to FIG. 2, the integrated search device (100) can determine the neural network architecture through the NAS module (110) (step S210). The integrated search device (100) can determine the accelerator architecture according to the neural network architecture through the HDX evaluation module (130) (step S230). The integrated search device (100) can predict the loss of the neural network architecture and hardware metrics regarding the accelerator architecture through the HDX evaluation module (130) (step S250). The integrated search device (100) can adjust the gradients of the loss function (Loss) and the constraint function (Const) through the gradient adjustment module (150) so that the predicted hardware metrics based on gradient descent proceed in a direction that satisfies the specified constraints (step S270).

[0060] Hereinafter, with reference to FIGS. 3 to 7, an integrated search method for an artificial neural network and a computational accelerator structure considering stiffness constraints according to the present invention will be explained in more detail.

[0061] Neural Architecture Search (NAS) can automate the design of DNN architectures to address the increasing size of networks and the corresponding manual design effort. In the early stages of Neural Architecture Search, reinforcement learning (RL) or evolutionary algorithms (EA) have been adopted for network generation.

[0062] However, these algorithms can have very high search costs and can require up to thousands of GPU days due to the full training required for all candidates. Differentiable neural architecture search mitigates these costs by creating a SuperNet and finding paths within it. In other words, differentiable neural architecture search can find state-of-the-art networks in several times less time. Nevertheless, optimizing solely based on network performance may be insufficient because it does not consider hardware efficiency. Some recent work to address this issue incorporates hardware costs by adding loss conditions, while others attempt to reduce network size under latency constraints using simple latency models. However, these methods cannot be used for integrated search because they only consider network design and do not reflect the relationship between network accuracy and hardware architecture in the model.

[0064] Initial tasks for integrated search leverage simplicity by utilizing variations of reinforcement learning or evolutionary algorithms. Each candidate network is trained for evaluation, and accelerator designs are analyzed for hardware efficiency. These values ​​generate the rewards that the agent uses to produce the next candidate solution. However, these all inherit the same problems from RL-based NAS methods, which require costly training to evaluate each candidate solution. Furthermore, integrated search requires a larger network / hardware search space than searching only networks.

[0065] In this regard, a differentiable approach has been adopted for integrated search. Auto-NBA constructed an integrated search pipeline using a differentiable accelerator search engine, and DANCE trained an auxiliary neural network for hardware search and cost evaluation. However, they did not adequately address the stiffness constraint problem. This invention proposes a holistic method for handling stiffness constraints in differentiable integrated search.

[0067] A very simple way to consider rigidity constraints within differentiable integrated search is to adjust the relative weights for hardware costs. For example, when considering accuracy (cross-entropy), latency, and energy simultaneously, a loss function can be set up as shown in Equation 1 below.

[0068] [Mathematical Formula 1]

[0069]

[0070] Here, By adjusting it, it is possible to control which part of the function the solution focuses on more. Increasing it can indirectly instruct the search process to consider hardware metrics more, but giving a larger penalty does not directly lead to a decrease in constraint metric values.

[0071] Figure 3 clearly illustrates this problem. This is the result of performing 3 searches from each point while adjusting from 0.001 to 0.010, and the average value of the 3 searches is indicated by a thick circle. Even if a slight trend is observed, inconsistency prevails in both the direction and variance of the trajectory. Considering a scenario where a designer intends to design a latency-based neural network-accelerator architecture pair under certain constraints (e.g., 33.3 ms) using existing integrated search methods, the initial You can attempt a search and adjust the value during multiple search processes. However, These discrepancies and latencies can make it difficult to obtain a suitable solution, not to mention the enormous time cost of running multiple searches. Designing an effective strategy is necessary to solve the rigidly constrained integrated search problem.

[0073] A differentiable integrated search with rigid constraints can be defined by the following mathematical equation 2.

[0074] [Mathematical Formula 2]

[0075]

[0076] Here, represents the current value of a constrained metric such as latency or energy, and represents the target value (e.g., 33.3 ms for delay time). and represents the neural network architecture parameters and hardware accelerator configuration, respectively. is the weight of the NAS supernet, and is the selected neural network architecture. Is and Represents the hardware metrics evaluated for. The purpose of integrated exploration is the neural architecture loss (Loss) defined by the user. NAS ) and hardware costs (Cost HW It can be expressed using two evaluation metrics of ).

[0078] Referring to FIG. 4, an integrated search method executed in an integrated search device (100) according to the present invention is illustrated. That is, the integrated search method according to the present invention may correspond to a differentiable integrated search method that considers hard-constrained constraints called HDX (Hard-constrained Differentiable co-eXploration). The left part in FIG. 4 (a) may correspond to a network search module (i.e., corresponding to the NAS module (110) of FIG. 1), and may generate a network that is finally searched by finding a path within a supernet using backpropagation.

[0079] The right side of FIG. 4(a) may correspond to an evaluator (i.e., corresponding to the HDX evaluation module (130) of FIG. 1) that searches for an optimal hardware accelerator design and evaluates cost metrics using architecture parameters obtained from the network search module. The evaluator network eval() is the core of a differentiable integrated search that considers the relationships between hardware accelerators to ensure that the gradient flows toward the supernet. The evaluator network may consist of a hardware generative network gen() and an estimation network est(). The hardware generative network can take neural architecture parameters as input and output an optimal hardware implementation (i.e., of Equation 2). (corresponding to ). Hardware-generated networks can be jointly trained during integrated exploration to adapt to constraints without relying on a specific cost function.

[0080] The estimation network can take the outputs of the network search module and the generative network to output hardware-related metrics. The estimation network can be pre-trained based on the network and accelerator search space. To this end, traditional (non-differentiable) cost estimation frameworks such as MAESRTO, Timeloop, and Accelergy can be used as ground truths. After pre-training, the estimation network is frozen during exploration and can be used only to infer hardware costs given the network architecture. Through this, the above Equation 2 can be transformed as the following Equation 3.

[0081] [Mathematical Formula 3]

[0082]

[0083] Here, is the weight for the hardware-generated network.

[0085] The present invention allows for the application of gradient control as an effective solution to stiffness constraints in a differentiable integrated search method. Gradient control is frequently used to achieve various objectives, such as continuous functions or differential equations. Here, gradient control is applied to the integrated search to satisfy stiffness constraints. Referring to figures (b) and (c) of FIG. 4, the gradient control method artificially generates a force capable of pushing the gradient in a direction that agrees with the constraints. The new gradient The conditions under which the method is applied to calculate can be defined as shown in the following mathematical formula 4.

[0086] [Mathematical Formula 4]

[0087]

[0088] Here, is the original gradient of the global loss function defined as in the following Equation 5, as in Equation 3 above.

[0089] [Mathematical Formula 5]

[0090]

[0091] is the slope of the constraint loss, It can be defined as. Is Since it is a function of It can be backpropagated to find the gradient for. In the ideal case, nothing is done because the constraint is already satisfied. If the constraint is still not satisfied, the inner product of the two slopes can be calculated to determine the agreement in that direction. (i.e., if the angle between the two gradients is 90 degrees or less), it means that the gradient descent update will contribute to satisfying the constraint. Therefore, it is interpreted as a match in direction and is the same It can be used as is (corresponding to Figure (b) in Fig. 4). However, as illustrated in Figure (c) of Fig. 4, if there is a discrepancy (i.e., ), slope By forcing a change in direction by that amount It can be calculated from to guarantee a reduction in target cost after gradient descent. This It can be reconstructed as. Here, is a small value to ensure gradual movement in the direction that satisfies the constraint. and To update, the network architecture parameter is The optimal for It can solve.

[0092] is parallel to the rigidity constraint gradient, and its magnitude can be set to a minimum to limit its influence on the loss function gradient. This can be simply calculated using the pseudo-inverse and can be expressed as follows.

[0093] [Mathematical Formula 6]

[0094]

[0095] To control the size of the pull, small multiplication coefficient Uses Using The policy for updating is as follows.

[0096] Some initial values ​​for This exists. If the target metric does not satisfy the constraints to Strengthen the pool by multiplying by ( ), if the constraint is satisfied is reset to the initial value ( ). Using gradient descent, the weights of the hardware-generated network You can also train. Therefore, in the same way Calculate for, but for hardware-generated network updates instead This can be used. By applying this, it can also be applied even when there are multiple stiffness constraints, and in this case, it can be expressed by the following mathematical formula 7.

[0097] [Mathematical Formula 7]

[0098]

[0100] Basically, the hardware cost function is the balanced weighted sum of three hardware cost metrics. HW It can be used as and can be expressed as the following mathematical formula 8.

[0101] [Mathematical Formula 8]

[0102]

[0104] C E , C L and C ABy controlling , conditions can be established for the method of measuring the balance between each cost metric. To match the scale of these hyperparameters, mJ, ms, and μm for each cost 2 Units can be used.

[0105] Additionally, the estimation and generative networks can be modeled as 5-layer perceptrons with residual connections. To train the estimation network, a dataset is first constructed by randomly sampling 10.8 million network accelerator pairs (2.95e-9% of the total search space) from the search space and evaluated on hardware metrics using Timeloop and Accelergy. Using this dataset, the estimation network is trained for 200 epochs with a batch size of 256. Weight updaters are performed using an Adam optimizer with a learning rate of 1e-4. The accuracy of the estimation network is over 99% across all metrics, making it sufficiently powerful as an engine for integrated search. The generative network is randomly initialized and co-trained with a NAS supernet. As gradients regulated under rigidity constraints are backpropagated, the generative network can learn how to generate accelerators that satisfy the constraints of a given neural network architecture.

[0106] The search space may use ProxylessNAS as the backbone. It can be composed of various configurations of MBConv tasks with kernel sizes of {3, 5, 7} and expansion ratios of {3, 6}. The total number of layers can be 18 and 21 for the CIFAR-10 and ImageNet datasets, respectively. The invention is orthogonal to the NAS implementation and can be selected from differentiable NAS algorithms such as DARTS or OFA. Eyeriss can be used as the backbone architecture of the accelerator. This can be composed of a 2D PE array having a Multiply-Accumulate (MAC) device in which each PE is connected to a register file. Thus, the hardware accelerator design space can include PE array sizes ranging from 12 x 8 to 20 x 24 and register file sizes per PE ranging from 16B to 256B. In addition, the search space may include data flows of WS (Weight-Stationary), OS (Output-Stationary), and RS (Row-Stationary).

[0108] The experimental results regarding the present invention are described below.

[0109] Experiments

[0110] Experimental Environment

[0111] Experiments were conducted using the CIFAR-10 and ImageNet datasets. For all reported hardware metrics (latency, energy, and chip area), direct evaluations of the hardware designed in the time loop and accelerator were used instead of values ​​output by the cost estimation network to prevent potential errors in the trained model. Network accuracy evaluation was performed by training the final network architecture from scratch for 300 epochs using a batch size of 64. An SGD optimizer with Nesterov momentum was used, employing cosine learning rate scheduling with an initial value of 0.008, and the weight decay term and momentum were 1e-3 and 0.9, respectively. Augmentation for the training data was adopted from AutoAugment on both datasets.

[0113] Comparison with Existing Methods

[0114] Table 1 below lists existing differentiable integrated search methods compared to HDX.

[0115] [Table 1]

[0116]

[0117] Here, the methods subject to comparison are as follows.

[0118] NAS→HW: Only a differentiable general NAS is present, followed by a thorough hardware search using a time loop.

[0119] Auto-NBA: A differentiable integrated search method that directly searches for hardware parameters using gradient descent, and represents the relationship between the hardware and the DNN as a lookup table.

[0120] DANCE: A differentiable integrated search method without rigid constraints in which a hardware generation network and a cost estimation network are modeled as neural networks.

[0121] DANCE + Soft Constraint: A term for soft constraints used to denote a rational approach to considering existing solution constraints. Added.

[0122] HDX: This is a proposed method.

[0123] As shown in Table 1 above, HDX is the only method that considers rigid constraints in conjunction with differentiable integrated search. For quantitative comparison, an algorithm was devised to find a constrained solution using integrated search without rigid constraints. Since a constrained metric (e.g., latency) that is too low often implies an inefficient solution (in terms of other metrics (e.g., accuracy and energy)), a standard was established for solutions that achieve 50% to 100% of the target constraint. A rough sketch of an algorithm similar to binary search is as follows.

[0124] (1) Control parameters that indirectly affect the metric in constraint conditions (e.g., delay time) and Select ).

[0125] (2) Perform a search using the basic control parameter values.

[0126] (3) Perform a search by doubling the control parameter at each step until the metric is under the constraint.

[0127] (4) If the metric is less than 50% of the target value, the control parameter is reduced using a binary search method.

[0128] This problem cannot be accurately solved using binary search due to the significant variability per search and non-linear relationships. In the actual implementation, additional features were added to handle corner cases and avoid falling into invalid parameter regions (which can occur due to variability per search). The algorithm was run 100 times and the average was reported, and the average error was also reported as a proxy for solution quality.

[0129] As shown in Table 1 above, all baselines require multiple searches to find a solution that satisfies the constraints. For DANCE, it takes 6.6 searches with an average of 12.2 GPU times, and even under soft constraints, it takes 4.9 searches and 9.1 GPU times. On the other hand, HDX always finds a solution that satisfies the constraints in a single search. Furthermore, the number of baseline searches varies significantly depending on the selection of default values ​​for control parameters. Guessing the correct initial parameters is difficult and can result in higher iteration costs. This supports the necessity of HDX.

[0131] Co-exploration Results

[0132] Figure 5 shows a two-dimensional plot of the experimental results of integrated search using various techniques on the CIFAR-10 dataset. The horizontal axis represents the accuracy of the artificial neural network, while the vertical axis represents latency in the left and middle cases, and all hardware parameters of the loss function in the right case. Two cases were tested with latency of 30 frames (33.3 ms) and 60 frames (16.6 ms). For all integrated search methods, By changing to 0.001 to 0.005, five solutions were obtained to fairly compare the various design points obtained from each approach.

[0133] NAS→HW refers to a methodology (individual design) in which an artificial neural network is designed first and the accelerator structure is derived accordingly, rather than through integrated design; DANCE and Auto-NBA utilize loss function techniques in backpropagation-based methodologies that do not consider existing stiffness constraints. Finally, HDX represents the technique proposed in this invention. In the case of NAS→HW Since it is impossible to apply it directly, 10 solutions of various ranges are obtained for reference. For DANCE and Auto-NBA, the black marker is not constrained, and the color marker is obtained with the soft constraint of that color.

[0134] In all experiments, of mathematical formula 10 It uses this to make the difference scales of each metric nearly identical for a fair comparison. In the case of the NAS-only method, It was used during the hardware design phase.

[0135] The left and middle of Figure 5 show the relationship between error and latency, and the colored horizontal bar represents two latency targets. Regardless of the value of , it is easy to see that all solutions found by HDX meet the given rigid constraints. Furthermore, all solutions have a latency just below the constraint, demonstrating that the solutions are not over-optimized for the limited metric (latency). While DANCE and Auto-NBA can leverage a trade-off between hardware metrics and accuracy, they cannot control whether they satisfy the constraints. Even when using soft constraints, most failed to obtain solutions within the constraints.

[0136] In the case of individual designs, hardware-related goals were not considered due to an overemphasis on accuracy; consequently, both techniques considering loss function-based methodologies failed to achieve the rigidity constraint goals in many cases for both 30 and 60 frames. In particular, while there were many instances where the goal was achieved by chance for 30 frames, the goal was not met in cases with difficult constraints, such as 60 frames, despite modifications to the loss function. This is because the loss function only provides a slight guide for optimization and cannot enforce actual rigidity constraints.

[0137] The plot on the right represents the quality of loss function optimization by dividing the loss function into two components: accuracy and hardware metrics, with results being better as they move toward the bottom left. As can be easily seen in the figure, the proposed method does not lag behind existing methods in terms of loss function quality; in fact, there are instances where it is superior. This demonstrates that this method achieves rigidity constraints by not sacrificing quality.

[0139] Solution Quality Found by HDX

[0140] The HDX method proposed in this invention demonstrates that it can 1) handle constraints on all three metrics (latency, energy, and chip area), 2) handle multiple constraints, and 3) obtain a solution of overall excellent quality. The case on the right side of Fig. 5 is... By displaying the error together, the quality of the solution can be evaluated in terms of Pareto optimization. Since the cases on the left and in the middle of Fig. 5 overlook other metrics, It must be fair to compare them together.

[0141] Looking at the plot, it can be seen that the solution quality of HDX is superior to the NAS→HW method and shows no degradation compared to the existing integrated search method. In fact, a strictly limited solution (16.6 ms) can find a better solution than the existing one in terms of Pareto optimization. To further investigate the quality of the solutions found by HDX, another series of experiments was conducted. The solution obtained through the existing integrated design method, the DANCE method, is called 'Anchor' and is listed in Table 2 below.

[0142] [Table 2]

[0143]

[0144] Table 2 above indicates that the proposed method can perform integrated design while satisfying stiffness constraints across various settings. The CIFAR-10 dataset was utilized, and stiffness constraints were designed based on the 'Anchor' solution obtained through the existing integrated design method. Among these, one or all three hardware metrics to be fixed as stiffness constraints were selected, and integrated search was performed using HDX. For example, in the case of A, the latency is 69 ms and the energy consumption is 37 mJ. By setting the stiffness constraint for latency to the same 69 ms and performing integrated design using the proposed method, it is possible to determine 1) whether the stiffness constraints can be satisfied, and 2) whether a solution of the same or similar quality to the solution designed without stiffness constraints can be derived. In fact, when designed with a latency of 69 ms, a solution of 43.99 ms was obtained, and it was demonstrated that the quality did not lag behind, with the total loss actually being lower. In addition, it was observed that when rigidity constraints were applied to latency, energy, and chip area, an excellent solution was obtained while successfully satisfying all constraints.

[0146] Results from ImageNet Dataset

[0147] Table 3 below shows the results of two integrated designs performed on ImageNet under a 125 ms constraint.

[0148] [Table 3]

[0149]

[0150] As shown in Table 3, the HDX method proposed in this invention was the only one that satisfied the constraints in both cases, and in terms of solution quality, it can be confirmed from the loss value that it is the best quality among the solutions that satisfied the constraints. In addition, the Top-1 error and global loss show that the quality of the solution found by HDX is not damaged at all compared to DANCE or its variations.

[0152] Sensitivity Study on Pulling Magnitude

[0153] Referring to FIG. 6, the only adjustable factor that the HDX method proposed in the present invention has It shows the response to, and the performance of the proposed technique It shows that the result was not obtained by arbitrarily adjusting it.

[0154] FIGS. 6(a) to (c) show various types of 1e-2, 7e-3, and 4e-3, respectively. This shows how global loss and latency change in a search with limited latency (33.3 ms). It shows that although there is a difference in that the constraint is satisfied slightly later when α is small and slightly earlier when α is large, the constraint is ultimately successfully satisfied in all cases, and the latency obtained is not significantly lower than the constraint.

[0155] in other words, Regardless of the value of , the curve for the constrained value shows a similar trend. Initially, the global loss is primarily optimized, while the latency is kept constant. At this stage, the pooling size (See Equation 7) is still increasing and is not strong enough to make a meaningful change. At a certain point It becomes strong enough to pull the solution in the direction of reducing the delay time. Once the delay time satisfies the constraint, the global loss begins to decrease while maintaining the delay time. There is no significant difference between the global loss and the delay time in the final solution, which indicates that the HDX method proposed in this invention is an adjustable factor. It shows that it is insensitive to.

[0157] Analysis on the Searched Solutions

[0158] Referring to FIG. 7, (a) and (b) show example neural network and accelerator structures obtained under 60-frame and 30-frame latency constraints, respectively. Here, Is kernel and It refers to an MBConv block with an expansion ratio.

[0159] In the case of the 60 frames in Fig. 7 (a), since latency constraints are strict, the neural network mostly uses small kernels, and instead, the number of layers can be increased to secure a trade-off between latency and accuracy. To satisfy strict constraints while maintaining accuracy, the network mainly uses small 3x3 kernels, and using smaller kernels can reduce the number of multiplications quadratically. Therefore, reducing the kernel size and increasing the number of layers can be a good choice for reducing inference latency. Accelerators can pursue latency reduction by utilizing a weight stationary (WS) structure, known for its speed, along with a large number of PEs. Looking at the accelerator design, a relatively large PE array (16×16) is used to achieve low latency, and WS dataflow, known for low latency, is required. Additionally, the network contains some kernels with a high channel expansion ratio. WS has advantages over the discovered network because it utilizes parallelism for fast execution.

[0160] On the other hand, for the neural network obtained under the 30-frame constraint of Fig. 7 (b), since there is ample time, many large kernels are selected, which is advantageous for increasing accuracy, and the accelerator can use a row stationary (RS) structure known for its excellent energy efficiency. For the design at 30fps, the design can settle on a solution that optimizes energy consumption while satisfying the constraints. This design can use larger kernels in the network and RS dataflow in the accelerator. RS is known for its excellent energy efficiency and can utilize parallelism in the spatial dimension of the kernel and activation. Therefore, using larger kernels is advantageous for the RS dataflow. To reduce energy consumption, the design includes fewer PEs (12×8), a larger RF to save off-chip access energy, and fewer layers in the network.

[0162] The present invention proposes HDX, a rigidly constrained differentiable integrated search method, which is an integrated search method according to the present invention. By conditionally applying gradient manipulation to shift the solution in a direction that satisfies the constraints, strict constraints can be reliably satisfied with high-quality solutions. The integrated search method according to the present invention can significantly reduce the computational load through a reinforcement learning-based neural network / accelerator integrated design that considers rigid constraints.

[0164] Although the present invention has been described above with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims. Explanation of the symbols

[0166] 100: Integrated Search Device 110: NAS Module 130: HDX Evaluation Module 150: Tilt adjustment module

Claims

Claim 1 An artificial neural network and computational accelerator architecture integrated search device considering rigid constraints, comprising: a NAS (Neural Architecture Search) module for determining a neural network architecture; an HDX (High-constrained Differentiable Co-Exploration) evaluation module for determining an accelerator architecture according to the determined neural network architecture and predicting the loss of the determined neural network architecture and hardware metrics regarding the determined accelerator architecture; and a gradient control module for adjusting the gradients of a loss function (Loss) and a constraint function (Const) based on gradient descent so that the predicted hardware metrics proceed in a direction that satisfies a specified constraint, wherein the gradient control module performs gradient control of the loss function to select the neural network architecture in a direction that satisfies the specified constraint when the predicted hardware metrics do not satisfy the specified constraint, or performs gradient control of the hardware cost function to select the accelerator architecture in a direction that satisfies the constraint. Claim 2 In claim 1, the NAS module uses backpropagation to find a path within the SuperNet and generates the neural network architecture that is finally searched, and cross-entropy loss (Loss CE An integrated search device for an artificial neural network and computational accelerator structure considering rigidity constraints, characterized by calculating ). Claim 3 An artificial neural network and computational accelerator structure integrated search device considering rigidity constraints according to claim 1, wherein the HDX evaluation module is constructed through prior learning and searches for optimal hardware according to the determined neural network architecture as the accelerator architecture, and determines at least one of a PE (Processing Element) array configuration (PEx, PEy), a register file (RF) configuration, and a dataflow (DF) configuration regarding the accelerator architecture; and a cost estimation network that predicts the hardware metric based on parameters regarding the neural network architecture and configurations regarding the accelerator architecture. Claim 4 An artificial neural network and computational accelerator structure integrated search device considering rigidity constraints, characterized in that, in paragraph 3, the cost estimation network is composed of a multilayer perceptron with residual connections, and the hardware metric is predicted by determining the latency, area, and energy consumption through the multilayer perceptron. Claim 5 In claim 4, the artificial neural network and computational accelerator structure integrated search device considering rigidity constraints is characterized by the cost estimation network calculating a balanced weighted sum regarding the delay time, area, and energy consumption as a cost function to predict the hardware metric. Claim 6 delete Claim 7 An integrated search device for an artificial neural network and computational accelerator structure considering stiffness constraints, wherein, in claim 1, the gradient control module calculates a control direction through a pseudo-inverse such that the angle between the gradient of the loss function and the gradient of the constraint function is 90 degrees or less. Claim 8 delete Claim 9 In an integrated search method performed in an integrated search device for an artificial neural network and computational accelerator architecture considering rigidity constraints, the method comprises: a NAS module execution step for determining a neural network architecture through a NAS (Neural Architecture Search) module; and an HDX evaluation module execution step for determining an accelerator architecture according to the determined neural network architecture and predicting the loss of the determined neural network architecture and hardware metrics regarding the determined accelerator architecture through an HDX (High-constrained Differentiable Co-Exploration) evaluation module. A method for integrated search of an artificial neural network and a computational accelerator structure considering rigidity constraints, comprising a gradient control module execution step through which the gradients of a loss function (Loss) and a constraint function (Const) are adjusted so that the predicted hardware metric proceeds in a direction that satisfies a specified constraint based on gradient descent, wherein the gradient control module execution step is characterized by performing gradient control of the loss function to select the neural network architecture in a direction that satisfies the specified constraint when the predicted hardware metric does not satisfy the specified constraint, or performing gradient control of the hardware cost function to select the accelerator architecture in a direction that satisfies the constraint. Claim 10 In claim 9, the NAS module execution step uses backpropagation to find a path within the SuperNet to generate the neural network architecture that is finally searched, and cross-entropy loss (Loss CE An integrated search method for artificial neural networks and computational accelerator structures considering rigidity constraints, characterized by calculating ). Claim 11 In claim 9, the HDX evaluation module execution step comprises: a hardware generation network execution step that searches for an optimal hardware according to the determined neural network architecture built through prior learning as the accelerator architecture and determines at least one of a PE (Processing Element) array configuration (PEx, PEy), a register file (RF) configuration, and a dataflow (DF) configuration regarding the accelerator architecture; and a cost estimation network execution step that predicts the hardware metric based on parameters regarding the neural network architecture and configurations regarding the accelerator architecture, characterized in that it includes a method for integrated search of an artificial neural network and a computational accelerator structure considering rigidity constraints. Claim 12 A method for integrated search of an artificial neural network and computational accelerator structure considering rigidity constraints, characterized in that, in claim 11, the cost estimation network execution step is composed of a multilayer perceptron with residual connections, and the hardware metric is predicted by determining the latency, area, and energy consumption through the multilayer perceptron. Claim 13 delete Claim 14 delete

Images