Creation and global tuning of application-specific machine learning accelerators

Abstract

A method, system, and apparatus, including a computer-readable medium, for globally tuning and generating an ML hardware accelerator are described. A design system selects an architecture that represents a baseline processor configuration. An ML cost model of the system generates performance data for the architecture by modeling at least how the architecture performs computations of a neural network that includes multiple layers. Based on the performance data, the architecture is dynamically tuned to meet performance goals when the architecture implements the neural network and executes machine learning computations for a target application. In response to dynamically tuning the architecture, the system generates a configuration of the ML accelerator that specifies a customized hardware configuration for implementing each of the multiple layers of the neural network.

Description

[Technical Field]

[0001] background This specification relates generally to integrated circuits used to perform machine learning calculations. [Background technology]

[0002] A neural network is a machine learning model that uses one or more layers of nodes to generate an output, e.g., a classification, for a received input. Some neural networks include one or more hidden layers in addition to an output layer. Some neural networks can be convolutional neural networks (CNNs) configured for image processing or recurrent neural networks (RNNs) configured for speech and language processing. Different types of neural network architectures can be used to perform various tasks related to classification or pattern recognition, prediction involving data modeling, and information clustering.

[0003] A neural network layer can have a corresponding set of parameters or weights. The weights are used to process inputs (e.g., batches of inputs) through the neural network layer to generate the layer's corresponding output for computing the neural network inference. The batches of inputs and sets of kernels can be represented as tensors, i.e., multidimensional arrays, of inputs and weights. A hardware accelerator is an integrated circuit dedicated to implementing neural networks. The circuit includes a memory with locations corresponding to elements of the tensors that can be traversed or accessed using the circuit's control logic.

[0004] Designing dedicated hardware accelerators is labor-intensive and time-consuming. For example, the design process often requires several months of effort and can include numerous design iterations. Furthermore, to meet application-specific performance and power targets, the design process requires a strategy for mapping the target application to the underlying hardware. While the computational graph of a neural network is static, the mapping effort can involve multiple design parameters that affect the actual performance of the circuit. Furthermore, manual exploration of the design space is often prohibitive due to the sheer size of different settings and interrelationships between different parameters. Summary of the Invention

[0005] overview This specification describes techniques for globally tuning a data processing architecture and automatically generating an application-specific machine learning (ML) accelerator based on the tuned architecture. The architecture can be a candidate architecture selected based on a set of application-level objectives. Exemplary application-level goals can include processor utilization, power consumption, data throughput, and latency. In some cases, the goals represent desired performance attributes of a user of an exemplary ML accelerator. Some (or all) of the goals may be received as user input to an exemplary hardware accelerator design system. The design system may determine one or more of the goals independently of the user input.

[0006] The system uses application-level goals (e.g., one or more inputs) to globally tune and dynamically optimize candidate architectures. For example, an architecture may be tuned and optimized to run a particular type of neural network to achieve efficiency in areas such as power consumption and processor utilization. The accelerator design system uses an architecture-specific cost model to tune various aspects of the architecture. The output of the cost model is used to define the final configuration of the accelerator. After optimization and tuning, the system automatically generates a hardware configuration that includes various architectural features, including scheduling / mapping options, to produce an application-specific (ML) accelerator optimized for implementing a particular neural network in hardware.

[0007] One aspect of the subject matter described herein may be embodied in a computer-implemented method for generating an application-specific machine learning (ML) accelerator. The method includes selecting an architecture representing a baseline processor configuration and generating performance data for the architecture using an ML cost model by modeling at least how the architecture performs computations of a first neural network including multiple layers. The method includes dynamically tuning the architecture based on the performance data to meet a performance goal when the architecture implements the first neural network and performs machine learning computations for a target application. The method also includes generating a configuration of the ML accelerator in response to dynamically tuning the architecture. The configuration specifies a customized hardware configuration for implementing each of the multiple layers of the first neural network.

[0008] These and other implementations may each optionally include one or more of the following features. For example, in some implementations, the method further includes generating an application-specific hardware ML accelerator based on the customized hardware configuration. In addition, the application-specific hardware ML accelerator can be optimized to implement each of the different layers of a neural network when the neural network is used to perform computations for a target application.

[0009] The performance goals include a plurality of distinct goals, and generating the application-specific ML accelerator includes generating an application-specific hardware ML accelerator configured to meet each distinct goal of the plurality of distinct goals when the application-specific hardware ML accelerator performs computations for the target application. In some implementations, generating performance data includes modeling use of the architecture to execute each of the plurality of layers of the first neural network with an ML cost model, and generating performance parameters for each of the plurality of layers with the ML cost model in response to modeling use of the architecture to execute each layer.

[0010] The performance parameters can correspond to each separate goal of a plurality of separate goals, the plurality of separate goals including at least one of a threshold processing latency, a threshold power consumption, a threshold data throughput, and a threshold processor utilization. In some implementations, dynamically tuning the architecture includes determining a mapping of computations for input tensors that causes the application-specific hardware ML accelerator to utilize a threshold percentage of hardware computing units of the hardware ML accelerator, and dynamically tuning the architecture based on the determined mapping.

[0011] Dynamically tuning the architecture may include dynamically tuning the architecture based on operations performed by each of the plurality of ML cost models of the global tuner, and dynamically tuning the architecture based on operations performed by at least one of a random tuner or a simulated annealing tuner of the global tuner. In some implementations, the architecture represents one or more hardware blocks of an integrated circuit, and dynamically tuning the architecture includes dynamically tuning the architecture to meet a respective performance target for each of the one or more hardware blocks when the architecture implements the first neural network to perform computations for the target application.

[0012] Configuring the hardware ML accelerator specifies a customized software configuration for the first neural network, and generating the application-specific hardware ML accelerator includes generating the application-specific hardware ML accelerator based on the customized hardware configuration and the customized software configuration. In some implementations, the ML cost model includes one or more individual analytical models. The architecture-aware cost model is configured to estimate the performance of the architecture based on a deterministic data flow of data processed using the architecture.

[0013] Other implementations of this and other aspects include corresponding systems, devices, and computer programs encoded on computer storage devices and configured to perform the actions of the methods. One or more computer systems may be so configured by software, firmware, hardware, or combinations thereof installed on the systems that, when operated, cause the systems to perform the actions. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the systems to perform the actions.

[0014] The subject matter described herein can be implemented in particular embodiments to realize one or more of the following advantages.

[0015] The disclosed technology provides a framework that can be used to expedite the architecture exploration process for specifying optimized hardware and software configurations, including efficient scheduling / mapping of operations for implementing neural networks in hardware circuits. Based on this process, a hardware design system can automatically generate an output configuration that specifies an optimized hardware mapping for a system for a given set of PPA (performance, power, area) constraints. The PPA constraints can be hardware accelerator performance thresholds with respect to at least processor utilization, power consumption, latency, block size, and / or data throughput.

[0016] The design system can identify an example network model having a fixed number of layers and determine optimal attributes of the identified hardware architecture (e.g., systolic array, computational tile, etc.), including its microarchitectural attributes such as block connectivity, hardware layout, or memory. In addition to these optimized hardware attributes, the design system determines efficient scheduling and data allocation for per-layer processing, such that an application-specific ML accelerator can be generated that meets (or exceeds) user- or system-specified requirements for layer-specific processing while consuming minimal amounts of power and circuit area.

[0017] The details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a block diagram of an exemplary computing system for generating and globally tuning machine learning accelerators. [Figure 2] FIG. 1 is a block diagram illustrating an example system for globally tuning an application-specific machine learning accelerator. [Figure 3] FIG. 1 illustrates an exemplary framework for tuning multi-layer neural networks. [Figure 4] 1 is a flow diagram of an example process for tuning and optimizing a graph execution schedule of a multi-layer neural network. [Figure 5] 1 is a flow diagram of an exemplary process used to generate and globally tune a machine learning accelerator. [Figure 6]FIG. 2 is a block diagram of an exemplary application-specific hardware accelerator generated using the system of FIG. 1. [Figure 7] FIG. 10 is a diagram illustrating an example of an input tensor, a weight tensor, and an output tensor. DETAILED DESCRIPTION OF THE INVENTION

[0019] Like reference numbers and designations in the various drawings refer to like elements. Detailed Description 1 is a block diagram of an exemplary hardware accelerator design system 100 (“system 100”). Generally, system 100 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a special-purpose processor, etc.), memory, and / or data storage that collectively form processing resources used to perform functions for globally tuning and generating customized hardware machine learning accelerators.

[0020] As described below, using one or more input targets 102, system 100 is configured to develop and output design configurations for generating exemplary hardware accelerators. The hardware accelerators can be implemented as dedicated or application-specific hardware circuits that are optimized to perform specific types of machine learning tasks. For example, the application-specific circuits can be machine learning (ML) hardware accelerators configured to implement or operate multi-layer neural networks.

[0021] More specifically, application-specific circuits may be uniquely tuned and / or optimized according to different application goals, such as one or more inputs specified by a user. For example, when implementing a particular type of neural network (e.g., a multi-layer CNN), a candidate data processing architecture for the application-specific ML circuit may be optimized to meet (or exceed) threshold performance goals regarding processor utilization, power consumption, data throughput, and / or latency.

[0022] As used herein, a data processing "architecture" can refer to a hardware circuit architecture, a software / neural architecture, or both. In this manner, tuning and optimizing an architecture can include tuning attributes of the hardware architecture and tuning attributes of the neural architecture, such that the resulting architecture is optimized (e.g., fully optimized) to perform a given machine learning task according to each different application goal, which may be received or determined by system 100.

[0023] System 100 includes control logic for building and managing design space 104. Design space 104 may be built based on a combination of hardware devices and software routines executing in system 100. For example, the control logic may be implemented as a system controller or host device that executes programmed instructions to manage various design space operations. Operation of design space 104 may include processing multiple design items or parameters required to tune candidate architectures.

[0024] Generally, system 100 uses control logic to manage the activities and operations of design space 104. In addition to optimizing an architecture for a given ML task, in some implementations, the control logic of system 100 may itself be based on an ML model. For example, an ML model may be trained to process design inputs and control parameters necessary to tune a candidate architecture based on a set of input objectives. In some implementations, the control logic executes or applies an example optimization algorithm that tunes the candidate architecture according to the set of input objectives and operations performed by an example cost model (described below).

[0025] The candidate architectures are selected from at least an architecture repository 106 of the system 100. The system 100 may identify or select the candidate architectures from the architecture repository 106 based on at least the input objects 102. The architecture repository 106 includes information describing multiple different hardware architectures that may be used to generate the application-specific hardware ML accelerator.

[0026] For example, a first hardware architecture accessed via the architecture repository 106 may define a systolic array architecture, while a second, different hardware architecture accessed via the architecture repository 106 may define a hardware architecture based on an arrangement of computational tiles. Similarly, a third architecture accessed via the architecture repository 106 may define a hardware architecture based on respective sets of tightly coupled data processing lanes forming separate vector processing units (VPUs), while a fourth architecture accessed via the architecture repository 106 may define a hardware architecture including at least two vector processor cores interacting with a large shared scratchpad memory and a matrix computation unit.

[0027] The candidate architecture selected for optimization and tuning can be, for example, a combination of a hardware circuit architecture and a neural architecture obtained from the architecture repository 106. The neural architecture may be obtained from a network graph module 108 that includes multiple different types of neural network graphs. For example, the system 100 can select a candidate architecture based on the input targets 102, an example hardware layout of an integrated circuit (IC), and an example neural network graph.

[0028] In some implementations, the system 100 selects candidate architectures based on one or more input objectives 102 that bias the system toward selecting a particular hardware architecture for a given neural network architecture. For example, the system 100 can select candidate architectures based on one or more hardware variables. The hardware variables can represent control parameters that constrain the architecture selection and cause the design space 104 to select a particular type of hardware architecture from the repository 106 for the given neural architecture, e.g., obtained from the graph module 108.

[0029] The system 100 includes an optimization and tuning module 112 that interacts with one or more cost models to globally tune the exemplary data processing architecture. For example, the system 100 includes an architecture-aware cost model 114 that can include one or more individual data models 114. In some cases, each of these individual data models is a respective cost model 114 that is configured to perform ML-based analysis to tune a candidate architecture based on a set of input objectives. The architecture-aware cost model 114 estimates the performance of the candidate architecture based on a deterministic data flow of data processed using the architecture.

[0030] In some implementations, the system 100 includes each cost model 114 based on one of two types of cost models: an analytical cost model or an ML-based cost model. Both models can receive the same inputs and produce the same outputs, as discussed in the optimization loop described below. Generally, the difference between these two types of cost models is how each model internally predicts its costs. There are various differences between analytical cost models and ML-based cost models.

[0031] For example, the analytical cost model can be a roofline-based model that considers various "ceilings" based on a set of hardware mapping parameters and neural network graphs. The analytical cost model does not require training data. Given an input, the analytical cost model uses "internal logic" to derive bottlenecks and output costs. Internally, one or more hardware blocks used to implement the analytical cost model can be configured to share a "cost module." The shared cost module is operable to generate costs given the hardware mapping parameters and neural network computations operated on the hardware blocks. In some cases, the analytical cost model produces particularly accurate cost outputs for applications with deterministic data flows.

[0032] The ML-based cost model requires labeled data to train a machine learning model that can predict at least latency and throughput. For example, the machine learning model can be trained to predict cost values ​​for different application-level objectives, including one or more of the PPA constraints. The ML-based cost model can be implemented using supervised learning and multilevel perceptrons. In some implementations, training data for the ML-based cost model is obtained through high-level synthesis and RTL simulation. To account for the distributed nature of the inputs, the inputs of the ML-based cost model can be converted into embeddings that are learned using standard techniques such as stochastic gradient descent. In some cases, the ML-based cost model is trained offline. The trained ML-based cost model is used during an optimization loop (described below) to dynamically optimize candidate architectures.

[0033] Each of the set of optimization and tuning modules 112 and cost models 114 can function as an extension of the design space 104. In some implementations, the set of optimization and tuning modules 112 and cost models 114 represents a global tuner that tunes attributes of both the hardware blocks and neural networks of the candidate architectures. Control logic of the design space 104 can be used to control or manage the operation of the global tuner. For example, the global tuner can interact with different aspects (e.g., variables and constraints) of the design space 104 to tune the candidate architectures based on control signals generated using the control logic. This is described in more detail below with reference to FIG. 2.

[0034] The optimization and tuning module 112 includes an exemplary tuner 116 and an exemplary scheduler / mapper 118. In some implementations, the tuner 116 and the scheduler / mapper 118 interact to perform exemplary tuning and optimization tasks of the module 112 (described below). As noted above, the data processing architecture can be a combination of a hardware circuit architecture, obtained, for example, from the architecture repository 106, and a neural architecture obtained from the neural network graph module 108. The hardware architecture can include multiple individual hardware blocks, each containing hardware features such as systolic array cells, vector processor lanes, or individual computational tiles.

[0035] Tuner 116 and scheduler / mapper 118 cooperate to i) configure candidate mappings of neural network layers to one or more hardware blocks, and ii) tune the respective microarchitecture of each hardware block for the candidate mappings based on one or more application goals 102. In this manner, optimization and tuning module 112 is configured to tune the respective microarchitecture of each hardware block such that a given hardware block is optimized for executing one or more layers of a neural network.

[0036] To achieve the desired performance goals, the optimization and tuning module 112 can interact with the architecture-aware cost model 114 to configure candidate mappings and iterate through the process of tuning the microarchitecture of each hardware block. This tuning iteration can include signal communication from the optimization and tuning module 112 to the design space 104, e.g., via optional data path 120. The communication can be, for example, to obtain new inputs, variables, constraints, or architectural features for augmenting the hardware blocks of the candidate architecture based on the performance estimates generated by the cost model 114. The system 100 can include a tuning loop 122 that represents the iterative process.

[0037] The system 100 generates an example output configuration 130 based on the processing operations of the design space 104, the optimization and tuning module 112, and the architecture-aware cost model 114. As described below, the system 100 can automatically generate an application-specific ML hardware accelerator (e.g., an integrated circuit) based on the output configuration 130.

[0038] 2 is a block diagram illustrating an example system 200 including a global tuner 202. In some cases, system 200 is included within system 100 as a subsystem of software / computational modules or hardware circuitry having programmed instructions executable by one or more processing units.

[0039] Operation of system 200 provides a global tuning framework for automatically generating application-specific ICs customized to perform learning tasks, such as training and inference, for a target application. In some implementations, the target application (or device) is a customized hardware accelerator with a fixed hardware configuration. In some other implementations, the target application is a type of workload related to image classification, object detection, autonomous vehicle navigation, graphics processing, or scientific computing.

[0040] The global tuner 202 is configured to globally tune / optimize candidate architectures according to different application goals 102 to generate application-specific ML accelerators. The global tuner 202 includes a design space builder 204 that builds the design space 104 based on one or more tuner variables and constraints 210. The design space builder 204 communicates with a design space explorer 212 and one or more cost models 214 for the global tuner 202. The cost models 214 correspond to individual models of the architecture-aware cost model 114 described above.

[0041] Based on the parsed neural network graph of module 108, design space builder 204 and design space explorer 212 can interact to implement a neural architecture search (NAS) system to select a neural network architecture ("neural architecture") that performs optimally for the target application. The NAS may employ various search techniques, such as techniques based on reinforcement learning, evolutionary search, differentiable search, etc. Design space builder 204 and design space explorer 212 may employ similar approaches to explore different hardware architectures that can be efficiently tuned and optimized for the target application.

[0042] The design space builder 204 and the design space explorer 212 implement the NAS and hardware architecture exploration techniques based on one or more tuner variables and constraints 210. The tuner variables and constraints 210 include various unroll factors, max mapper input / output data widths, or max reducer input / output data widths. As described above, a neural network layer can have a corresponding set of kernels (e.g., weights / parameters). The kernels can be convolution kernels with four dimensions: C—input channels, K—output channels, R—kernel height, and S—kernel width. An exemplary convolution operation can be represented as nested loops using four dimensional parameters (C, K, R, S). The set of kernels is represented as a multidimensional tensor, and nested loops can be used to traverse different dimensions of the tensor. In this context, an unroll factor corresponds to the unrolling of each of the nested loops. The global tuner 202 supports nested loop unrolling for all unroll factors and can tune candidate architectures with respect to these factors.

[0043] The mapper and reducer input / output data widths affect how large tensors are reduced into smaller pieces that are mapped to a given computational tile or cell. For example, input and output tensors can be quite large, and these tensors are not generated all at once. To reduce the area and power of a hardware accelerator that processes these tensors, the system 100 can utilize tensor tiling to divide the input and output tensors into multiple smaller pieces. For example, the system 100 can divide (or reduce) a large input tensor into smaller pieces based on mapping constraints. The mapping constraints may be coupled to goals such as power, area, latency, and / or throughput. The global tuner 202 can use these goals to determine the configuration and size of a set of computational tiles for a candidate architecture. The global tuner 202 can map computations for different pieces of an input tensor to a given tile in the set of computational tiles.

[0044] The max mapper input / output data width and max reducer input / out data width are constraints that directly affect the data throughput of the candidate architecture. The tuner variables and constraints 210 can include other items related to exploring candidate architectures for generating a hardware ML accelerator customized to run a given neural network for a target application. In some implementations, smaller tile sizes require longer data transmission times, and therefore, overall chip performance can also come into play here. All these different tuner variables and constraints 210 can result in different hardware designs, with implications for performance, power, and area. Thus, the global tuner 202 forms a design space from these variables / constraints and strikes a balance between performance, power, and area by selecting optimal parameters for customizing the hardware and neural architecture.

[0045] The global tuner 202 can dynamically tune the candidate architecture based on operations performed by at least each individual ML cost model 214. In some implementations, the global tuner 202 dynamically tunes the candidate architecture based on operations performed by at least one of i) a random search tuner, ii) a simulated annealing tuner, or iii) a progressive tuner. Each of the random search tuner, the simulated annealing tuner, and the progressive tuner corresponds to the tuner 116 described above. For the block partition model, the global tuner 202 implements a specific tuning trajectory associated with the simulated annealing tuner. Each of the random tuner, the simulated annealing tuner, and the progressive tuner may be implemented in software, hardware, or both. The functionality associated with each of these tuners may be integrated into the tuner 116 implemented in the global tuner 202.

[0046] The global tuner 202 uses a random search tuner to randomly sample the search space to obtain a trial configuration, such as a baseline processor configuration for the candidate architecture. The cost of running the target application on the trial configuration / architecture is obtained by querying the performance and power cost models of the ML cost model 214.

[0047] Simulated annealing, which can be implemented as a tuner in global tuner 202, is a probabilistic technique for approximating the global optimum for a given function. At each step, the tuner considers a neighboring hardware design point d' of a current hardware design point d and probabilistically determines whether to move the current design point toward design point d' or remain with design point d. A temperature variable is generated to control the acceptance probability. The simulated annealing tuner is configured to iterate these steps until the probability result indicates arrival at the optimal design point for the target application. For example, a probability score exceeding a threshold score can indicate that a particular design point performs optimally for the target application for a given set of constraints.

[0048] Adjacent hardware design points may be randomly generated. In some implementations, adjacent hardware design points have similar or very similar hardware parameter selections (e.g., unrolling, tiling, mapping, or scheduling) as the current hardware design point. The similarity of parameter selections may be characterized by the amount (or percentage) of overlap in hardware parameter selections between the two design points. In some other implementations, adjacent hardware design points may have one or more of the same hardware parameter selections as the current hardware design point.

[0049] The global tuner 202 uses a progressive tuner to implement a progressive search method of an exemplary design space, such as the design space of a NAS. This progressive search method can be used to reduce design space exploration time for tuning candidate architectures. In some implementations, the global tuner 202 performs the progressive search method to explore the design space as a step in designing and tuning ML hardware to meet (or exceed) a predetermined throughput requirement, such as a fixed data rate input to a machine learning block of an integrated circuit. The progressive search method can include at least i) initializing a baseline design as a minimum design for all neural network layers, and ii) querying the cost model 214 to identify bottleneck layers that have a data throughput lower than the data rate requirement. If the cost model 214 does not identify or indicate a bottleneck and / or the global tuner 202 determines that a layer of the neural network does not act as a bottleneck, execution of the search method terminates.

[0050] The progressive search method may further include the steps of: iii) exhaustively exploring the search space with respect to the bottleneck to determine a design configuration that minimizes the bottleneck by meeting (or exceeding) the throughput requirements while having the lowest cost in overall model performance; and iv) using the design configuration determined in step iii) as a new baseline design and then proceeding to step ii) again. In some implementations, the baseline design is a baseline processor configuration that includes the minimum hardware (and neural) architecture / design parameters for operating all layers of a given neural network. Exhaustively exploring the search space includes iteratively exploring different design configurations by using each design configuration to implement the multilayer neural network, evaluating the respective data throughput of each design configuration, and calculating a respective cost value for each different design configuration.

[0051] 2, the input goals 102 can be user-defined, system-defined, or both. For example, the input goals 102 can be received as a user configuration file or as a system-generated input file. The configuration or input file can specify various application-level goals 102, for example, derived from a set of PPA constraints. For example, the input file can include a set of application-level goals such as processor utilization, power consumption, data throughput, hardware block size, and / or latency. The input file also includes respective hardware accelerator performance thresholds for each application-level goal.

[0052] In some implementations, the input file includes a goal 102 indicating that the target application requires multiple vector operations. Based on this indication, the control logic can trigger a hardware variable 110 in the design space 104 that is set as a vector parameter (vector_ctrl). The design space 104 can use the vector_ctrl parameter to constrain the selection of candidate architectures, for example, to architectures that include multiple vector processing lanes that form tightly coupled VPUs.

[0053] 2, some (or all) of the cost models 214 perform ML-based analysis to tune the candidate architectures. According to a set of input goals 102, the global tuner 202 tunes the hardware and neural architecture of the candidate architectures based on one or more optimization algorithms. For example, the global tuner uses the cost models 214 to model the use of the candidate architectures to execute each layer of a multi-layer neural network with respect to predetermined hardware blocks of the neural network. In response to modeling the use of the architectures to execute each layer, the ML cost models 214 generate performance parameters that describe how the architectures perform for each layer.

[0054] In some implementations, an optimization algorithm is used to implement a cost model interaction loop, e.g., an optimization loop. For example, the optimizer or global tuner 202 (e.g., simulated annealing, progressive, random, etc.) can generate a set of hardware mapping parameters, such as the number of PEs and systolic array dimensions. The hardware mapping parameters, along with a neural network graph including layer dependencies and a quantization scheme (e.g., fixed), are sent to the cost model 214. The cost model 214 generates costs, such as latency, throughput, and power, based on the input. The cost output of the cost model can be fed back to the optimizer as a step in the optimization loop. The optimizer can process the cost output and determine the next hardware mapping strategy to explore. The global tuner 202 can iterate this optimization loop until a convergence condition is met or the search space is fully explored.

[0055] In some implementations, the first cost model 214 of the global tuner 202 is used to calculate performance estimates / parameters for the hardware attributes of the candidate architecture, while the second cost model 214 is used to calculate performance estimates / parameters for the neural network implemented in the candidate architecture. The first and second cost models 214 may be the same or different. The cost model 214 may use a single optimization algorithm to calculate performance estimates for tuning the architecture and optimizing the performance of the candidate architecture. In some other implementations, the cost model 214 uses different optimization algorithms to calculate performance estimates for optimizing various aspects of the architecture performance.

[0056] The global tuner 202 can use at least the design space builder 204, the design space explorer 212, and the cost model 214 to implement different design spaces and optimization strategies for the various hardware and neural network architectures being explored. For example, within each hardware block of a candidate architecture, the global tuner 202 explores different implementations that specifically target one layer, such as layer-specific tiling and tuning of systolic array dimensions. The global tuner 202 can explore layer transformations to increase parallelism. For example, the global tuner 202 can transform dense / 1×1 convolutions into n×n convolutions to increase throughput and / or utilization of compute units across one or more hardware blocks.

[0057] In some implementations, based on its optimization algorithm, cost model 214 calculates a utilization estimate from an indication that a dense convolution is assigned to one compute unit of a hardware block that includes multiple compute units. Global tuner 202 can compare the utilization estimate to a utilization threshold specified by application goal 102 (or constraint 210). Global tuner 202 can determine whether the calculated utilization estimate is lower than the threshold. In response to determining that the calculated utilization estimate is lower than the threshold, global tuner 202 can convert the dense 1×1 convolution to an n×n convolution to increase utilization of the compute units across the given hardware block. The utilization estimate is a performance parameter (or estimate) generated by cost model 214.

[0058] For multi-dimensional arrays of processing engines (e.g., cells, tiles, or processing lanes), the global tuner 202 can determine the optimal size / area and expected power density needed to achieve desired performance goals. The global tuner 202 can vary the number of processing engines (PEs) in each dimension of the array based on the determined size. The systems 100, 200 are configured such that one or more deep hardware customizations for one layer of the neural network do not disable or adversely affect the efficient operation of other layers of the neural network.

[0059] The global tuner 202 generates an output configuration 230 in response to tuning the candidate architecture. The output configuration 230 is used to automatically generate an application-specific ML accelerator. The output configuration 230 may represent an ML model (or algorithm) and a corresponding architecture configuration. The system 200 uses an exemplary code generation module 240 to convert data representing the output configuration 230 into high-level synthesis (HLS) code. For example, the code generation module 240 may generate a firmware implementation of the ML algorithm for the hardware accelerator using a high-level synthesis language (HLS).

[0060] Generally, the global tuner 202 is used to generate one or more application-specific ML accelerators that are fully customized for a target application. For example, customization can include items such as heterogeneous quantization and a microarchitecture tuned for one or more neural network layers. In some implementations, the global tuner 202 and system 200 are used to generate a customized architecture by identifying optimal hardware parameters, such as at least the microarchitecture, spatial mapping, and temporal mapping to optimize the overall architecture for a set of PPA constraints (e.g., goals 102).

[0061] Hardware features may be separated on or within a chip. Optimizing the spatial mapping of an architecture involves hardware blocks used to run different neural network operations spatially separated within a chip or integrated processor block. For example, a candidate architecture may be optimized for spatial mapping by using a specific arrangement of dedicated hardware blocks to perform dedicated operations in a neural network. This mapping allows the hardware blocks to be tuned for a particular algorithm or computational pattern.

[0062] Relative to other designs, architectures with optimized spatial mapping can provide improved performance and energy efficiency. Improvements may be realized from an arrangement of dedicated hardware blocks tuned to execute at least a particular algorithm or computational pattern. In some implementations, one or more dedicated hardware blocks are configured to process fixed-dimensional tensors, support a fixed quantization scheme, and be tuned for a particular neural network layer.

[0063] Optimizing the architecture's time mapping (307) includes hardware blocks that are time-shared among different operations in the neural network. For example, a candidate architecture may be optimized for time mapping by reusing the same hardware blocks to perform a wide variety of different operations in the neural network. While being more general in its use of a given hardware block, this approach can increase the programmability of the hardware. Furthermore, this approach can provide application developers with more flexibility regarding the neural networks that can be run in the hardware. In some examples, the optimized time mapping provides time-sharing of different layers in the same hardware block and support for multiple quantization schemes.

[0064] Customization can result in application-specific ML accelerators that consume significantly less power and area when compared to other processing devices that are not customized for the target application.

[0065] 3 illustrates an exemplary framework 300 for tuning a multi-layer neural network. Using this framework, system 100 can iteratively map computational nodes in a neural network graph to different features of the microarchitecture (or processing engine) in a given hardware block. For example, framework 300 may be implemented in global tuner 202 or optimization and tuning module 112 to determine and construct dependencies between various computational nodes in the neural network graph. The dependencies may be determined, for example, when ML cost model 214 models the execution of each layer of the neural network by a candidate architecture. ML cost model 214 generates performance parameters that provide an assessment of how the candidate architecture will perform when executing each layer of the neural network.

[0066] In the example of FIG. 3, neural network 302 includes five layers (L1-L5), with the first layer being L1, the second layer being L2, etc. These five layers may have an initial mapping to different hardware features (e.g., processing engines) of the candidate architecture. For example, each of the five layers may map to different cells of a systolic array, different systolic array blocks, different multiply-accumulate cells (MACs) of a computational tile, or different computational tiles. In some implementations, individual cells of a systolic array and individual MACs of a computational tile represent microarchitectural aspects of the candidate architecture.

[0067] The cost model 214 can calculate performance estimates for candidate architectures that implement the neural network 302. The performance estimates include parameters that indicate the duration for processing a given layer, the overall processing latency, and the PE utilization. The cost model 214 processes the durations to generate a neural architecture schedule 304 that is optimized for a set of timing constraints. Based on the performance estimates, the global tuner 202 can determine that the time required to compute layers L1+L2+L5 is approximately the same as the time required to compute layers L3+L4.

[0068] Based on this determination, the global tuner 202 can remap layers L1, L2, and L5 to reuse the same hardware feature B1, while layers L3 and L4 can be remapped to reuse the same hardware feature B2 (306). In some examples, B1 and B2 are respective processing engines 308, 310, such as computational tiles or systolic arrays, MACs, systolic array cells, or even mathematical operation units (ALUs) of vector processing lanes of a VPU. The global tuner 202 can perform the remapping as part of a tuning operation to reduce processing latency and optimize candidate architectures for executing neural network models according to the latency requirements specified in the targets 102.

[0069] For a given neural network, each layer may require different computation cycles. For example, after spatial remapping, some PEs may have more idle time than other PEs due to computational imbalance. This can be referred to as load imbalance. System 100 can compensate for or overcome load imbalance by utilizing tuning and optimization mechanisms that allow PE reuse across different layers, at least in the form of time. For example, tuner 116 and scheduler / mapper 118 can detect load imbalance and adjust attributes of candidate architectures to evenly balance computation cycles in each PE.

[0070] As described above, the five layers of neural network 302 may have an initial mapping in which each layer is mapped to a different hardware feature (e.g., processing engine) of the candidate architecture. Performance estimates for this initial mapping may include utilization parameters that indicate the overall underutilization of computational power in each processing engine to which the layers may be mapped. Based on these estimates and parameters, global tuner 202 may perform a remapping to increase processing utilization, for example, by remapping layers L1, L2, and L5 to reuse the same processing engine B1 and remapping layers L3 and L4 to reuse the same processing engine B2. This remapping may be performed to increase the overall utilization in each of B1 and B2 and optimize the candidate architecture for executing the neural network model according to the utilization (and latency) requirements specified in target 102.

[0071] The global tuner 202 can tune the candidate architecture to reallocate other operations to any remaining PEs (e.g., B3, B4, B5). In some cases, the global tuner 202 engages the design space explorer 212 to augment the hardware layout of the candidate architecture to reduce the number of PEs (e.g., from 5 to 2). In some other cases, the global tuner 202 engages the design space explorer 212 to reconfigure the PEs to increase the amount of parallelism across at least B1 and B2. The global tuner 202 may determine that the remaining PEs (e.g., B3, B4, B5) will be required to process smaller data sets after the remapping. Based on this determination, the global tuner 202 can, for example, adjust the compute-to-memory ratio of the microarchitecture of these PEs to optimize the size and utilization of the PEs for processing smaller data sets.

[0072] The framework 300 can accommodate an exemplary algorithm or computation sequence that takes as input a neural network graph along with application-level goals (e.g., inference time, throughput, power, etc.) and applicable hardware constraints 110, 210. The global tuner 202 can use the framework 300 as a basis for performing layer-by-layer spatial mapping explorations on various architectural knobs. Various architectural knobs may be supported by the framework 300, including: i) design styles such as systolic arrays or fully unrolled designs; ii) multiple mappers (e.g., systolic array clusters); iii) multiple systolic arrays per cluster; iv) input and output tiling; and v) hardware dimensional transformations for dense layers.

[0073] Each remapping or tuning to achieve optimization for a given constraint 210 may trigger a corresponding adjustment to the candidate architecture for another constraint. For example, remapping for B1 and B2 to optimize for a given timing or latency constraint may require an increase in throughput requirements for the PE. Thus, various architecture knobs often need to be refined to new (or other existing) requirements. In some implementations, system 100 iterates through its tuning of the candidate architecture to balance the interplay between at least latency, timing, and utilization to optimize the candidate architecture for each of these constraints. In some other implementations, system 100 balances the interplay between multiple constraints, variables, and goals.

[0074] Each of the architectural knobs can have a positive or negative impact on end-to-end application performance. Furthermore, each of the architectural knobs can also affect the effect of the architectural knobs on the mapping of other layers. Thus, based on at least the machine learning aspects of its control logic and the architecture-aware cost model 114, system 100 is configured to provide a holistic view of the candidate architectures under evaluation to accurately predict these positive or negative impacts.

[0075] A candidate architecture may include multiple processing engines, and one or more layers may be mapped to another processing engine based on a predetermined merging rule (e.g., conv2d + BN + activation merging; conv2d + maxpooling merging). The merging rule may be predefined, for example, as an instruction or a coded rule in the network graph module 108. In some implementations, two or more graph nodes (or layers) are merged if the computation of the next layer can be performed according to the computation of the previous layer (e.g., conv2d(+BN) + activation). As an example, the computation for a batch normalization (BN) layer may be merged with the computation for a 2D convolutional layer. Also, for each layer output provided as input to a subsequent layer, if the amount of input and computation for the subsequent layer is a threshold size and has a specific spatial and temporal locality, this subsequent layer may be merged with the previous layer that generated the layer output. An example of this may correspond to the layer output of a 2D convolutional layer provided as input to a pooling layer (e.g., conv2d + pooling).

[0076] In some implementations, to tune the candidate architecture, the global tuner 202 performs an initial mapping of each layer to a corresponding PE and generates a performance estimate for the initial mapping. Based on the performance estimate for the initial mapping, the global tuner 202 can iteratively map different combinations of layers to the PEs to tune the initial mapping. The global tuner 202 generates a performance estimate for each iteration, thereby identifying a mapping whose performance estimate matches the set of PPA constraints of the target 102.

[0077] When tuning a candidate architecture, the global tuner 202 uses one or more cost models 214 to iterate through different mappings and calculate performance parameters for each mapping. From the performance parameters, the system 100 identifies a mapping of computations that performs optimally for a given set of PPA constraints 210. In some implementations, the system 100 can iteratively map computational nodes for different vector operations to subsets of vector processing lanes in the VPU with a temporal mapping that specifies the sequence of nodes operating within the processing lanes.

[0078] In some implementations, the framework 300 uses an architecture-aware analytical cost model 114 to predict the cost of each trial (hardware / neural configuration) because (1) cycle-accurate simulation for each trial is time-consuming and there are often millions to billions of unique design points to evaluate, and (2) because neural network calculations are computationally intensive and can be expressed as nested loops, analytical models can be constructed with high fidelity. The optimization and tuning module 112 samples the search space, queries the cost model 114 for the cost of each design point, and follows a specific exploration trajectory to search the design space 104. The cost of each design point and the exploration trajectory of the design space 104 are implemented to optimize candidate architectures by tuning the architecture to minimize, at least, the processing cost of each design point. In some cases, the exploration trajectory differs due to different tuner algorithms employed by the tuner 116.

[0079] 4 is a flow diagram of an example process 400 for scheduling graph execution of a multi-layer neural network. As described above, the global tuner 202 generates an output configuration 230 that is used to automatically generate an application-specific ML accelerator. The system 200 uses an example code generation module 240 to convert data representing the output configuration 230 into HLS code.

[0080] The neural network graph 402 is for a customized, application-specific ML accelerator and illustrates an example assignment or mapping for a set of neural network layers. In the example of Figure 4, a first neural network layer L1 may be mapped to a given PE based on a particular hardware configuration 404a and software configuration 404b, while a second, different neural network layer L2 may be mapped to a given PE based on a particular hardware configuration 406a and software configuration 406b. In some implementations, L1 and L2 may be mapped to the same PE or different PEs.

[0081] 5 is a flow diagram illustrating an example process 500 for generating and globally tuning an application-specific machine learning accelerator. Process 500 can be implemented or performed using system 100 described above. The description of process 500 may refer to the above-mentioned computing resources of system 100. The steps or actions of process 500 may be enabled by programmed firmware or software instructions executable by one or more processors of the devices and resources described herein.

[0082] Referring now to process 500, system 100 selects 502 an architecture. For example, a controller of system 100 may select a candidate architecture that represents a baseline processor configuration. The candidate architecture may include a hardware architecture and a neural architecture that corresponds to the neural network graph. In some implementations, the architecture is identified and selected based on search operations performed by design space builder 204 and design space explorer 212 against the hardware layouts in architecture repository 104 and the neural architectures in network graph module 108.

[0083] The system 200 can implement the NAS and hardware architecture search techniques based on one or more tuner variables or PPA constraints 210. The PPA constraints can be user-specified goals 102 that define performance requirements for the hardware accelerator. For example, the requirements can be thresholds for processor utilization, power consumption, processing latency, and data throughput. In some implementations, selecting an architecture includes obtaining input criteria that specify performance goals and identifying multiple candidate architectures for implementing the special-purpose processor. For example, control logic for managing the design space 104, including the design space builder 204 and the explorer 212, can select a candidate architecture from among multiple candidate architectures based on the input criteria.

[0084] The system 100 generates performance data for the architecture (504). For example, the ML cost model 214 generates the performance data for the candidate architecture by modeling at least how the architecture performs computations of a first neural network that includes multiple neural network layers. In some implementations, the neural network is a known neural network, such as a multi-layer ResNet-50, which is a convolutional neural network that is 50 layers deep.

[0085] The system 100 dynamically tunes the architecture based on the performance data (506). For example, based on the performance data, the optimization and tuning module 112 dynamically tunes the candidate architecture to meet one or more performance objectives. More specifically, the optimization and tuning module 112 interacts with the architecture-aware cost model 114 to model the performance of the candidate architecture for each layer of the neural network. For example, the ML cost model 214 generates performance parameters that provide an assessment of how the candidate architecture will perform when executing each layer of the neural network.

[0086] System 100 uses tuning loop 122 to evaluate, tune, and optimize the architectural implementation of the first neural network based on performance parameters. In some implementations, system 100 uses global tuning (e.g., via global tuner 202) to find an optimized per-op mapping per system for efficient neural network execution on the target hardware platform. In some other implementations, system 100 uses global tuning to find an optimized graph execution schedule whenever allowed, such as processing engine (PE) reuse across multiple layers. This is described above with reference to FIG. 3.

[0087] For example, the global tuner 202 is configured to tune a candidate architecture by remapping two or more layers (e.g., L1, L2, L5) to the same subset of computational tiles or MACs to optimize the selected neural architecture for the target application. The architecture may be optimized for an exemplary application, such as a training / inference device or an image classification workload. The control logic of the system 100 can use the timing of clocked signals to send commands and control signals to the optimization and tuning module 112 and the architecture-aware cost model 114 at the appropriate times to generate performance data used to achieve the remapping. The optimization and tuning module 112 is configured to perform application-specific tuning and optimization to generate a hardware layout for an integrated circuit that accelerates the ML workload. The optimization and tuning module 112 (and the cost model 114) can incorporate some (or all) of the functionality of the global tuner 202, such that descriptions of operations performed by the global tuner 202 are translated into operations of the optimization and tuning module 112.

[0088] The system 100 generates 508 a configuration of the ML accelerator in response to dynamically tuning the architecture. In some implementations, the tuning and optimization of step 506 is embodied in an output configuration 230 that allows for the generation of a dedicated integrated circuit having a layer-by-layer customized hardware architecture. This aspect of customization can enable hardware ML accelerator circuits to achieve orders of magnitude improvement in energy efficiency over conventional approaches based on a single, comprehensive hardware block.

[0089] For example, after optimizing and tuning the candidate architecture, system 100 generates a compatible hardware configuration 230 including various architectural features and scheduling / mapping strategies, which system 100 can use by at least code generation module 240 to generate an application-specific ML accelerator. System 200 uses code generation module 240 to convert data representing configuration 230 into high-level synthesis (HLS) code. Code generation module 240 can use the high-level synthesis language (HLS) to generate a firmware implementation of the ML algorithm for the hardware accelerator. System 100 can then generate 510 the application-specific hardware ML accelerator based on the firmware implementation and HLS operations.

[0090] 6 is a block diagram of an exemplary application-specific hardware ML accelerator 600. Hardware accelerator 600 is generated using the techniques disclosed herein, including at least exemplary operations of systems 100 and 200. Using code generator 240, system 100 is configured to generate a hardware layout for application-specific ML accelerator 600 that specifies respective portions of hardware circuitry, each of which may be customized to operate a particular layer of a neural network.

[0091] The hardware accelerator 600 can use separate hardware blocks 603a, 603b, 603c, 603d, 603e, and 603f to execute one or more layers (e.g., if they share common characteristics) in a streaming and pipelined manner. Each hardware block 603 is specifically tuned to those layers (e.g., quantization, layer-specific tiling, systolic array dimensions, etc.) to enable low power and high utilization across the hardware accelerator 600. In some implementations, each hardware block 103 has an association or mapping with a specific layer of a neural network, and the association of the hardware block 103 with a layer of the neural network (e.g., L1, L2, L3, L4, or L5, as described above) is based, in part, on the features and optimization effort associated with that layer of the neural network.

[0092] Data flow instructions 601a, 601b, 601c, 601d, 601e, and 601f provide an example sequence of communicating neural network data between hardware blocks 603. In some implementations, these data flow instructions 601a, 601b, 601c, 601d, 601e, and 601f are pre-configured communication sequences based, for example, on optimization and tuning operations of global tuner 202. The communicated neural network data can include computation result data, such as outputs of computational units in particular hardware blocks 603, neural network inputs / activations, parameter weight data, and other neural network parameter-related data.

[0093] Each hardware block 603 may include a microarchitecture customized for a target application. The global tuner 202 is configured to optimize communication across different hardware blocks during its global tuning operation to balance the architecture design at the system level. Such optimizations include interface tiling for rate matching in data transmission, the number of computation blocks (e.g., input channel blocking) for rate matching in computation, buffer sizing, etc. For example, hardware block 603a may include inter-die input blocks 606a, 609b, inter-die output blocks 611a, 611b, and a host interface unit 613, while hardware block 603b includes inter-die input blocks 621a, 621b, inter-die output blocks 623a, 623b, and a host interface unit 614.

[0094] A customized configuration of accelerator 600 can include the first layer of a neural network being mapped to hardware block 603a and the last layer of the neural network being mapped to hardware block 603d. Global tuner 202 can configure this architecture to incorporate, for example, a feedback layer between hardware blocks 603a, 603d to balance the interaction between per-op space mapping for efficient neural network execution and the size / area constraints of PPA constraints 210. For example, hardware accelerator 600 can be configured to use the least amount of hardware to efficiently perform neural network computations while still being able to match throughput / latency based on application-specific requirements.

[0095] 7 illustrates an example tensor or multidimensional matrix 700 that includes an input tensor 704, variations of a weight tensor 706, and an output tensor 708. Tensor 700 is an exemplary machine learning data structure that may be processed or generated using an ML hardware accelerator, such as accelerator 600. For example, system 100 can be used to tune and optimize candidate architectures for processing at least tensors 704 and 706 and automatically generate a customized hardware ML accelerator 600 configured to implement a neural network that receives and processes data associated with these tensors.

[0096] Each of the tensors 700 includes elements corresponding to data values ​​for a computation performed in a given layer of the neural network. The computation may include multiplication of an input / activation tensor 704 with a parameter / weight tensor 706 over one or more clock cycles to generate an output, such as an activation / output value that can be provided as an input to another neural network layer. In the example of FIG. 7 , each output in the set of outputs may correspond to a respective element of an output tensor 708. In some examples, the input tensor 704 is an activation tensor. Multiplying the activation tensor 704 with the corresponding weight tensor 706 includes multiplying the activation from an element of the tensor 704 with a weight from an element of the tensor 706 to produce a partial sum.

[0097] In some implementations, the hardware blocks 603 of the ML accelerator 600 are respective processor cores operating on vectors that may contain multiple separate elements along the same (or different) dimensions of several multidimensional tensors. Each of the multiple elements can be represented using X, Y coordinates (2D) or X, Y, Z coordinates (3D), depending on the dimensionality of the tensor. The hardware layout of the ML accelerator 600 can be optimized to compute multiple partial sums according to a given set of PPA constraints. The partial sums correspond to products resulting from multiplying batch inputs by corresponding weight values.

[0098] An input weight multiplication may be written as a sum of products of each weight element multiplied by a separate input of an input volume, such as a row or slice of an input tensor 704. The row or slice may represent a given dimension, such as a first dimension 710 of the input tensor 704 or a second, different dimension 715 of the input tensor 704. The dimensions may be mapped to various vector processing units across the hardware block 603, such that the ML accelerator 600 periodically performs its calculations in a manner that eliminates load imbalance and achieves thresholding utilization in each hardware block 603 according to a given set of input targets 102.

[0099] In some implementations, an exemplary set of computations can be used to calculate outputs for a convolutional neural network layer. The computations for a CNN layer can include performing a 2D spatial convolution between a 3D input tensor 704 and at least one 3D filter (weight tensor 706). For example, convolving one 3D filter 706 on the 3D input tensor 704 can generate a 2D spatial plane 720 or 725. The computations can include calculating sums of dot products for particular dimensions of the input volume. For example, spatial plane 720 can include output values ​​for sums of products computed from inputs along dimension 710, while spatial plane 725 can include output values ​​for sums of products computed from inputs along dimension 715. The computations for generating the sums of products for the output values ​​in each of spatial planes 720 and 725 can be performed using hardware block 603 generated and tuned using techniques described herein.

[0100] Embodiments and functional operations of the subject matter described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware containing the structures disclosed herein and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or to control the operation of a data processing apparatus.

[0101] Alternatively or additionally, the program instructions may be encoded in an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these.

[0102] The term "computing system" encompasses all kinds of apparatus, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. An apparatus can include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus can also include code that creates an execution environment for the computer program in question, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0103] The computer program (which may also be called or described as a program, software, software application, module, software module, script, or code) can be written in any type of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0104] A computer program may, but need not, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple coherent files, e.g., files storing one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0105] The processes and logic flows described herein can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and an apparatus can be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit).

[0106] A computer suitable for executing a computer program can be based, for example, on a general-purpose or special-purpose microprocessor, or both, or on any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. Some elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to receive or transmit data therefrom, or both. However, a computer need not have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0107] Computer-readable media suitable for storing computer program instructions and data include all types of non-volatile memory, media, and memory devices, such as semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0108] To provide for user interaction, embodiments of the subject matter described herein can be implemented in a computer having a display device, e.g., an LCD (liquid crystal display) monitor, for displaying information to a user, and a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction. For example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback. Input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0109] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components, e.g., as a data server, or includes middleware components, e.g., an application server, or includes front-end components, e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), e.g., the Internet.

[0110] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0111] While the specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be particular to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as operative in a combination and even initially claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to subcombinations or variations of the subcombination.

[0112] Similarly, although operations are shown in the figures in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown, or sequentially, or that all illustrated operations be performed, to achieve desired results. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0113] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order or sequence shown to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. 1. A computer-implemented method for generating an application-specific machine learning (ML) accelerator, comprising: selecting an architecture representing a baseline processor configuration; modeling, using an ML cost model, how the architecture performs computations of a first neural network including multiple layers; generating performance data for the architecture by the ML cost model in response to modeling the architecture performing a first neural network calculation; dynamically tuning the architecture to meet a performance target representing an expected performance of the architecture when the architecture implements the first neural network and performs machine learning computations for a target application based on the performance data; determining a customized hardware configuration for implementing each of the plurality of layers of the first neural network in response to dynamically tuning the architecture; generating a configuration of an ML accelerator based on the dynamically tuned architecture and the customized hardware configuration.

2. generating an application-specific hardware ML accelerator based on the customized hardware configuration; 2. The method of claim 1, wherein the application-specific hardware ML accelerator is optimized to implement each of the layers of the first neural network when the first neural network is used to perform computations for the target application.

3. The performance goal includes a plurality of separate goals, and generating the application-specific ML accelerator comprises:

3. The method of claim 2, comprising generating an application-specific hardware ML accelerator configured to meet each separate goal of the plurality of separate goals when the application-specific hardware ML accelerator performs calculations for the target application.

4. The step of generating performance data includes: modeling the use of the architecture to implement each of the plurality of layers of the first neural network with the ML cost model; and generating performance parameters of the architecture for each of the plurality of layers with the ML cost model in response to modeling the use of the architecture to execute each layer.

5. the performance parameter corresponds to each separate goal of the plurality of separate goals; The method of claim 4 , wherein the plurality of separate goals comprises at least one of a threshold processing latency, a threshold power consumption, a threshold data throughput, and a threshold processor utilization.

6. The step of dynamically tuning the architecture includes: determining a mapping of computations for an input tensor that causes the application-specific hardware ML accelerator to utilize a threshold percentage of hardware computation units of the hardware ML accelerator when the application-specific hardware ML accelerator processes the input tensor; and dynamically tuning the architecture based on the determined mapping.

7. The step of dynamically tuning the architecture includes: dynamically tuning the architecture based on operations performed by each of a plurality of ML cost models of a global tuner; and dynamically tuning the architecture based on operations performed by at least one of a random tuner or a simulated annealing tuner of the global tuner.

8. The architecture is for an integrated circuit and includes one or more hardware blocks of the integrated circuit, and the step of dynamically tuning the architecture includes, for the one or more hardware blocks:

7. The method of claim 6, further comprising dynamically tuning the architecture to meet respective performance targets for the hardware blocks when the architecture implements the first neural network and performs computations for the target application using the first neural network.

9. the configuration of the hardware ML accelerator specifies a customized software configuration for the first neural network; 7. The method of claim 6, wherein generating the application-specific hardware ML accelerator comprises generating the application-specific hardware ML accelerator based on the customized hardware configuration and the customized software configuration.

10. the ML cost model is an architecture-aware cost model that includes one or more individual analytical models; The method of claim 6 , wherein the architecture-aware cost model is configured to estimate the performance of the architecture based on a deterministic data flow of data processed using the architecture.

11. 1. A system including a processing unit and a non-transitory machine-readable storage device storing instructions for creating an application-specific machine learning (ML) accelerator, the instructions being executable by the processing unit to cause performance of operations, the operations including: selecting an architecture representing a baseline processor configuration; modeling, using an ML cost model, how the architecture performs computations of a first neural network including multiple layers; generating performance data for the architecture by the ML cost model in response to modeling the architecture performing a first neural network calculation; dynamically tuning the architecture to meet a performance target representing an expected performance of the architecture when the architecture implements the first neural network and performs machine learning computations for a target application based on the performance data; determining a customized hardware configuration for implementing each of the plurality of layers of the first neural network in response to dynamically tuning the architecture; generating a configuration of an ML accelerator based on the dynamically tuned architecture and the customized hardware configuration.

12. generating an application-specific hardware ML accelerator based on the customized hardware configuration; 12. The system of claim 11, wherein the application-specific hardware ML accelerator is optimized to implement each of the plurality of layers of the first neural network when the first neural network is used to perform computations for the target application.

13. The performance goal includes a plurality of separate goals, and generating the application-specific ML accelerator comprises:

13. The system of claim 12, further comprising generating an application-specific hardware ML accelerator configured to meet each separate goal of the plurality of separate goals when the application-specific hardware ML accelerator performs calculations for the target application.

14. The step of generating performance data includes: modeling the use of the architecture to implement each of the plurality of layers of the first neural network with the ML cost model; and generating performance parameters of the architecture for each of the plurality of layers with the ML cost model in response to modeling the use of the architecture to execute each layer.

15. the performance parameter corresponds to each separate goal of the plurality of separate goals; The system of claim 14 , wherein the plurality of separate goals comprises at least one of a threshold processing latency, a threshold power consumption, a threshold data throughput, and a threshold processor utilization.

16. The step of dynamically tuning the architecture includes: determining a mapping of computations for an input tensor that causes the application-specific hardware ML accelerator to utilize a threshold percentage of hardware computation units of the hardware ML accelerator when the application-specific hardware ML accelerator processes the input tensor; and dynamically tuning the architecture based on the determined mapping.

17. The step of dynamically tuning the architecture includes: dynamically tuning the architecture based on operations performed by each of a plurality of ML cost models of a global tuner; and dynamically tuning the architecture based on operations performed by at least one of a random tuner or a simulated annealing tuner of the global tuner.

18. The architecture is for an integrated circuit and includes one or more hardware blocks of the integrated circuit, and the step of dynamically tuning the architecture includes, for the one or more hardware blocks:

17. The system of claim 16, further comprising dynamically tuning the architecture to meet respective performance targets for the hardware blocks when implementing the first neural network and using the first neural network to perform computations for the target application.

19. the ML cost model is an architecture-aware cost model that includes one or more individual analytical models; 17. The system of claim 16, wherein the architecture-aware cost model is configured to estimate performance of the architecture based on a deterministic data flow of data processed using the architecture.

20. A program that causes a processing device to execute the method according to any one of claims 1 to 10.

Images