Synthesis of Zero-Loss Low-Power Approximate DNN Accelerators by Large-Scale Search

Abstract

Systems and methods are provided for designing approximate, low-power deep learning accelerator chips that execute deep learning models with little loss of accuracy. A set of approximate systolic arrays may be generated. The performance of each approximate systolic array in the set of approximate systolic arrays for processing a deep neural network (DNN) may be determined. Each layer of the DNN may be mapped to an approximate systolic array in the set of approximate systolic arrays. A subset of the set of approximate systolic arrays may be selected for inclusion in the inference chip design based on the mapping and the performance of each approximate systolic array in the set of approximate systolic arrays.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Patent Application No. 17 / 532,572, filed November 22, 2021, the disclosure of which is incorporated herein by reference. [Background technology]

[0002] background Scaling neural network models often improves the accuracy of the outputs provided by these neural network models. However, as the size of these neural network models continues to grow, so does the amount of energy consumed by inference accelerators that implement these deep neural network models. Therefore, even small efficiency improvements in deep learning neural network models can dramatically reduce the global energy consumption by inference accelerators that implement deep learning models. Summary of the Invention [Problem to be solved by the invention]

[0003] Computation units that perform basic mathematical operations when implementing deep learning models often account for the majority of the energy consumed by inference accelerators. To improve the power efficiency of these computation units, full-precision floating-point calculations can be replaced with low-bit precision quantized operations. However, the increased efficiency of low-bit quantization can result in a decrease in the accuracy of the output by deep learning models. In some cases, model weight adjustments are performed to recover the accuracy lost through approximation. However, this weight adjustment step may not be sufficient to avoid accuracy degradation.

[0004] Further efforts to reduce energy consumption by inference accelerators implementing deep learning models include using approximation units, which are more power-efficient than quantization operators. These approximation units can implement approximation operators (e.g., multipliers and adders) that adjust the approximation to the numerical distribution observed during neural network evaluation. Tailoring the approximation to a specific application during evaluation can potentially provide a better tradeoff between power consumption and accuracy than if quantization were approximated uniformly across all inputs. However, errors introduced by the approximation units can be amplified as the deep learning model progresses through each subsequent neural network layer, potentially leading to inaccurate outputs. [Means for solving the problem]

[0005] overview The techniques described herein are directed to generating an inference chip design for production. One aspect of the present disclosure is directed to a method for generating an inference chip. The method may include generating, by one or more processors, a set of approximate systolic arrays; determining, by the one or more processors, the performance of each approximate systolic array in the set of approximate systolic arrays for processing a deep neural network (DNN); mapping, by the one or more processors, each layer of the DNN to an approximate systolic array in the set of approximate systolic arrays; and selecting, by the one or more processors, a subset of the set of approximate systolic arrays for inclusion in the inference chip design based on the mapping and the performance of each approximate systolic array in the set of approximate systolic arrays.

[0006] Another aspect of the present disclosure is directed to a system for generating an inference chip design for production. The system includes one or more processors and a memory that stores instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: generate a set of approximate systolic arrays, determine the performance of each approximate systolic array in the set of approximate systolic arrays for processing a deep neural network (DNN), map each layer of the DNN to an approximate systolic array in the set of approximate systolic arrays, and select a subset of the set of approximate systolic arrays for inclusion in the inference chip design based on the mapping and the performance of each approximate systolic array in the set of approximate systolic arrays.

[0007] In some examples, a second set of approximate systolic arrays may be generated, a performance of each approximate systolic array in the second set of approximate systolic arrays for processing the DNN may be determined, each layer of the DNN may be mapped to an approximate systolic array in the second set of approximate systolic arrays, and a subset of the systolic arrays may be updated based on the mapping and the performance of each approximate systolic array in the second set of approximate systolic arrays, wherein the updated subset of approximate systolic arrays includes at least one systolic array from the second set of approximate systolic arrays.

[0008] In some examples, the set of approximate systolic arrays is generated to satisfy an architecture template, the architecture template defining the number and / or size of the systolic arrays for the inference chip design. In some examples, each approximate systolic array in the set of approximate systolic arrays is generated using a predefined multiply-accumulate unit (MAC), and the predefined MAC is stored in a bank.

[0009] In some examples, each approximation systolic array in the set of approximation systolic arrays is generated using multiply-accumulate units (MACs), and at least one of the MACs is generated based on predefined criteria. In some examples, the predefined criteria include one or more of power usage or accuracy.

[0010] In some examples, determining the performance of each approximate systolic array includes determining a power usage of each approximate systolic array. In some examples, determining the performance of each approximate systolic array further includes determining an accuracy of each approximate systolic array.

[0011] In some examples, mapping each layer of the DNN to an approximation systolic array in the set of approximation systolic arrays includes configuring, for each layer of the DNN, a router in the inference chip design to a particular approximation systolic array in the subset of approximation systolic arrays based on the power usage and accuracy of each approximation systolic array.

[0012] In some examples, the inference chip design includes at least one full-precision systolic array. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram of an exemplary framework for generating an optimized inference chip, according to aspects of the present disclosure. [Figure 2] FIG. 1 is a diagram of an optimized inference chip according to an aspect of the present disclosure. [Figure 3] 1 is a graph illustrating principal components of various MACs, according to an embodiment of the present disclosure. [Figure 4] 10 is a graph illustrating power consumption of a single MAC and a systolic array according to an embodiment of the present disclosure. [Figure 5]10 is a graph illustrating the operating temperatures of an optimized MXU including a collection of approximate systolic arrays versus an MXU formed from an exact systolic array, in accordance with aspects of the present technique. [Figure 6] FIG. 1 is a block diagram of an exemplary computing environment for implementing a framework, in accordance with aspects of the present technology. DETAILED DESCRIPTION OF THE INVENTION

[0014] Detailed Description The techniques described herein are directed to designing approximate, low-power deep learning accelerator chips that execute deep learning models with little loss of accuracy. Furthermore, the approximate, low-power deep learning accelerator chips described herein ("inference chips") do not need to retrain deep learning models to maintain accuracy. Instead, the inference chips are designed to process deep learning models with the same or nearly the same accuracy as chips constructed from "exact" units (e.g., arithmetic units capable of full-precision calculations).

[0015] An inference chip may be designed using an architecture template. This template can generate a diverse set of efficient designs with a reconfigurable routing array to banks of systolic arrays containing approximation units, such as approximate adders and / or approximate multipliers. This allows error-tolerant layers in a deep learning model to be dynamically routed to more approximation units in the inference chip accelerator at run time, resulting in significant power savings. Furthermore, layers in the deep learning model that are more sensitive to errors can be evaluated with more accurate approximation units and / or precise units, minimizing or avoiding the occurrence of errors. By co-designing the approximation units on the inference chip with the software mapping of layers in the deep learning model to the systolic array, power savings can be achieved without sacrificing accuracy.

[0016] Figure 1 is a flow diagram illustrating a design process for creating a low-power, high-precision inference chip. As described herein, the task of selecting an approximate multiply-and-remove unit for an inference chip design occurs simultaneously with mapping DNN layers to the chip's systolic array, which contains a collection of processing elements (PEs).

[0017] Each systolic array can include a two-dimensional array of PEs, each of which includes one or more multiply-accumulate units (MACs). Additionally, each PE can include buffers for input operands and output results. Each systolic array is globally activated, and parameter memory is shared among all systolic arrays. Typically, each systolic array includes a single type of MAC, but some systolic arrays may include multiple types of MACs. Additionally, while MACs are discussed in the examples described herein, other arithmetic units can also be used (e.g., adders). By designing the hardware and simultaneously mapping DNN layers to the systolic array, the inference chip can be custom-tuned for a specific class of neural network.

[0018] The flow diagram illustrates an inference chip design framework 100 for generating systolic arrays of approximate and / or exact units that satisfy a given architecture template 101 and architecture parameters 115 input to an accelerator architecture generator 106, including a systolic array generator 105. The architecture template 101 of the framework 100 can define requirements corresponding to the layout and configuration of the inference chip. For example, the architecture template 101 can define the number of systolic arrays and the size of the systolic arrays of the inference chip. For example, the architecture template can define that the inference chip has three systolic arrays, two composed of approximate units and one composed of exact units. Other templates may define other quantities and types of arrays, which approximate units (e.g., MACs) to use, etc. The architecture parameters 115 can include the configuration of heterogeneous systolic arrays, memory and system parameters, etc. During initial processing by the framework 100, the architecture parameters may be "seeded." The seeded parameters may be predefined or input each time the framework starts the process.

[0019] MACs that can be used to create processing elements in a systolic array can be stored in bank 103 of the framework. The MACs can have attributes that meet certain criteria, such as error tolerance and power usage. For example, a first MAC can have attributes such as a 1% error for an exact MAC and a 99% power usage for an exact MAC. In another example, a second MAC can have a 3% error for an exact MAC and a 90% power usage for an exact MAC. Bank 103 can store any number of configured MACs, possibly including an exact MAC that performs full-precision calculations (rather than approximate calculations). In some cases, the MACs can be generated by the framework. In this regard, the framework can generate MACs that fit a particular use case, such as a MAC with a particular error rate and power usage. During the creation of a systolic array, the framework can generate MACs that fit a particular use case that is not met by the predefined MACs stored in bank 103.

[0020] The systolic array generator 105 of the accelerator architecture generator 106 can use the MACs stored in the bank 103 to generate a diverse set of systolic arrays to meet the requirements of a given architecture template (and architecture parameters 115) while ensuring that the set of systolic arrays can accurately execute the DNN 107. In this regard, the accelerator architecture generator 106 can receive the architecture template 101 and the architecture parameters 115. The accelerator architecture generator 106 can then provide some or all of the received data to the systolic array generator. For example, the accelerator architecture generator 106 may inform the systolic array generator 105 of parameters of the systolic array design, such as the size and configuration of the systolic array (e.g., an inference chip includes x% MACx and y% MACy). The MACx and MACy can be selected from the bank 103.

[0021] The systolic array generator 105 may generate a set of systolic arrays according to the received parameters. In this regard, the framework's systolic array generator 105 may select MACs with varying precision and power attributes according to the received parameters. The systolic array generator 105 may generate code representing the systolic arrays, such as in Verilog, C++, or other such languages.

[0022] Each layer of the DNN 107 may be input to a layer-wise mapping 120 function of the framework 100. The layer-wise mapping 120 function may assign each layer of the DNN 107 to a specific systolic array of the set of systolic arrays generated by the systolic array generator 105. As described herein, the optimal approximation level for processing a DNN can vary dramatically between different layers of a deep neural network. The layer-wise mapping function 120 may map each DNN level based on its approximation tolerance.

[0023] As described herein, the power and accuracy of each generated set of systolic arrays may be tested, and an optimal set of systolic arrays may be selected for an inference chip design custom-tuned to run the DNN 107. While DNNs are described herein, the design process for creating low-power, high-accuracy inference chips may be performed for other types of neural networks. For example, each generated set of systolic arrays may be tested for accuracy using fast accuracy estimation 109, as described herein. Additionally, power and performance estimates (collectively “performance estimates”) may be determined for each generated set of systolic arrays, as indicated by block 111 and described herein.

[0024] The accuracy and performance estimates may be input into an objective function 112 described herein to determine an optimized combination of systolic arrays. In this regard, the objective function 112 may evaluate how well each DNN layer mapping performs in terms of accuracy and performance.

[0025] Next, architecture search algorithm 113 can take the objective function results (and other previous results of the objective function) and determine whether improvements are possible with a different set of systolic arrays. In this regard, architecture search algorithm 113 can receive current objective function results for the current parameters and past objective function results for previous parameters. Based on these parameters, architecture search algorithm 113 can determine the next set of parameters to try (illustrated by the arrows from 113 to 115 in FIG. 1 ). Framework 100 aims to find the optimal value of the objective function through an iterative search of the parameter space that results in the optimal configuration.

[0026] This process can be repeated a certain number of times, or until all possible combinations of systolic arrays, layer-wise mappings, etc. are exhausted. The most efficient and accurate set of systolic arrays is then selected. Although not shown, the framework can output a design plan for configuring the inference chip according to the selected set of systolic arrays.

[0027] An optimized combination of systolic arrays can constitute a multi-tile matrix multiplication unit (MXU). Each "tile" of a multi-tile MXU can contain a systolic array with specific precision and power attributes. For example, Figure 2 shows a multi-tile MXU 205 containing three systolic arrays, including array A 207, array B 209, and array Z 211. While the MXU 205 is shown containing only three arrays, the MXU can contain any number of systolic arrays. As further shown in Figure 2, activation buffer 201 can receive results from previous layers of the DNN. These previous results are routed by learned router 203 to a systolic array for processing the next layer of the DNN. Layer routing may be pre-programmed so that the router directs specific layers of the DNN to predefined systolic arrays for processing. For example, Figure 2 shows the current DNN layer being processed by array Z 211.

[0028] Accuracy Estimation Both training and inference of large deep neural networks can involve trillions of arithmetic operations. Using exact arithmetic and standard data types, this evaluation can be parallelized on high-throughput GPUs. However, simulating inexact arithmetic operations using approximate MACs can be challenging because existing hardware does not support such operations. For example, directly evaluating approximate MACs in a circuit simulator like the Verilator circuit simulator can take 3.75 ± 0.95 microseconds on a high-performance server for a single 8-bit exact multiplication. Therefore, evaluating a large neural network like a 50-layer deep ResNet-50 can take approximately 4.2 hours at 4 GFLOPs per 224 × 224 frame. On the entire ImageNet validation set, evaluating one approximate multiplier can take approximately 23 years.

[0029] Furthermore, the call may be cached, but the full 2 ​​for an N-bit multiplier N+N The lookup table may not be scalable to wider bit widths. For example, for a 16-bit input, the lookup table would be over 16 gigabytes. Due to its large size, it is not possible to use GPU acceleration.

[0030]

number

[0031] N-bit error matrix ε∈R 2N+N To save memory when storing , a truncated singular value decomposition is performed for k<<2 using the following equation (1): N It may be calculated as:

[0032]

number

[0033] The total memory consumption is O(n 2 ) to O(nk). In equation (1), Σ is the error matrix, σ i is the i-th eigenvalue of the error matrix, u i and v i are the i-th left singular vector and right singular vector of the error matrix, respectively. Parameter k<<2 N controls the amount of approximation, and a smaller k increases the amount of approximation but reduces the memory required to store the error matrix. For small k≦50, the total memory consumption is less than 20MB. During evaluation, the result of the i×j approximate multiplication may be recalculated.

[0034] All results may be evaluated using a classification dataset, such as a large-scale image dataset like the ImageNet2012 dataset. To improve the evaluation speed of the end-to-end model on the target dataset, one can evaluate a small sample, such as a 10% sample of the full validation set. For example, when using ImageNet2012, one can use a sample of 5000 images. The ordering of the model on the sample validation set usually correlates with its performance on the full dataset.

[0035] Overall, the optimizations described here can result in a speedup of 7200x compared to direct circuit simulation in Verilator. This strategy also makes GPU evaluation with automatic retraining feasible in the future. Compressing lookup tables with low-rank decomposition allows the model evaluation described here to be extended to circuits with wider bit widths.

[0036] Furthermore, the approach described herein is complementary to other known quantization methods. For example, post-training quantization of dynamic range may be performed, where weights are statically quantized to 8 bits before inference. During inference, activities may be scaled to the uint8 range of [0; 255] and then quantized, although other ranges may be used. They can then be dequantized using Equation (2) below.

[0037]

number

[0038] where q1 represents the weight matrix, q2 represents the activation matrix, and Z represents the zero point of each. More advanced quantization methods using quantization-aware training can achieve higher power savings.

[0039]

number

[0040] Semiconductor design and test software can be used to evaluate the performance of systolic arrays across an MXU. Testing is capable of evaluating semiconductor designs in sub-10 nm processes. For evaluation, a single clock domain is applied to each systolic array, and the clock frequency is determined by the slowest systolic array. In this regard, the clock frequency is typically based on the "exact" systolic array, but other "approximate" systolic arrays may be slower and therefore used to set the clock frequency. Furthermore, while this example describes a single clock frequency, some or all systolic arrays in an MXU design may be tested at different frequencies. During synthesis, different MXU variants are constrained to the slowest clock frequency. This allows the synthesis tool to select smaller gate sizes for the approximate MXU variants with higher inherent performance, resulting in some power and area savings.

[0041] In practice, DNN accelerators are typically thermal design power (TDP) limited, rather than latency limited. Therefore, further gains can be obtained by overclocking the inference chip using the thermal savings from the approximate systolic array. For example, Figure 5 shows that an optimized approximate MXU runs at significantly lower temperatures compared to an MXU with only an "exact" systolic array.

[0042] Circuit Search and Layer Mapping Formulation of the approximate circuit mapping problem. The performance of a particular design also depends on how the nests of convolutional loops are mapped to the arrays: mapping DNN layers to specific systolic arrays ensures that the resulting accelerator matches what happens after synthesis.

[0043] To maintain high end-to-end task accuracy, it is necessary to consider which layers of a deep learning model are approximation tolerant. In this regard, the optimal approximation level can vary dramatically between different layers of a deep neural network. However, it is difficult to simultaneously consider the tasks of selecting an approximation systolic array for chip design and mapping layers of a DNN to the systolic array. In this respect, these two subproblems (selecting an approximation unit for chip design and mapping layers of a DNN to the approximation unit) are each challenging combinatorial optimization problems. Together, they result in an O(K) optimization problem of mapping K MAC design candidates to N deep neural network layers. N ) search space. O is the "big O" notation that represents the complexity of the search space. Furthermore, 268 There may be many workloads with a search space, more or less. In this case, the search space defines how to design an approximate MAC. The goal of the optimization algorithm is to find an optimal design within this search space. Therefore, random searching will not work.

[0044] Bayesian optimization using Gaussian process (GP) bandits can be leveraged to efficiently discover high-accuracy yet energy-efficient cross-layer approximation circuit configurations. This approach improves the sample efficiency of black-box optimization by modeling the unknown reward function f:x → y with a Gaussian process.

[0045] To formulate the approximate circuit mapping problem, consider the following optimization problem, which seeks the minimum-power mapping of approximate circuits onto deep neural network layers:

[0046]

number

[0047]

number

[0048] The accuracy oracle, "ACC," models the impact of cross-layer interactions due to approximation. Given a specific assignment of approximation multipliers to layers, ACC calculates the expected accuracy of the model on a particular dataset. Accuracy oracles are very difficult to model because errors made in one layer compound in subsequent layers. Therefore, model accuracy is measured on a validation set.

[0049] This reduces the optimization problem to a black-box combinatorial optimization problem. Furthermore, Bayesian optimization methods typically fail when applied to the above optimization problem because they exhibit slow convergence with performance similar to random search. Furthermore, Bayesian optimization struggles with high-dimensional states, discrete structures, and constrained search spaces. Therefore, the above optimization problem can be reformulated to allow the use of known Bayesian optimization tools.

[0050] Calibration of the single layer approximation To find a highly accurate approximate circuit design, feasible solutions may be constrained by the minimum validation accuracy threshold in constraint (2b). However, the accuracy oracle (ACC) is not known. Therefore, to reduce the complexity of the search space, an offline study may be performed using only approximate multipliers for the target layer. All other layers can be evaluated using exact multipliers. This model provides an upper bound on the expected accuracy from cross-layer approximations. Mappings with exceptionally low expected accuracy may be pruned.

[0051] Continuous relaxation of the state space GP bandits are primarily designed to optimize over discrete search spaces. Bayesian optimization frameworks typically support discrete search spaces by embedding them into real-valued boxes. However, this embedding is sample-inefficient because it does not consider the relationships between different categorical variables. For example, this solution suffers from instability due to quantization errors caused by rounding continuous predictor variables to the nearest feasible point.

[0052] To address this issue, we can use estimates of per-layer accuracy degradation from single-layer approximations to compute an ordered set representing the relative ordering of each approximate multiplier. In this regard, the ordering can be defined as the profiled end-to-end accuracy for approximating a single layer k with a particular multiplier. However, direct search with unrounded precision is an unstable relaxed optimization problem. As a result, two approximate multipliers achieve similar accuracy but consume significantly different power. Therefore, the linear ordering of multipliers can be relaxed to a partially ordered set in which equivalent values ​​within a fixed threshold of precision are considered incomparable. The resulting linear ordering can then be resolved by removing the least efficient multiplier from each pair of incomparable multipliers with similar precision.

[0053] This procedure derives a linear ordering of multipliers for each of the N layers of the neural network. To define the distance in the new dimension after mapping, we can apply min-max scaling to the top-1 accuracy (i.e., high probability) of each multiplier obtained from the single-layer approximate calibration. Given this new formulation of the search space, we define the following cost optimization objective: for each of the N layers, we define a stepwise cost function Q to map the choice of real-valued approximate multipliers (between 0 and 1) to the energy consumption of the nearest layer, with truncation. i :R→R may be defined.

[0054] The relaxed optimization problem then becomes:

[0055]

number

[0056] Unconstrained optimization using barrier functions Recent research has begun to explore multi-objective optimization using Bayesian optimization, but these approaches are generally significantly less sample-efficient than single-objective optimizers. To optimize inference chip designs, one can consider the two-dimensional Pareto frontier between accuracy and energy consumption. In practice, it may be useful to also limit the area of ​​the final systolic array to avoid degenerate solutions in which redundant approximate multipliers with similar accuracy are instantiated on a single chip.

[0057] The barrier method can be used to remove constraints (3b) and (3c). The barrier method removes each constraint of the form x ≤ b by solving for the objective function β(x,b) = -log(bx) or β(x;b) = e x-b As x approaches the constraint b, the penalty goes to ∞. Using the barrier method, the objective is expressed as

[0058]

number

[0059] This updated objective allows for the removal of constraints (3b) and (3c). An exponential barrier function can be utilized to allow for soft constraint violations. In terms of precision, the target precision τ can be 0.68 or higher or lower, and the weight α2 can be 8 or higher or lower. In terms of area, the target area ratio (including the exact scale factor) can be φ=400% or higher or lower, and the scale α2 can be 1:2 or higher or lower.

[0060] Example computing environment 6 is a block diagram of an exemplary computing environment 600 implementing the exemplary framework as shown in FIG. 1. For example, the framework may be implemented in one or more devices having one or more processors in one or more locations, such as a server computing device 615. The user computing device 612 and the server computing device 615 may be communicatively coupled to one or more storage devices 630 via a network 660. The storage device 630 may be a combination of volatile and non-volatile memory and may be in the same physical location as the computing devices 612, 615 or in a different physical location. For example, the storage device 630 may include any type of non-transitory computer-readable medium capable of storing information, such as a hard drive, a solid-state drive, a tape drive, optical storage, a memory card, ROM, RAM, DVD, CD-ROM, writable memory, and read-only memory.

[0061] The server computing device 615 may include one or more processors 613 and memory 614. The memory 614 may store information accessible by the processor 613, including instructions 621 that may be executed by the processor 613. The memory 614 may also include data 623 that may be retrieved, manipulated, or stored by the processor 613. The memory 614 may be a type of non-transitory computer-readable medium that can store information accessible by the processor 613, such as volatile and non-volatile memory. The processor 613 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0062] Instructions 621 may include one or more instructions that, when executed by processor 613, cause one or more processors to perform the operations defined by the instructions. Instructions 621 may be stored in object code format for direct processing by processor 613 or in other formats, including interpretable scripts or a collection of independent source code modules that are interpreted on demand or pre-compiled. Instructions 621 may include instructions for implementing framework 100 consistent with aspects of the present disclosure. Framework 100 may be executed using processor 613 and / or using other processors located remotely from server computing device 615.

[0063] Data 623 may be retrieved, stored, or modified by processor 613 according to instructions 621, such as a bank of approximate MACs 103, architecture templates 101, architecture parameters 115, etc. Data 623 may be stored in a computer register, a relational or non-relational database, as a table with multiple different fields and records, or as a JSON, YAML, proto, or XML document. Data 623 may also be formatted in a computer-readable format, such as, but not limited to, binary values, ASCII, or Unicode. Furthermore, data 623 may include sufficient information to identify related information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memory, including other network locations, or information used by a function to calculate the related data.

[0064] The user computing device 612 may also be configured similarly to the server computing device 615, with one or more processors 616, memory 617, instructions 618, and data 619. The user computing device 612 may also include a user output 626 and a user input 624. The user input 624 may include any suitable mechanism or technology for receiving input from a user, such as a keyboard, a mouse, a mechanical actuator, a soft actuator, a touchscreen, a microphone, and a sensor.

[0065] The server computing device 615 may be configured to transmit data to the user computing device 612, which may be configured to display at least a portion of the received data on a display implemented as part of the user output 626. The user output 626 may also be used to display an interface between the user computing device 612 and the server computing device 615. The user output 626 may alternatively or additionally include one or more speakers, transducers, or other audio outputs, haptic interfaces, or other tactile feedback that provide non-visual and non-audible information to a platform user of the user computing device 612.

[0066] While FIG. 6 illustrates processors 613, 616 and memories 614, 617 as being within computing devices 615, 612, the components described herein, including processors 613, 616 and memories 614, 617, may include multiple processors and memories that may operate in different physical locations rather than within the same computing device. For example, some of the instructions 621, 618 and data 623, 619 may be stored on a removable SD card, while others may be stored in a read-only computer chip. Some or all of the instructions and data may be stored physically remote from but still accessible by the processors 613, 616. Similarly, processors 613, 616 may include a collection of processors capable of performing simultaneous and / or sequential operations. Computing devices 615, 612 may each include one or more internal clocks that provide timing information, which may be used to time operations and programs executed by the computing devices 615, 612.

[0067] The server computing device 615 may be configured to receive requests to process data from the user computing device 612. For example, the environment 600 may be part of a computing platform configured to provide various services to users through various user interfaces and / or APIs that expose platform services. One or more services may be a machine learning framework or set of tools for generating a neural network or other machine learning model according to a specified task and training data. The user computing device 612 may receive and transmit data specifying target computing resources to be allocated to run a neural network trained to perform a particular neural network task.

[0068] The devices 612, 615 may be capable of direct and indirect communication over the network 660. The devices 615, 612 may establish listening sockets capable of accepting initiated connections to send and receive information. The network 660 itself may include a variety of configurations and protocols, including the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local network, and a private network using one or more proprietary company communication protocols. The network 660 may support a variety of short- and long-range connections. The short- and long-range connections may be made over different bandwidths, such as 2.402 GHz to 2.480 GHz (commonly associated with the Bluetooth® standard), 2.4 GHz and 5 GHz (commonly associated with the Wi-Fi® communication protocol), or various communication standards, such as the LTE® standard for wireless broadband communication. The network 660 may additionally or alternatively support wired connections between the devices 612, 615, including various types of Ethernet® connections.

[0069] 6, it is understood that aspects of the disclosure may be implemented according to a variety of different configurations and quantities of computing devices, including in paradigms for serial or parallel processing, or in a distributed network of multiple devices. In some implementations, aspects of the disclosure may be performed on a single device, and any combination thereof.

[0070] Aspects of the present disclosure may be implemented in digital circuitry, in a computer-readable storage medium, as one or more computer programs, or as a combination of one or more of the foregoing. The computer-readable storage medium may be non-transitory, for example, executable by a cloud computing platform, and as one or more instructions stored on a tangible storage device.

[0071] The phrase "configured to" is used herein in various contexts with respect to a computer system, hardware, or portion of a computer program, engine, or module. When a system is said to be configured to perform one or more operations, this means that the system has appropriate software, firmware, and / or hardware installed on the system that, when operated, causes the system to perform one or more operations. When hardware is said to be configured to perform one or more operations, this means that the hardware includes one or more circuits that, when operated, receive inputs and, in accordance with the inputs, generate outputs corresponding to the one or more operations. When a computer program, engine, or module is said to be configured to perform one or more operations, this means that the computer program includes one or more program instructions that, when executed by one or more computers, cause the one or more computers to perform one or more operations.

[0072] Although the operations illustrated in the figures and recited in the claims are shown in a particular order, it is understood that the operations may be performed in an order different from that shown, and that some operations may be omitted, performed multiple times, and / or performed in parallel with other operations. Furthermore, the separation of different system components configured to perform different operations should not be understood as requiring the components to be separated. The described components, modules, programs, and engines may be integrated together as a single system or may be part of multiple systems. One or more processors at one or more locations implementing an exemplary STOC according to aspects of the present disclosure may perform the operations illustrated in the figures and recited in the claims.

[0073] Unless otherwise specified, the foregoing alternatives are not mutually exclusive and can be implemented in various combinations to achieve unique advantages. Because these and other variations and combinations of the features described above can be utilized without departing from the subject matter defined by the claims, the description of the foregoing examples should be construed as illustrative, not limiting, of the subject matter defined by the claims. Additionally, the provision of examples described herein, as well as phrases such as "such as," "including," and the like, should not be construed as limiting the subject matter of the claims to any particular example; rather, the example is intended to illustrate only one of many possible implementations. Furthermore, the same reference numbers in different drawings may identify the same or similar elements.

Claims

1. 1. A computer-implemented method for generating a speculative chip design, comprising: generating, by one or more processors, a set of approximate systolic arrays to meet architectural parameters of the inference chip design; determining, by the one or more processors, the performance of each approximation systolic array in the set of approximation systolic arrays processing a deep neural network (DNN); mapping, by the one or more processors, each layer of the DNN to an approximate systolic array in the set of approximate systolic arrays; selecting, by the one or more processors, a subset of the set of approximate systolic arrays based on the mapping and the performance of each of the approximate systolic arrays in the set of approximate systolic arrays; updating the architecture parameters so as to improve an objective function relating to the architecture parameters; generating a second set of approximate systolic arrays to satisfy the updated architecture parameters; determining the performance of each approximation systolic array in the second set of approximation systolic arrays processing the DNN; mapping each layer of the DNN to an approximate systolic array in the second set of approximate systolic arrays; updating the subset of approximate systolic arrays based on the mapping and the performance of each approximate systolic array in the second set of approximate systolic arrays, wherein the updated subset of approximate systolic arrays includes at least one systolic array from the second set of approximate systolic arrays.

2. The method of claim 1 , wherein the set of approximate systolic arrays is generated to satisfy an architecture template, the architecture template defining a number and / or size of systolic arrays for the inference chip design.

3. A computer-implemented method for generating an inferential chip design, comprising: generating, by one or more processors, a set of approximate systolic arrays; determining, by the one or more processors, the performance of each approximation systolic array in the set of approximation systolic arrays processing a deep neural network (DNN); mapping, by the one or more processors, each layer of the DNN to an approximate systolic array in the set of approximate systolic arrays; selecting, by the one or more processors, a subset of the set of approximate systolic arrays for inclusion in the inference chip design based on the mapping and the performance of each of the approximate systolic arrays in the set of approximate systolic arrays; the set of approximate systolic arrays is generated to satisfy an architecture template, the architecture template defining the number and / or size of systolic arrays for the inference chip design; A method wherein each approximation systolic array in the set of approximation systolic arrays is generated using a predefined multiply-accumulate unit (MAC).

4. The method described in claim 3, wherein the predefined MAC is stored in a bank.

5. 3. The method of claim 2, wherein each approximate systolic array in the set of approximate systolic arrays is generated using a multiply-accumulate unit (MAC), and at least one of the MACs is generated based on a predefined criterion.

6. The method of claim 5 , wherein the predefined criteria include one or more of power usage or accuracy.

7. 2. The method of claim 1, wherein determining the performance of each of the approximate systolic arrays comprises determining a power usage of each of the approximate systolic arrays.

8. 8. The method of claim 7, wherein determining the performance of each of the approximate systolic arrays further comprises determining the accuracy of each of the approximate systolic arrays.

9. A computer-implemented method for generating an inferential chip design, comprising: generating, by one or more processors, a set of approximate systolic arrays; determining, by the one or more processors, the performance of each approximation systolic array in the set of approximation systolic arrays processing a deep neural network (DNN); mapping, by the one or more processors, each layer of the DNN to an approximate systolic array in the set of approximate systolic arrays; selecting, by the one or more processors, a subset of the set of approximate systolic arrays for inclusion in the inference chip design based on the mapping and the performance of each of the approximate systolic arrays in the set of approximate systolic arrays; determining the performance of each of the approximation systolic arrays includes determining at least one of power usage and accuracy of each of the approximation systolic arrays; Mapping each layer of the DNN to an approximate systolic array in the set of approximate systolic arrays comprises: For each layer of the DNN, configuring a router in the inference chip design to a particular approximation systolic array in the subset of approximation systolic arrays based on at least one of power usage and accuracy of each approximation systolic array.

10. The method of claim 1 , wherein the inference chip design includes at least one full-precision systolic array.

11. 1. A system for generating inferential chip designs, comprising: one or more processors; a memory storing instructions that, when executed by said one or more processors, cause said one or more processors to perform the method of any one of claims 1 to 10.

Images