Synthesis of Zero-Loss Low-Power Approximation DNN Accelerators by Large-Scale Search
Customizable systolic array-based inference chips dynamically route layers to approximation or exact units, addressing energy consumption and accuracy issues in deep learning models, achieving efficient power savings and accuracy maintenance.
Patent Information
- Application Number
- JP2023571641
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-22
- Filing Date
- 2022-10-27
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2042-10-27
AI Technical Summary
Deep learning neural network models consume significant energy due to full precision floating point calculations, and replacing them with low-bit quantization operations can reduce accuracy, while approximation units introduce errors that amplify through subsequent layers, leading to inaccurate outputs.
Designing inference chips with customizable systolic arrays that dynamically route layers to approximation or exact units based on error resilience, achieving power savings without compromising accuracy by co-designing hardware and software mapping.
The approach results in low-power, high-accuracy inference chips that maintain model accuracy without retraining, offering significant power savings and improved efficiency compared to traditional methods.
Smart Images

Figure 2024540786000001_ABST
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Patent Application No. 17 / 532,572, filed November 22, 2021, the disclosure of which is incorporated herein by reference. [Background technology]
[0002] background Scaling neural network models often improves the accuracy of the outputs provided by these neural network models. However, as these neural network models continue to grow in size, so does the amount of energy consumed by inference accelerators that implement these deep neural network models. Thus, even small efficiency improvements in deep learning neural network models can dramatically reduce the global energy consumption by inference accelerators that implement deep learning models. Summary of the Invention [Problem to be solved by the invention]
[0003] Computation units that perform basic mathematical operations in implementing deep learning models often account for a large portion of the energy consumed by inference accelerators. To improve the power efficiency of these computation units, full-precision floating-point calculations can be replaced with lower-bit precision quantized operations. However, the increased efficiency of lower-bit quantization can result in reduced accuracy of the output by deep learning models. In some cases, model weight adjustments are performed to recover the accuracy lost through approximations. However, this weight adjustment step may not be sufficient to avoid accuracy degradation.
[0004] Further efforts to reduce energy consumption by inference accelerators implementing deep learning models include using approximation units, which are more power-efficient than quantization operators. These approximation units can implement approximation operators (e.g., multipliers and adders) that tailor the approximation to the numerical distribution observed during neural network evaluation. Tailoring the approximation to a specific application during evaluation may result in a better tradeoff between power consumption and accuracy than if quantization were approximated uniformly across all inputs. However, errors introduced by the approximation units may be amplified as the deep learning model progresses through each subsequent neural network layer, leading to inaccurate outputs. [Means for solving the problem]
[0005] overview Techniques described herein are directed to generating an inference chip design for production. One aspect of the disclosure is directed to a method for generating an inference chip. The method may include generating, by one or more processors, a set of approximate systolic arrays; determining, by one or more processors, a performance of each approximate systolic array in the set of approximate systolic arrays that processes a deep neural network (DNN); mapping, by one or more processors, each layer of the DNN to an approximate systolic array in the set of approximate systolic arrays; and selecting, by the one or more processors, a subset of the set of approximate systolic arrays for inclusion in the inference chip design based on the mapping and the performance of each approximate systolic array in the set of approximate systolic arrays.
[0006] Another aspect of the present disclosure is directed to a system for generating an inference chip design for production. The system includes one or more processors and a memory that stores instructions that, when executed by the one or more processors, cause the one or more processors to perform the steps of generating a set of approximate systolic arrays, determining the performance of each approximate systolic array in the set of approximate systolic arrays that processes a deep neural network (DNN), mapping each layer of the DNN to an approximate systolic array in the set of approximate systolic arrays, and selecting a subset of the set of approximate systolic arrays for inclusion in the inference chip design based on the mapping and the performance of each approximate systolic array in the set of approximate systolic arrays.
[0007] In some examples, a second set of approximate systolic arrays may be generated, a performance of each approximate systolic array in the second set of approximate systolic arrays for processing the DNN may be determined, each layer of the DNN may be mapped to an approximate systolic array in the second set of approximate systolic arrays, and a subset of the systolic arrays may be updated based on the mapping and the performance of each approximate systolic array in the second set of approximate systolic arrays, wherein the updated subset of approximate systolic arrays includes at least one systolic array from the second set of approximate systolic arrays.
[0008] In some examples, the set of approximate systolic arrays is generated to satisfy an architecture template, the architecture template defining a number and / or size of systolic arrays for the inference chip design. In some examples, each approximate systolic array in the set of approximate systolic arrays is generated using a predefined multiply-accumulate unit (MAC), and the predefined MAC is stored in a bank.
[0009] In some examples, each approximation systolic array in the set of approximation systolic arrays is generated using a multiply-accumulate unit (MAC), and at least one of the MACs is generated based on predefined criteria, in some examples, the predefined criteria includes one or more of power usage or accuracy.
[0010] In some examples, determining the performance of each approximation systolic array includes determining a power usage of each approximation systolic array. In some examples, determining the performance of each approximation systolic array further includes determining an accuracy of each approximation systolic array.
[0011] In some examples, mapping each layer of the DNN to an approximation systolic array in the set of approximation systolic arrays includes configuring a router in the inference chip design to a particular approximation systolic array in the subset of approximation systolic arrays based on the power usage and accuracy of each approximation systolic array for each layer of the DNN.
[0012] In some examples, the inference chip design includes at least one full-precision systolic array. [Brief description of the drawings]
[0013] [Figure 1] FIG. 1 is a block diagram of an example framework for generating an optimized inference chip, according to an aspect of the present disclosure. [Diagram 2] FIG. 2 is a diagram of an optimized inference chip, according to an aspect of the present disclosure. [Diagram 3] 1 is a graph illustrating principal components of various MACs, according to an embodiment of the present disclosure. [Figure 4] 1 is a graph illustrating power consumption of a single MAC and a systolic array according to an embodiment of the present disclosure. [Diagram 5]1 is a graph illustrating the operating temperatures of an optimized MXU that includes a collection of approximate systolic arrays versus an MXU formed from an exact systolic array, in accordance with aspects of the present technique. [Figure 6] FIG. 1 is a block diagram of an exemplary computing environment for implementing a framework in accordance with aspects of the present technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] Detailed Description The techniques described herein are directed to designing approximate, low-power deep learning accelerator chips that perform deep learning models with little loss of accuracy. Moreover, the approximate, low-power deep learning accelerator chips described herein ("inference chips") do not need to retrain deep learning models to maintain accuracy. Instead, the inference chips are designed to process deep learning models with the same or nearly the same accuracy as chips composed of "accurate" units (e.g., arithmetic units capable of full-precision computation).
[0015] The inference chip may be designed using an architecture template. The template may generate a diverse set of efficient designs with reconfigurable wiring arrays to banks of systolic arrays containing approximation units such as approximate adders and / or approximate multipliers. This allows error-tolerant layers in the deep learning model to be dynamically routed to more approximation units in the inference chip accelerator at run time, resulting in significant power savings. Additionally, layers in the deep learning model that are more sensitive to errors may be evaluated with more accurate approximation units and / or accurate units to minimize or avoid the occurrence of errors. By co-designing the approximation units on the inference chip with the software mapping of layers in the deep learning model to the systolic array, power savings may be achieved without compromising accuracy.
[0016] 1 is a flow diagram illustrating a design process for creating a low-power, high-precision inference chip. As described herein, the task of selecting an approximate multiplication and approximation unit for an inference chip design occurs simultaneously with mapping a DNN layer to a systolic array of a chip that contains a collection of processing elements (PEs).
[0017] Each systolic array may include a two-dimensional array of PEs, with each PE including one or more multiply-accumulate units (MACs). Additionally, each PE may include buffers for input operands and output results. Each systolic array is globally activated, and a parameter memory is shared among all systolic arrays. Typically, each systolic array includes a single type of MAC, but some systolic arrays may include multiple types of MACs. Additionally, although MACs are described in the examples described herein, other arithmetic units may also be used (e.g., adders). By designing the hardware and simultaneously mapping DNN layers to the systolic arrays, the inference chip can be custom-tuned to a specific class of neural network.
[0018] The flow diagram illustrates an inference chip design framework 100 for generating systolic arrays of approximate and / or exact units that satisfy a given architecture template 101 and architecture parameters 115 input to an accelerator architecture generator 106, which includes a systolic array generator 105. The architecture template 101 of the framework 100 may define requirements corresponding to the layout and configuration of the inference chip. For example, the architecture template 101 may define the number of systolic arrays and the size of the systolic arrays of the inference chip. For example, the architecture template may define that the inference chip has three systolic arrays, two composed of approximate units and one composed of exact units. In other templates, other amounts and types of arrays may be defined, which approximate units (e.g., MACs) to use, etc. The architecture parameters 115 may include the configuration of heterogeneous systolic arrays, memory and system parameters, etc. In an initial process by the framework 100, the architecture parameters may be "seeded". The seeded parameters may be predefined or may be input each time the framework starts the process.
[0019] MACs that may be used to create processing elements in a systolic array may be stored in a bank 103 of the framework. The MACs may have attributes that meet certain criteria, such as error tolerance and power usage. For example, a first MAC may have attributes such as 1% error for an exact MAC and 99% power usage for an exact MAC. In another example, a second MAC may have 3% error for an exact MAC and 90% power usage for an exact MAC. The bank 103 may store any number of configured MACs, possibly including an exact MAC that performs full precision calculations (rather than approximate calculations). In some cases, the MACs may be generated by the framework. In this regard, the framework may generate MACs that fit a particular use case, such as a MAC with a particular error percentage and power usage. During the generation of a systolic array, the framework may generate MACs that fit a particular use case that is not met by the predefined MACs stored in the bank 103.
[0020] The systolic array generator 105 of the accelerator architecture generator 106 can use the MACs stored in the bank 103 to generate a set of diverse systolic arrays to meet the requirements of a given architecture template (and architecture parameters 115) while ensuring that the set of systolic arrays can accurately execute the DNN 107. In this regard, the accelerator architecture generator 106 can receive the architecture template 101 and the architecture parameters 115. The accelerator architecture generator 106 can then provide some or all of the received data to the systolic array generator. For example, the accelerator architecture generator 106 may inform the systolic array generator 105 of the parameters of the design of the systolic array, such as the size of the systolic array, the configuration (e.g., an inference chip includes x% MACx and y% MACy). The MACx and MACy can be selected from the bank 103.
[0021] The systolic array generator 105 may generate a set of systolic arrays according to the received parameters. In this regard, the framework's systolic array generator 105 may select MACs with varying precision and power attributes according to the received parameters. The systolic array generator 105 may generate code representing the systolic arrays, such as in Verilog, C++, or other such languages.
[0022] Each layer of the DNN 107 may be input to a layer-wise mapping 120 function of the framework 100. The layer-wise mapping 120 function may assign each layer of the DNN 107 to a particular systolic array of the set of systolic arrays generated by the systolic array generator 105. As described herein, the optimal approximation level for processing a DNN may vary dramatically between different layers of a deep neural network. The layer-wise mapping function 120 may map each DNN level based on its approximation tolerance.
[0023] As described herein, each generated set of systolic arrays may be tested for power and accuracy, and an optimal set of systolic arrays may be selected for an inference chip design custom-tuned to run the DNN 107. Although DNNs are described herein, the design process for creating low-power, high-accuracy inference chips may be performed for other types of neural networks. For example, each generated set of systolic arrays may be tested for accuracy using fast accuracy estimation 109 as described herein. Additionally, power and performance estimates (collectively "performance estimates") may be determined for each generated set of systolic arrays, as indicated by block 111 and described herein.
[0024] The accuracy and performance estimates may be input into an objective function 112 described herein to determine an optimized combination of systolic arrays. In this regard, the objective function 112 may evaluate how well each DNN layer mapping performs in terms of accuracy and performance.
[0025] The architecture search algorithm 113 can then take the objective function results (and other previous results of the objective function) and determine whether improvements are possible with a different set of systolic arrays. In this regard, the architecture search algorithm 113 can receive the current objective function results for the current parameters and past objective function results for previous parameters. Based on these parameters, the architecture search algorithm 113 can determine the next set of parameters to try (illustrated by the arrows from 113 to 115 in FIG. 1). The framework 100 aims to find the optimal value of the objective function through an iterative search of the parameter space that results in an optimal configuration.
[0026] This process can be repeated a number of times or until all possible combinations of systolic arrays, layer-wise mappings, etc. are completed. The most efficient and accurate set of systolic arrays is then selected. Although not shown, the framework can output a design plan for configuring the inference chip according to the selected set of systolic arrays.
[0027] An optimized combination of systolic arrays can comprise a multi-tile matrix multiplication unit (MXU). Each "tile" of a multi-tile MXU can include a systolic array with specific precision and power attributes. For example, FIG. 2 shows a multi-tile MXU 205 that includes three systolic arrays, including array A 207, array B 209, and array Z 211. Although the MXU 205 is shown to include only three arrays, the MXU can include any number of systolic arrays. As further shown in FIG. 2, the activation buffer 201 can receive results of previous layers of the DNN. These previous results are routed by the learned router 203 to a systolic array for processing the next layer of the DNN. The routing of layers may be pre-programmed such that the router directs a particular layer of the DNN to a pre-defined systolic array for processing. For example, FIG. 2 shows a current DNN layer being processed by array Z 211.
[0028] Accuracy Estimation Both training and inference of large deep neural networks can involve trillions of arithmetic operations. With exact arithmetic and standard data types, this evaluation can be parallelized on high-throughput GPUs. However, simulating inexact arithmetic operations using approximate MACs can be challenging because existing hardware does not support such operations, making direct evaluation of inexact arithmetic difficult. For example, directly evaluating approximate MACs in a circuit simulator like the Verilator circuit simulator can take 3.75 ± 0.95 microseconds on a high-performance server for a single 8-bit exact multiplication. Thus, evaluation of a large neural network like a 50-layer deep ResNet-50 can take approximately 4.2 hours at 4 GFLOPs per 224 × 224 frame. On the entire ImageNet validation set, it would take approximately 23 years to evaluate one approximate multiplier.
[0029] Furthermore, the calls may be cached, but the full 2 for an N-bit multiplier N+N The lookup table may not be scalable to wider bit widths. For example, for a 16-bit input, the lookup table would be over 16 gigabytes. Due to its large size, it is not possible to use GPU acceleration.
[0030]
number
[0031] N-bit error matrix ε∈R 2N+N To save memory in storing, a truncated singular value decomposition is used for k<<2 using the following equation (1): N It may be calculated as:
[0032]
number
[0033] The total memory consumption is O(n 2 ) to O(nk). In equation (1), Σ is the error matrix, σ i is the i-th eigenvalue of the error matrix, u i and v i are the i-th left singular vector and right singular vector of the error matrix, respectively. Parameter k<<2 N controls the amount of approximation, and smaller k increases the amount of approximation but reduces the memory usage to store the error matrix. For small k≦50, the total memory consumption is less than 20MB. During evaluation, the result of the i×j approximate multiplication may be recomputed.
[0034] All results may be evaluated using classification datasets, such as large-scale image datasets like the ImageNet2012 dataset. To improve the evaluation speed of the end-to-end model on the target dataset, one can evaluate a small sample, such as a 10% sample of the full validation set. For example, when using ImageNet2012, one can use a sample of 5000 images. The ordering of the model on the sample validation set usually correlates with its performance on the full dataset.
[0035] Overall, the optimizations described here can result in a speedup of 7200x compared to direct circuit simulation in Verilator. This strategy also makes future GPU evaluation with automatic retraining feasible. Compressing the lookup tables in the low-rank decomposition allows the model evaluation described here to be extended to circuits with larger bit widths.
[0036] Moreover, the approach described herein is complementary to other known quantization methods. For example, post-training quantization of the dynamic range may be performed where weights are statically quantized to 8 bits before inference. During inference, the activities may be scaled and then quantized to the uint8 range of [0;255], although other ranges may be used. They can then be dequantized using equation (2) below.
[0037]
number
[0038] q1 represents the weight matrix, q2 represents the activation matrix, and Z represents the zero points of each. More advanced quantization methods using quantization-aware training can achieve higher power savings.
[0039]
number
[0040] Semiconductor design and test software may be used to evaluate the performance of systolic arrays across MXUs. Testing is capable of evaluating semiconductor designs in sub-10 nm processes. For evaluation, a single clock domain is applied to each systolic array, and the clock frequency is determined by the slowest systolic array. In this regard, the clock frequency is usually based on the "exact" systolic array, while other "approximate" systolic arrays may be slower and therefore used to set the clock frequency. Furthermore, although a single clock frequency is described in this example, some or all systolic arrays in the MXU's design may be tested at different frequencies. During synthesis, the different MXU variants are constrained to the slowest clock frequency. This allows the synthesis tool to select smaller gate sizes for the approximate MXU variants with higher inherent performance, thus saving some power and area.
[0041] In practice, DNN accelerators are typically thermal design power (TDP) limited, rather than latency limited. Therefore, further gains can be obtained by using the thermal savings of the approximate systolic array to overclock the inference chip. For example, Figure 5 shows that an optimized approximate MXU runs at significantly lower temperatures compared to an MXU with only an "exact" systolic array.
[0042] Circuit exploration and layer mapping Formulation of the Approximate Circuit Mapping Problem The performance of a particular design also depends on how the nests of convolutional loops are mapped onto arrays: mapping DNN layers onto specific systolic arrays ensures that the generated accelerator matches what happens after synthesis.
[0043] To maintain high end-to-end task accuracy, it is necessary to consider which layers of a deep learning model are approximation tolerant. In this regard, the optimal approximation level can vary dramatically between different layers of a deep neural network. However, it is difficult to simultaneously consider the tasks of selecting an approximation systolic array for chip design and mapping layers of a DNN to the systolic array. In this respect, these two subproblems (selecting an approximation unit for chip design and mapping layers of a DNN to the approximation unit) are challenging combinatorial optimization problems, respectively. Taken together, we propose an O(K) MAC design candidate to map N deep neural network layers. N ) search space. O is the "big O" notation that expresses the complexity of the search space. Furthermore, a maximum of 2 268 There may be a search space of 1, or more, or fewer, workloads. In this case, the search space defines how to design an approximate MAC. The goal of the optimization algorithm is to find an optimal design within this search space. Therefore, random searching will not work.
[0044] Bayesian optimization using Gaussian process (GP) bandits can be leveraged to efficiently discover configurations of cross-layer approximation circuits that are highly accurate yet energy-efficient. This approach improves the sample efficiency of black-box optimization by modeling the unknown reward function f:x→y with a Gaussian process.
[0045] To formulate the approximate circuit mapping problem, consider the following optimization problem, which seeks a minimum-power mapping of approximate circuits onto deep neural network layers:
[0046]
number
[0047]
number
[0048] The accuracy oracle, ACC, models the effects of cross-layer interactions due to approximations. Given a particular assignment of approximation multipliers to layers, ACC calculates the expected accuracy of the model on a particular dataset. Accuracy oracles are very difficult to model because errors made at one layer compound on subsequent layers. Therefore, the accuracy of the model is calculated on a validation set.
[0049] This reduces the optimization problem to a black-box combinatorial optimization problem, and Bayesian optimization methods typically fail when applied to the above optimization problem because they exhibit slow convergence with performance similar to random search. Furthermore, Bayesian optimization struggles with high-dimensional states, discrete structures, and constrained search spaces. Therefore, the above optimization problem can be reformulated to allow the use of known Bayesian optimization tools.
[0050] Calibration of the single layer approximation To find a highly accurate approximate circuit design, feasible solutions may be constrained by the minimum validation accuracy threshold in constraint (2b). However, the accuracy oracle “ACC” is not known. Therefore, to reduce the complexity of the search space, an offline study using only approximate multipliers for the target layer may be performed. All other layers can be evaluated using exact multipliers. This model provides an upper bound on the expected accuracy from the cross-layer approximation. Mappings with exceptionally low expected accuracy may be pruned.
[0051] Continuous relaxation of state space GP bandits are primarily designed to optimize over discrete search spaces. Bayesian optimization frameworks typically support discrete search spaces by embedding them into real-valued boxes. However, this embedding is sample inefficient because it does not consider the relationships between different categorical variables. For example, this solution suffers from instability due to quantization errors caused by rounding continuous predictors to the nearest feasible point.
[0052] To address this issue, estimates of per-layer accuracy degradation from single-layer approximations can be used to compute an ordered set that represents the relative ordering of each approximate multiplier. In this regard, an ordering can be defined as the profiled end-to-end accuracy for approximating a single layer k with a particular multiplier. However, direct search in unrounded precision is an unstable relaxed optimization problem. As a result, two approximate multipliers achieve similar accuracy but consume significantly different power. Therefore, the linear ordering of multipliers may be relaxed to a partially ordered set where equates within a fixed threshold of accuracy are considered incomparable. The completed linear ordering can then be resolved by removing the least efficient multiplier from each pair of incomparable multipliers with similar precision.
[0053] This procedure derives a linear ordering of multipliers for each of the N layers of the neural network. To define the distance in the new dimension after mapping, a min-max scaling can be applied to the top-1 accuracy (i.e., high probability) of each multiplier obtained from the single-layer approximation calibration. Given this new formulation of the search space, the following cost optimization objective is defined: For each of the N layers, a stepwise cost function Q is calculated to map the choice of real values of the approximation multipliers (between 0 and 1) to the energy consumption of the closest layer, with truncation. i :R→R may be defined.
[0054] Then the relaxed optimization problem becomes:
[0055]
number
[0056] Unconstrained optimization using barrier functions Recent work has begun to explore multi-objective optimization using Bayesian optimization, but these approaches are generally significantly less sample-efficient than single-objective optimizers. To optimize inference chip designs, one can consider the two-dimensional Pareto frontier between accuracy and energy consumption. In practice, it may be useful to also limit the area of the final systolic array to avoid degenerate solutions in which redundant approximate multipliers with similar accuracy are instantiated on a single chip.
[0057] The barrier method can be used to remove constraints (3b) and (3c). The barrier method treats each constraint of the form x ≤ b as a function of the objective function β(x,b) = -log(bx) or β(x;b) = e x-b As x approaches the constraint b, the penalty goes to ∞. Using the barrier method, the objective can be expressed as
[0058]
number
[0059] This updated objective allows for the removal of constraints (3b) and (3c). An exponential barrier function can be leveraged to allow for soft constraint violations. In precision terms, the target precision τ can be 0.68 or higher or lower, and the weight α2 can be 8 or higher or lower. In area terms, the target area fraction (including the exact scale factor) can be φ=400% or higher or lower, and the scale α2 can be 1:2 or higher or lower.
[0060] Example computing environment FIG. 6 is a block diagram of an exemplary computing environment 600 implementing the exemplary framework as shown in FIG. 1. For example, the framework may be implemented in one or more devices having one or more processors at one or more locations, such as a server computing device 615. The user computing device 612 and the server computing device 615 may be communicatively coupled to one or more storage devices 630 via a network 660. The storage device 630 may be a combination of volatile and non-volatile memory and may be in the same physical location as the computing devices 612, 615 or in a different physical location. For example, the storage device 630 may include any type of non-transitory computer-readable medium that may store information, such as a hard drive, a solid-state drive, a tape drive, optical storage, a memory card, ROM, RAM, DVD, CD-ROM, writeable memory, and read-only memory.
[0061] The server computing device 615 may include one or more processors 613 and memory 614. The memory 614 may store information accessible by the processor 613, including instructions 621 that may be executed by the processor 613. The memory 614 may also include data 623 that may be retrieved, manipulated, or stored by the processor 613. The memory 614 may be a type of non-transitory computer-readable medium that may store information accessible by the processor 613, such as volatile and non-volatile memory. The processor 613 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs).
[0062] The instructions 621 may include one or more instructions that, when executed by the processor 613, cause one or more processors to perform operations defined by the instructions. The instructions 621 may be stored in object code format for direct processing by the processor 613, or in other formats, including interpretable scripts, or a collection of independent source code modules that are interpreted on demand or pre-compiled. The instructions 621 may include instructions for implementing the framework 100 consistent with aspects of the present disclosure. The framework 100 may be executed using the processor 613 and / or using other processors located remotely from the server computing device 615.
[0063] Data 623 may be retrieved, stored, or modified by processor 613 according to instructions 621, such as a bank of approximate MACs 103, architecture templates 101, architecture parameters 115, etc. Data 623 may be stored in a computer register, a relational or non-relational database, as a table with multiple different fields and records, or as a JSON, YAML, proto, or XML document. Data 623 may also be formatted in a computer readable format, such as, but not limited to, binary values, ASCII, or Unicode. Additionally, data 623 may include sufficient information to identify related information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memory including other network locations, or information used by a function to calculate the related data.
[0064] The user computing device 612 may also be configured similarly to the server computing device 615, with one or more processors 616, memory 617, instructions 618, and data 619. The user computing device 612 may also include a user output 626, and a user input 624. The user input 624 may include any suitable mechanism or technology for receiving input from a user, such as a keyboard, a mouse, a mechanical actuator, a soft actuator, a touch screen, a microphone, and a sensor.
[0065] The server computing device 615 may be configured to transmit data to the user computing device 612, which may be configured to display at least a portion of the received data on a display implemented as part of the user output 626. The user output 626 may also be used to display an interface between the user computing device 612 and the server computing device 615. The user output 626 may alternatively or additionally include one or more speakers, transducers, or other audio outputs, haptic interfaces, or other haptic feedback that provide non-visual and non-audible information to a platform user of the user computing device 612.
[0066] Although FIG. 6 illustrates the processors 613, 616 and memories 614, 617 as being within the computing devices 615, 612, the components described herein, including the processors 613, 616 and memories 614, 617, may include multiple processors and memories that may operate in different physical locations rather than within the same computing device. For example, some of the instructions 621, 618 and data 623, 619 may be stored on a removable SD card, while others may be stored in a read-only computer chip. Some or all of the instructions and data may be stored in a location physically remote from the processors 613, 616, but still accessible by the processors 613, 616. Similarly, the processors 613, 616 may include a collection of processors that may perform simultaneous and / or sequential operations. The computing devices 615, 612 may each include one or more internal clocks that provide timing information, which may be used to time operations and programs executed by the computing devices 615, 612.
[0067] The server computing device 615 may be configured to receive requests to process data from the user computing device 612. For example, the environment 600 may be part of a computing platform configured to provide various services to users through various user interfaces and / or APIs that expose platform services. One or more services may be a machine learning framework or set of tools for generating a neural network or other machine learning model according to a specified task and training data. The user computing device 612 may receive and transmit data specifying target computing resources to be allocated to run a neural network trained to perform a particular neural network task.
[0068] The devices 612, 615 may be capable of direct and indirect communication over the network 660. The devices 615, 612 may set up listening sockets capable of accepting initiating connections to send and receive information. The network 660 itself may include a variety of configurations and protocols, including the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local networks, and private networks using one or more company-proprietary communication protocols. The network 660 may support a variety of short-range and long-range connections. The short-range and long-range connections may be made at different bandwidths, such as 2.402 GHz to 2.480 GHz (commonly associated with the Bluetooth® standard), 2.4 GHz and 5 GHz (commonly associated with the Wi-Fi® communication protocol), or various communication standards, such as the LTE® standard for wireless broadband communication. The network 660 may additionally or alternatively support wired connections between the devices 612, 615, including various types of Ethernet® connections.
[0069] 6, it is understood that aspects of the disclosure may be implemented according to a variety of different configurations and quantities of computing devices, including in paradigms for serial or parallel processing, or in a distributed network of multiple devices. In some implementations, aspects of the disclosure may be performed on a single device, and any combination thereof.
[0070] Aspects of the present disclosure may be implemented in a digital circuit, a computer-readable storage medium, as one or more computer programs, or a combination of one or more of the foregoing. The computer-readable storage medium may be non-transitory, for example executable by a cloud computing platform, as one or more instructions stored on a tangible storage device.
[0071] The phrase "configured" is used herein in various contexts relating to a computer system, hardware, or part of a computer program, engine, or module. When a system is said to be configured to perform one or more operations, this means that the system has appropriate software, firmware, and / or hardware installed on the system that, when operated, causes the system to perform one or more operations. When some hardware is said to be configured to perform one or more operations, this means that the hardware includes one or more circuits that, when operated, receive inputs and generate outputs corresponding to the one or more operations according to the inputs. When a computer program, engine, or module is said to be configured to perform one or more operations, this means that the computer program includes one or more program instructions that, when executed by one or more computers, cause the one or more computers to perform one or more operations.
[0072] Although the operations illustrated in the figures and recited in the claims are shown in a particular order, it is understood that the operations may be performed in an order different from that shown, and that some operations may be omitted, performed multiple times, and / or performed in parallel with other operations. Furthermore, the separation of different system components configured to perform different operations should not be understood as requiring the components to be separated. The components, modules, programs, and engines described may be integrated together as a single system or may be part of multiple systems. One or more processors in one or more locations implementing an exemplary STOC according to aspects of the present disclosure may perform the operations illustrated in the figures and recited in the claims.
[0073] Unless otherwise specified, the above alternatives are not mutually exclusive and can be implemented in various combinations to achieve unique advantages. These and other variations and combinations of the above features can be utilized without departing from the subject matter defined by the claims, and therefore the description of the above examples should be construed as illustrative, not limiting, of the subject matter defined by the claims. In addition, the provision of examples described herein, as well as phrases such as "such as," "including," and the like, should not be construed as limiting the subject matter of the claims to any particular example, but rather, the example is intended to illustrate only one of many possible implementations. Furthermore, the same reference numbers in different drawings may identify the same or similar elements.
Claims
1. 1. A computer-implemented method for generating a speculative chip design, comprising: generating, by one or more processors, a set of approximate systolic arrays; determining, by the one or more processors, a performance of each approximation systolic array in the set of approximation systolic arrays for processing a deep neural network (DNN); mapping, by the one or more processors, each layer of the DNN to an approximate systolic array in the set of approximate systolic arrays; selecting, by the one or more processors, a subset of the set of approximate systolic arrays for inclusion in the inference chip design based on the mapping and performance of each approximate systolic array in the set of approximate systolic arrays.
2. generating a second set of approximation systolic arrays; determining the performance of each approximation systolic array in the second set of approximation systolic arrays processing the DNN; mapping each layer of the DNN to an approximation systolic array in the second set of approximation systolic arrays; 2. The method of claim 1 , further comprising: updating the subset of approximate systolic arrays based on the mapping and a performance of each of the approximate systolic arrays in the second set of approximate systolic arrays, wherein the updated subset of approximate systolic arrays includes at least one systolic array from the second set of approximate systolic arrays.
3. The method of claim 1 , wherein the set of approximate systolic arrays is generated to satisfy an architecture template, the architecture template defining a number and / or size of systolic arrays for the inference chip design.
4. 4. The method of claim 3, wherein each approximation systolic array in the set of approximation systolic arrays is generated using a predefined multiply-accumulate unit (MAC), the predefined MACs being stored in a bank.
5. 4. The method of claim 3, wherein each approximation systolic array in the set of approximation systolic arrays is generated using a multiply-accumulate unit (MAC), at least one of the MACs being generated based on a predefined criterion.
6. The method of claim 5 , wherein the predefined criteria includes one or more of power usage or accuracy.
7. 2. The method of claim 1, wherein determining the performance of each of the approximate systolic arrays comprises determining a power usage of each of the approximate systolic arrays.
8. 8. The method of claim 7, wherein determining the performance of each of the approximate systolic arrays further comprises determining the accuracy of each of the approximate systolic arrays.
9. Mapping each layer of the DNN to an approximation systolic array in the set of approximation systolic arrays comprises:
9. The method of claim 8, comprising: for each layer of the DNN, configuring routers in the inference chip design to particular approximation systolic arrays in the subset of approximation systolic arrays based on power usage and accuracy of each of the approximation systolic arrays.
10. The method of claim 1 , wherein the inference chip design includes at least one full precision systolic array.
11. 1. A system for generating an inferential chip design, comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to: generating a set of approximating systolic arrays; determining the performance of each approximate systolic array in the set of approximate systolic arrays for processing a deep neural network (DNN); mapping each layer of the DNN to an approximation systolic array in the set of approximation systolic arrays; selecting a subset of the set of approximate systolic arrays for inclusion in the inference chip design based on the mapping and the performance of each approximate systolic array in the set of approximate systolic arrays.
12. The instructions further cause the one or more processors to: generating a second set of approximation systolic arrays; determining the performance of each approximation systolic array in the second set of approximation systolic arrays processing the DNN; mapping each layer of the DNN to an approximation systolic array in the second set of approximation systolic arrays; and updating the subset of approximate systolic arrays based on the mapping and a performance of each of the approximate systolic arrays in the second set of approximate systolic arrays, wherein the updated subset of approximate systolic arrays includes at least one systolic array from the second set of approximate systolic arrays.
13. 12. The system of claim 11, wherein the set of approximate systolic arrays is generated to satisfy an architecture template, the architecture template defining a number and / or size of systolic arrays for the inference chip design.
14. 14. The system of claim 13, wherein each approximation systolic array in the set of approximation systolic arrays is generated using a predefined multiply-accumulate unit (MAC), the predefined MACs being stored in a bank.
15. 14. The system of claim 13, wherein each approximation systolic array in the set of approximation systolic arrays is generated using a multiply-accumulate unit (MAC), at least one of the MACs being generated based on a predefined criterion.
16. The system of claim 15 , wherein the predefined criteria includes one or more of power usage or accuracy.
17. 12. The system of claim 11, wherein determining the performance of each of the approximate systolic arrays comprises determining a power usage of each of the approximate systolic arrays.
18. 20. The system of claim 17, wherein determining the performance of each of the approximate systolic arrays further comprises determining the accuracy of each of the approximate systolic arrays.
19. Mapping each layer of the DNN to an approximation systolic array in the set of approximation systolic arrays comprises:
20. The system of claim 18, further comprising: for each layer of the DNN, configuring a router in the inference chip design to a particular approximation systolic array in the subset of approximation systolic arrays based on the power usage and accuracy of each approximation systolic array.
20. The system of claim 11 , wherein the inference chip design includes at least one full precision systolic array.
Citation Information
Patent Citations
Low-power hardware acceleration method and system for convolution neural network computation
US20190164043A1
