Reducing power consumption of a neural network accelerator by weight reordering

Abstract

To directly address the switching problem, an improved compiler can be implemented to prepare a DNN for hardware execution on a DNN accelerator in a manner that considers switching activity during multiply-accumulate (MAC) operations and reorders weights to reduce switching activity. The resulting compiled DNN can reduce energy consumption by reducing or minimizing switching activity in the DNN and can effectively reduce dynamic power consumption in the DNN accelerator. The compiler can determine and implement the improved weight ordering during a model compilation process. The compiler can be guided by one or more weight permutation and reordering rules to ensure that the weights are ordered to minimize or reduce switching activity as much as possible without changing the output accuracy of the model or violating the strict data path of the MAC array. When deployed on a DNN accelerator, the compiled model with weight reordering will exhibit significantly lower dynamic power consumption.

Description

[0001] Cross-references to (one or more) related applications This application claims priority to and / or benefits from U.S. Provisional Application No. 63 / 738,314, filed December 23, 2024, entitled “Reducing Power Consumption of Neural Network Accelerators by Weight Reordering”. The entire contents of the U.S. Provisional Application are incorporated herein by reference. Background Technology

[0002] Deep neural networks (DNNs) are widely used in a variety of artificial intelligence (AI) and machine learning (ML) applications, ranging from computer vision to speech recognition and natural language processing, due to their ability to achieve high accuracy. However, this high accuracy comes at the cost of enormous computational costs. DNNs have extremely high computational requirements because there may be a large number of operations and a large amount of data to read and write. Attached Figure Description

[0003] The embodiments will be readily understood from the following detailed description taken in conjunction with the accompanying drawings. For ease of description, the same reference numerals denote the same structural elements. In the accompanying figures, the embodiments are shown by way of example rather than limitation.

[0004] Figure 1 The switching activities in a processing unit according to some embodiments of the present disclosure and their impact on switching power consumption are illustrated.

[0005] Figure 2 The following illustrates a switching activity without weight reordering according to some embodiments of the present disclosure.

[0006] Figure 3 Theoretical minimum switching activities with optimal weight reordering are illustrated according to some embodiments of this disclosure.

[0007] Figure 4 The present disclosure illustrates how weighted reordering can be used to reduce switching activity by applying a centrally based greedy heuristic sorting algorithm according to some embodiments of the present disclosure.

[0008] Figure 5 A centrally based greedy heuristic weight reordering algorithm according to some embodiments of the present disclosure is shown.

[0009] Figure 6 A model compilation process with weight reordering to reduce switching power consumption is illustrated according to some embodiments of this disclosure.

[0010] Figure 7 The correct weight loading order over time is shown according to some embodiments of this disclosure.

[0011] Figure 8The following are examples of incorrect weight loading order over time according to some embodiments of the present disclosure.

[0012] Figure 9 Some embodiments according to this disclosure are shown, in which the weight order is maintained in each round, but the weight order is changed in successive rounds.

[0013] Figure 10 Sample layer computing and hardware models according to some embodiments of this disclosure are shown.

[0014] Figure 11 The diagram illustrates the unchanged computations performed in layers 1 and 2 according to some embodiments of the present disclosure.

[0015] Figure 12 The present disclosure illustrates some embodiments in which the input channel order and output channel order of layer 1 are reordered, while the input channel order and output channel order of layer 2 remain unchanged.

[0016] Figure 13 The following are examples of reordering the input channel order of layer 1 and reordering the input channel order of layer 2 according to some embodiments of the present disclosure.

[0017] Figure 14 An optimal layer selection algorithm according to some embodiments of the present disclosure is described.

[0018] Figure 15 A hierarchical weight reordering process according to some embodiments of the present disclosure is described.

[0019] Figure 16 A DNN system according to some embodiments of the present disclosure is shown.

[0020] Figure 17 A processing engine according to some embodiments of this disclosure is shown.

[0021] Figure 18 A sparse processing unit according to some embodiments of the present disclosure is shown.

[0022] Figure 19 Sparse computation in a sparse processing unit according to some embodiments of the present disclosure is illustrated.

[0023] Figure 20 A DNN module according to some embodiments of the present disclosure is shown.

[0024] Figure 21 A compiler according to some embodiments of this disclosure is shown.

[0025] Figure 22A flowchart illustrating some embodiments of the present disclosure is depicted, showing methods that can be executed by a compiler.

[0026] Figure 23 This is a block diagram of an exemplary computing device according to some embodiments of the present disclosure. Detailed Implementation

[0027] Overview Deep Neural Networks (DNNs) are widely used in computer vision, speech recognition, image and video processing, primarily because they can achieve accuracy exceeding human levels. A DNN typically consists of a series of layers. Each DNN layer can include one or more deep learning operations (also known as "neural network operations"), such as convolution, matrix multiplication, layer normalization, batch normalization, SoftMax operation, pooling, element-wise operations, linear operations, non-linear operations, activation functions, and so on.

[0028] DNN models can be executed by neural network accelerators or neural network hardware accelerators, for example, for training or inference, which implement circuitry designed to accelerate the execution of neural network operations. Neural network accelerators may be referred to as neural processing units (NPUs), DNN accelerators, AI accelerators, etc. In some architectures, a DNN accelerator may be or include one or more data processing units (DPUs). A DPU may also be referred to as a computation block or computation slice. A DPU may include a processing engine (PE) capable of performing data-parallel neural network operations. A PE may include one or more multiplication-accumulation (MAC) arrays. A MAC array may include an array of MAC processing elements. A DPU may also include a post-processing engine (PPE). MAC arrays are typically used to perform parallel multiplication of input activations and weights for matrix multiplication and convolution operations in DNNs.

[0029] As DNNs continue to grow in complexity and application scope, they have become useful across a wide range of industries, including autonomous vehicles, medical diagnostics, advanced robotics, and more. These models typically utilize significant computational resources, particularly for the MAC operations that govern neural network inference and training. The real-time or near-real-time processing requirements of power-constrained devices such as smartphones, drones, and Internet of Things (IoT) sensors can exacerbate this challenge.

[0030] In these environments, optimizing performance-to-power ratio is crucial, but a significant inefficiency remains unresolved: dynamic power consumption driven by switching activity in the MAC unit. Weight switching activity can account for a large portion of the total power consumed by the NPU, representing a substantial part of the power budget. This dynamic power is typically generated by bit transitions in weights and activations during MAC operations performed by the MAC array, each transition incurring a switch in the underlying digital logic and resulting in power costs. For large-scale accelerators executing thousands of operations in parallel, this cumulative switching activity can lead to dynamic power consumption, which accounts for a significant portion of the total power consumption. This inefficiency could be a critical obstacle to deploying DNNs in edge devices, where reducing power consumption is as important as maintaining computational efficiency and accuracy.

[0031] Despite advancements in pruning, quantization, and custom hardware techniques such as clock gating and power gating, these solutions do not directly address switching activity. Pruning reduces the number of MAC operations, and quantization reduces bit width, but neither optimizes transitions between consecutive weights. Therefore, even networks with aggressive pruning or quantization exhibit high switching-aware power consumption. Hardware techniques reduce power by disabling unused MAC units but do not handle active switching events, at best providing incremental improvements.

[0032] To directly address the switching problem, an improved compiler can be implemented to account for switching activity and reorder weights in a way that reduces switching activity, preparing the DNN for hardware execution on a DNN accelerator. The resulting compiled DNN reduces energy consumption by reducing or minimizing switching activity in the DNN and can effectively reduce dynamic power consumption in the DNN accelerator. The compiler can determine and implement improved weight ordering during the model compilation process. The compiler can be guided by one or more weight ordering and reordering rules to ensure that weights are ordered to minimize or reduce switching activity as much as possible without altering the model's output accuracy or violating the strict data path of the MAC array. The weight ordering optimized model can have weights systematically ordered according to these rules. When deployed on a DNN accelerator, the compiled model with weight reordering will exhibit significantly lower dynamic power consumption. Power savings can result from reduced switching activity in the MAC unit, which can be monitored using runtime power analysis tools.

[0033] The compiler receives the definition of a neural network model, which can include multiple layers of neural network operations. Layers can have parameters, also called weights, which the neural network model uses to make predictions. These weights can be organized into weight tensors, and these weight tensors can be applied to the input activations using the MAC array of a DNN accelerator to produce the output activations of that layer. The compiler analyzes the neural network model and the weights applied to the input activations in the layers. Specifically, the compiler determines how to feed the weights to the MAC array of the MAC processing element of the DNN accelerator. The compiler has access to a switching activity metric, which measures how frequently the binary bits in these weights change as they are fed to the MAC array. Frequent bit changes, called "switching activity," increase power consumption.

[0034] To conserve energy, the compiler determines and arranges these weights to reduce switching activity metrics, such as the measurement of the number of bit changes during quantization. This weight reordering reduces switching activity and lowers the amount of dynamic power consumed. The compiler generates machine-readable instructions that configure the DNN accelerator hardware to use these newly ordered weights or to feed weights according to the determined order. The compiler can generate machine-readable instructions that configure the DNN accelerator hardware to load the weights onto the MAC array according to the determined order and apply the weights to the input activations in the MAC array. Reordering the weights in the compiler means that the DNN accelerator hardware can perform MAC operations to apply the weights as accurately as before, but with significantly improved energy efficiency. This compiler solution reduces unnecessary power usage by reducing the weight reordering of switching activity, allowing the DNN accelerator to operate more efficiently without altering the underlying hardware or compromising the accuracy of the neural network model. This approach to dynamic power consumption is particularly valuable for devices with limited battery or power resources, such as smartphones and sensors, allowing them to run complex neural network models faster and more sustainably.

[0035] One way to quantify switching activity metrics is by measuring the switching activity between a pair of weights, where one weight in the pair can be fed as input to a processing element, and the other weight in the pair can be fed as input to the processing element at a subsequent time. In other words, the input switches from one weight in the pair to the other weight in the pair. A metric for the switching activity of a pair of weights can be the Hamming distance. Specifically, the Hamming distance is defined as the number of distinct positions of corresponding symbols (bits in the case of binary data). In other words, the Hamming distance quantifies the number of bit flips required to transform one binary string into another.

[0036] In some embodiments, the compiler may employ weight reordering techniques that strategically reduce or minimize switching activity metrics (e.g., Hamming distance between successive weight transitions). By reordering weights based on switching activity metrics, the compiled DNN model can result in a significant reduction in dynamic power consumption.

[0037] In some embodiments, a streamlined and more efficient reordering algorithm can be implemented to achieve reordering more efficiently without extremely high computational complexity. This algorithm can be referred to as a hub-based greedy heuristic sorting algorithm. The algorithm iteratively selects the current hub and updates the order based on the switching activity to the current hub. During the algorithm's iterations, the compiler can select one of several weights as the current hub. The compiler can then select the next weight with the minimum switching activity metric to the current hub from the order of the multiple weights. The compiler can then update the current hub to the next weight.

[0038] The MAC array of a DNN accelerator can have a template or a specific hardware arrangement or structure. The MAC array can have P MAC processing elements. For example, P=16 means there can be 16 parallel weighted processing paths per round (P can represent the number of MAC processing elements per row). Each MAC processing element in the MAC array can have M MAC units and perform M parallel MAC operations per cycle. In each round, the MAC array can process R input channels (ICs) or C values ​​(R can represent the C value per round). For example, R=16 means there can be 16 C values ​​distributed across 16 MAC processing elements in each round. A C-block or input channel block refers to a set of ICs processed together in a round, reflecting the structured C dimension of the weights. A K-block or output channel block refers to a set of output channels (OCs), representing a segment of filters processed together along the K dimension.

[0039] The hardware layout of a MAC array can specify how MAC operations are performed, how ICs and weights are loaded onto the MAC array (e.g., per round), and how OCs are generated. In particular, MAC arrays are designed around highly structured data flows that prioritize throughput and synchronization. For example, due to the hardware layout, MAC arrays follow strict data flows where weights and activations flow into MAC processing elements via fixed data paths that depend on predictable loading patterns. These data paths, implemented using adder trees and accumulators in the MAC processing elements, follow a strict and uniform order of input weights along the C dimension (channel dimension). The data paths are considered when weights are reordered within a layer. One constraint is a fixed intra-round order of C indices across the MAC processing elements for a given output channel K. During each round of computation, weights from a fixed set of C values ​​are distributed across the MAC processing elements in a predetermined order. Another constraint is the consistency of order between K and rounds. Once a specific C ordering is established in a round, the same sequence is repeated for K processed in that round and for future rounds of computation. This ensures simplified control and consistent timing across MAC processing elements, and eliminates the possibility of dynamically adjusting the order based on in-layer switching costs.

[0040] Furthermore, the interlayer dependency between the output channels (K) of one layer and the input channels C of the next layer introduces structural constraints that affect where and how weight reordering can be applied. When reordering weights across layers, constraints are considered that specify how weights, activations, and output channels are interconnected between layers. In a convolutional layer, activations (ACT) are multiplied by weights (WFT) to produce the output channel (K). The output channel K is then used as the input channel (C) of the next layer. Therefore, the reordering of K in layer i is performed as the order of C in layer i+1 to maintain functional correctness.

[0041] The circuitry in the MAC array of a DNN hardware accelerator and the mathematical operations performed by the MAC array impose constraints on how weights can be ordered. In some cases, it is impossible to reorder the weights of individual layers in a DNN model. To optimally identify layers for weight reordering, layer selection algorithms can be implemented to determine one or more layers to apply a weight reordering mechanism. Layer selection algorithms can select discontinuous layers that can generate the greatest overall switching reduction potential. In some embodiments, the compiler can select a subset of layers from a plurality of layers based on one or more of the switching activity score of each layer and the number of weights in each layer. In some embodiments, the compiler can select a subset of layers from a plurality of layers based on the switching cost of each layer under the constraint of selecting only discontinuous layers. In some embodiments, the compiler can select a subset of layers from a plurality of layers by iteratively comparing the cumulative switching cost when the current layer is skipped with an additional cumulative switching cost when the current layer is selected. The compiler can efficiently achieve the optimal amount of power reduction within accelerator-specific hardware constraints. By analyzing switching activity and weight distribution across layers, the compiler can select a subset of layers that maximizes efficiency while adhering to hardware constraints. This approach can improve the energy efficiency and performance gains of DNN accelerators without modifying the hardware, allowing them to scale and adapt to different AI workloads.

[0042] Once a layer of the DNN model is selected for weight reordering, a two-level sorting mechanism is implemented to reduce intra-layer switching in a way that respects hardware-imposed constraints. Specifically, level 0 sorting reduces switching within K blocks (intra-K block switching), and level 1 sorting reduces switching between K blocks (inter-K block switching). A K block has R rows corresponding to different input channels (C) of the layer.

[0043] For level 0 sorting, the compiler selects the central K-block exhibiting the highest switching activity. Starting with the first input channel block (first C-block) with R input channels, the compiler determines an ordering of multiple weights that reduces switching activity between the R rows of weights corresponding to the multiple input channels of that layer. The compiler determines a C-ordering of the R row weights of the central K-block that minimizes or reduces the switching activity metric. The C-ordering reduces intra-K-block switching caused by bit transitions between input channels (C) during MAC operations. Since the weights are constructed in the MxP group, the C values ​​are processed in the locking step, so the C-ordering is maintained throughout the group and implemented in the K-blocks within the first C-block. This process can be repeated individually for the other C-blocks of the layer to determine a unique C-ordering for each C-block that can result in the maximum reduction in switching.

[0044] For level 1 ordering, the compiler selects the central C block exhibiting the highest switching activity. The compiler determines a K-ordering that minimizes or reduces switching activity between K blocks, such as the last row of a K block and the first row of another K block. In other words, the compiler determines a K-ordering that minimizes or reduces switching activity between input channels during consecutive block evaluations. The compiler also determines an ordering of multiple weights that reduces switching activity between the last row of a multiple weight's row (the last row of a K block's weight row) and the first row of another multiple weight's row in the layer (the first / starting row of another K block's weight row). The rows of K block weights correspond to multiple input channels in the layer. The other rows of K block weights correspond to multiple other input channels in the layer. The K-ordering is uniformly implemented across the C blocks of the layer.

[0045] In some embodiments, the reordered weights are quantized. When weight reordering is applied after low-bit precision quantization, dynamic power consumption can be further reduced by leveraging the benefits of lower entropy and weight dynamic range compression.

[0046] In some embodiments, the weight reordering method can significantly reduce dynamic power consumption, as measured in some prototype NPU implementations. This reduction directly addresses total NPU power consumption previously dominated by weight switching activities, marking a transformative shift in energy-efficient hardware design. Furthermore, this reduction is achieved without architectural modifications, ensuring compatibility with the existing hardware ecosystem and facilitating seamless integration. Unlike other methods, this weight reordering method maintains model accuracy while improving energy efficiency, resulting in a significant increase in trillions of operations per second per watt (TOPS / W). The weight reordering method can unlock next-generation energy-efficient AI hardware for power-constrained applications such as smartphones, IoT sensors, and autonomous systems. By focusing on switching activities—a currently overlooked dynamic power contribution—the weight reordering method delivers tangible, quantifiable benefits, addressing a key gap in the current state of DNN accelerator optimization.

[0047] The weight reordering mechanism minimizes dynamic power consumption in DNN accelerators by strategically reducing the Hamming distance between consecutive weight transitions. By optimizing the weight arrangement to reduce bit-switching activity during MAC operations, the algorithm directly enhances TOPS / W performance metrics, achieving unprecedented accelerator-level energy efficiency. The weight reordering method fundamentally addresses switching inefficiencies, making it an ideal solution for power-constrained environments such as mobile and edge devices.

[0048] One of its core advantages is that this method can significantly reduce power consumption through an algorithmic solution within the compiler, eliminating the need for costly architectural changes or dedicated hardware. Unlike other methods that rely on hardware-level optimizations such as clock gating or power gating, the weight reordering method operates at the compiler level and can be seamlessly integrated into DNN accelerators without time-consuming hardware redesign.

[0049] While the weight reordering algorithm is architecture-agnostic and compatible with a variety of DNN accelerators, its benefits can be particularly amplified in accelerators that utilize a stable output data stream. This widely adopted data stream can enhance the algorithm's weight reordering capabilities, further reduce switching activity, and maximize efficiency for power-constrained AI applications.

[0050] While many power optimization techniques can lead to performance degradation or accuracy loss, weight reordering methods can maintain the accuracy of DNN models by ensuring that weight reordering does not compromise the integrity and accuracy of the underlying computation. Heuristic reordering algorithms can preferably reorder weights in a power-saving manner without affecting the network's ability to perform high-precision tasks. This allows developers to achieve a balance between power efficiency and model accuracy that is difficult to achieve with other solutions.

[0051] Weight reordering methods can achieve a significant reduction in switching activity and directly translate this reduction into dynamic power savings across various DNN models, such as models quantized to 8-bit integer (INT8) precision. In the MobileNetV3-large example, the weight reordering method can reduce switching activity by up to half in intermediate and deeper layers, where fewer input channels allow for greater reordering flexibility, resulting in a significant reduction in dynamic power consumption at the accelerator level. In the ResNet50 example, the reduction in switching activity occurs primarily in later layers, achieving significant power savings at the accelerator level. In the Inception-V3 and GoogleNet examples, the method achieves consistent switching activity reduction across layers and dynamic power savings throughout the model. These improvements can be achieved with zero reduction in application-level model accuracy and without introducing additional hardware overhead in the accelerator architecture.

[0052] Embodiments of this disclosure can improve at least some of the aforementioned challenges and problems by providing weight reordering for a subset of layers in a DNN model. Weight reordering addresses the root cause of dynamic power inefficiency in MAC operations by directly minimizing bit transitions through strategic weight reordering techniques. In various embodiments, weight reordering techniques minimize dynamic power consumption in DNN accelerators by reducing switching activity. Compiler-based optimizations can significantly reduce bit transitions in MAC units by strategically rearranging weights to minimize switching activity metrics between successive operations, such as Hamming distance. Unlike other techniques, the method in this disclosure optimizes power consumption without requiring any hardware modifications or sacrificing model accuracy, making it both scalable and efficient. This approach achieves significant energy savings without modifying hardware or altering model accuracy, making it both energy-efficient and scalable across different hardware platforms.

[0053] TOPS / W serves as a key energy efficiency metric for both client and edge computing platforms. Compiler-level weight reordering algorithms significantly improve the power efficiency of DNN accelerators by reducing switching power at the processing unit (e.g., sparse processing unit) and data processing unit levels, thereby achieving high performance and reduced area and energy costs. This improvement directly supports efficient edge inference for a wide range of DNN applications, including imaging, video, and speech processing. Implementing weight reordering allows DNN models to be deployed on high-performance NPUs with a smaller silicon footprint and significantly reduced power consumption.

[0054] tensor The input or output data for deep learning operations can be arranged in a data structure called a tensor. A tensor is a data structure with multiple elements in one or more dimensions. Examples of tensors include vectors (which are one-dimensional (1D) tensors), matrices (which are two-dimensional (2D) tensors), three-dimensional (3D) tensors, four-dimensional (4D) tensors, and even tensors with higher dimensions. The dimensions of a tensor can correspond to axes, such as axes in a coordinate system. Dimension can be measured by the number of data points along an axis. The dimension of a tensor can define the shape of the tensor. A DNN layer can receive one or more input tensors and compute an output tensor based on one or more input tensors. Taking a convolutional layer as an example, the input tensors consist of an activation tensor (also called an "input feature map (IFM)" or "input activation tensor") containing one or more activations (also called "input elements") and a weight tensor. The weight tensor can be a kernel (2D weight tensor), a filter (3D weight tensor), or a set of filters (4D weight tensor). Convolution can be performed on the input activation tensors and weight tensors to compute the output activation tensor in the convolutional layer.

[0055] Tensors in DNNs can be stored in X-major (e.g., XYZ or XZY format), Y-major (e.g., YXZ or YZX format), or Z-major (e.g., ZXY or ZYX format). The format of a tensor defines the order in which data points in the tensor are stored, written, or retrieved. The first character indicates the contiguous dimension of the data points in memory. The second character indicates the dimension in which a data point can be accessed after all the contiguous data points in memory have been accessed. The third character indicates the dimension in which a data point is accessed after all the data points in the dimension represented by the second character have been exhausted. For example, in ZXY format, the access order starts from the Z dimension, then moves to the X dimension, and finally to the Y dimension. Data points in a tensor are contiguous in the Z dimension in memory, meaning that data points with the same (x, y) coordinates are contiguous in memory. Using tensor permutation, tensors can be read from memory in different formats.

[0056] Reason for focusing on dynamic power consumption: Switching between activities The deployment of DNNs in applications such as machine learning, autonomous systems, and natural language processing has driven significant advancements in DNN architecture. However, as the complexity of DNN architectures increases, their computational demands also rise dramatically, posing challenges, especially in power-constrained environments such as edge devices.

[0057] One problem in DNN accelerators is the high dynamic power consumption, primarily caused by switching activity during MAC operations. Switching activity involves transitions between binary states within the hardware circuitry during computation, and the frequent bit switching directly leads to power loss. This problem is potentially more pronounced in output-fixed architectures. In such architectures, weights are repeatedly accessed and fed into the MAC unit, further amplifying the switching activity. The frequent acquisition and application of weights can result in a higher number of bit transitions, exacerbating the dynamic power consumption problem. Each new weight introduces a new binary transition into the circuitry, leading to increased power consumption per computation cycle.

[0058] While various techniques such as pruning, quantization, and clock gating have been proposed to mitigate power consumption in DNNs, they fail to address the root cause of power inefficiency: the switching activity at weight levels. These methods either reduce the amount of active computation or introduce additional hardware complexity, but they cannot minimize the random and frequent bit switching during MAC operations. In cases where weights require frequent reloading, such as in output-fixed designs, these solutions do not achieve significant power savings.

[0059] Figure 1 The following illustrations depict switching activities in a processing unit and their impact on switching power consumption, according to some embodiments of the present disclosure. Specifically, Figure 1An exemplary data path is shown in a MAC array with P=16 MAC processing elements 102 (e.g., PE0, PE1, ..., PE15) per row. The MAC processing elements 102 in the illustration can perform M=4 parallel MAC operations per cycle using M=4 parallel MAC processing units. Each MAC processing element with M parallel processing units operates simultaneously on both the input channel (C) and kernel (K) dimensions. The data path can have an MPR configuration, where each MAC processing element has M MAC processing units, each row has P processing elements, and each round has R input channels. The MPR configuration enables high throughput but introduces fine-grained switching activity during weight loading.

[0060] As later Figure 16-19 As described in detail, the data processing unit of a DNN accelerator may include a processing engine. The processing engine may include a grid of processing units, such as sparse processing units. Each processing unit may have an array of MAC processing elements or a MAC array that performs MAC operations. Figure 1 A MAC processing element 102 is depicted as part of a MAC array within a processing unit. The processing engine may be supplemented by a post-processing engine to process the output of the processing engine. The data processing unit may also have local memory (e.g., static random access memory (SRAM)) for storing and loading activations and weights for each DNN layer, a loading module for loading data onto the processing engine, and an output module for discharging data from the post-processing engine. The processing engine may include a controller that coordinates the loading, computation, partial accumulation, and extraction of output activations from the processing engine. In a DNN, MAC operations can be used to compute the dot product of numerous weights and hidden layer activations to produce the output feature map of the next layer. The MAC processing element in the processing unit can perform one or more MAC operations using a local data path. The local data path may include an input activation register file, a weight activation file, multipliers, and accumulators.

[0061] Although the processing engines can perform many parallel MAC calculations, they suffer power loss due to high switching activity, as weights routed to various MAC processing elements (e.g., MAC processing element 102) switch from one calculation cycle to another. Figure 1As shown, the cyclical nature of input acquisition and application of new weights at the MAC processing element 102 leads to a large number of bit transitions, thus exacerbating power demands in each computation cycle. The breakdown of power consumption highlights that switching power constitutes a significant portion of the processing engine's power consumption, and is itself a significant portion of the overall DNN accelerator's power consumption. This breakdown emphasizes how switching activity directly contributes to overall power inefficiency. The "weight switching" highlighted by the arrow indicates the continuous loading and switching of weights at the input of the MAC processing element, resulting in numerous binary transitions in computation rounds. Depending on the C and K mappings, bit-level transitions during weight loading can occur both intra-round and inter-round. This repetitive switching process, especially with random and dynamic weight values, leads to a significant increase in power consumption.

[0062] In the exemplary output static data stream, weights are distributed across columns of the MAC processing element by the output channel K, and the input channel C flows along rows of the MAC processing element 102. The spatial dimensions (Fx, Fy) of the convolution are temporarily unwound, and an optional double-buffered register file hides the memory latency across rounds. In some output stabilization modes, the innermost loop maintains output stability, i.e., the partial sums (Psums) remain local to the register file or accumulator during accumulation. If appropriate local buffers are present, the outer loop (e.g., on Fx, Fy, or K) can still follow a fixed input or weight pattern. Innermost loop output stability improves energy efficiency by avoiding expensive local memory to register file transfers and supporting high-bit-width accumulation (e.g., 32-bit integers (INT32) for INT8 MAC operations). Deferring local memory write operations reduces both data volume and precision-related bandwidth overhead, as the final output is written back with lower precision (e.g., INT8).

[0063] In such Figure 1 In the configuration shown, each MAC processing element maintains its own weight register file, the contents of which can be updated with new weights in each calculation round, as seen between round 1 104 and round 2 106. Due to the distribution of different filters across columns and the limited weight reuse in high R and high P settings, the weight sequence loaded into the MAC processing element 102 can exhibit large bitwise differences between consecutive values ​​within and between rounds (e.g., ...). Figure 1 (As indicated by the arrow in the image). This leads to frequent bit switching, increasing switching activity in the MAC data path. This irregular switching pattern significantly increases dynamic power consumption, especially when weights are extracted from different output channels without spatial or value locality.

[0064] Some methods, such as pruning, quantization, and clock gating, while valuable in reducing computational load or increasing control over clock cycles, do not address the core problem of bit-level switching. While these techniques can manage the overall computational load, they ignore the root cause of inefficiency: frequent and random switching at the weight level. In output-fixed architectures, the amount of switching can be particularly high, as shown in the figure. In designs that continuously load weights, the power consumption caused by switching can be especially problematic, leading to additional binary conversions that significantly increase power consumption. In output-fixed architectures, where weights are accessed repeatedly with each computation, each new weight loaded introduces a new binary conversion in the circuit module. Depending on the dynamically changing weights, these conversions are not only frequent but also random, resulting in significant power consumption across computation cycles. Random bit switching within the MAC unit each time a weight is loaded can further increase power consumption, leading to inefficiency that is difficult to control with conventional methods. Therefore, while pruning, quantization, and clock gating are useful for managing the amount of active computation, they fail to address the root cause of inefficiency in these architectures—frequent and random switching at the weight level. In output-fixed architectures, where weights are continuously loaded to perform computations, the power savings offered by other methods are limited because they do not address the fundamental problem of multiple bit transformations. Therefore, despite attempts to reduce computational load through these methods, other approaches still fail to solve the basic challenges. This limitation significantly reduces the effectiveness of other techniques because they cannot provide significant power savings when repeated weight accesses are involved, for example, due to the inherent characteristics of output-fixed designs, such as… Figure 1 What is depicted.

[0065] In an examination of a ResNet-50 scenario, under a typical MPR configuration (M=4, P=16, R=32), the switching profile characteristics at the MAC processing element level for the first three layers exhibit cross-layer and time-varying behavior. Some layers, such as layer1.0.conv3, show consistently high switching activity, reflecting poor weight locality. These switching activities can account for a significant portion of the total chip power and may dominate energy distribution even when the MAC is fully utilized. Unlike static power or computational intensity, switching activity is heavily influenced by how weights are arranged in memory and how they flow into computational units. However, current accelerators lack the flexibility to reorder weights at runtime, and existing software optimizations do not target this level of granularity.

[0066] In various embodiments of this disclosure, the solution to this problem involves weight reordering techniques to reduce bit transitions during MAC operations. By strategically reordering the weights to minimize switching activity, the compiled model can achieve significant energy savings without modifying the hardware or risking a reduction in accuracy, making it scalable and efficient across a variety of DNN accelerators.

[0067] Hamming distance as a metric for switching activities To reduce dynamic power consumption, heuristic weight reordering algorithms can be implemented to find weight orders that reduce switching activity during MAC operations. Heuristic weight reordering techniques can include compiler-level reordering solutions that minimize the MAC data path (e.g., such as Hamming distance between consecutive weights in the MAC data path) using Hamming distance as a heuristic rule. Figure 1 The switching activity between consecutive weights in the data path shown.

[0068] Bit-transform-driven switching activity is a major source of dynamic power consumption in circuits, especially in DNN accelerators where a large number of MAC operations occur in parallel, and in DNN accelerators where static output data streams are implemented, such as... Figure 1 As shown. The heuristic used in this algorithm can be a switching activity metric, such as the Hamming distance between consecutive weight values. By minimizing the Hamming distance between consecutive weight values, the weight reordering algorithm can find or determine the order or arrangement of weights to ensure that consecutive MAC operations cause fewer bit transitions, resulting in a significant reduction in dynamic power consumption while maintaining the accuracy of the neural network.

[0069] Hamming distance can serve as a primary metric for this optimization, representing the number of bit differences between two binary numbers. In this context, the Hamming distance between two weight values ​​is directly related to the number of bit transitions during computation. A larger Hamming distance results in more switching, leading to higher power consumption. Therefore, Hamming distance can be used as an efficient probe, metric, or measure of switching activity. In many cases, switching activity is proportional to the Hamming distance between consecutive weights. By minimizing the cumulative Hamming distance over the sequence of weight values, weight reordering algorithms can reduce total switching activity and associated power costs.

[0070] Given two binary strings A and B of equal length, the Hamming distance d(A, B) can be calculated as follows: Figure 2-3 The motivation behind using Hamming distance to minimize switching activities according to various embodiments is illustrated. Figure 2 The following illustrates a switching activity without weight reordering according to some embodiments of the present disclosure. Figure 3 Theoretical minimum switching activities with optimal weight reordering are illustrated according to some embodiments of this disclosure.

[0071] like Figure 2As shown, in the original sequence of weights, the weights are arranged in a default (raw or naive) order, which leads to significant switching activity. Due to the large Hamming distance between consecutive weight pairs, this default order results in a high total switching count, such as 32-bit transitions across the sequence (e.g., total switching = 32). Theoretically, as Figure 3 As shown, by ensuring that each consecutive weight pair has the minimum possible Hamming distance, the optimal reordering can minimize the total switching activity (e.g., total switching = 19). However, reaching this theoretical minimum requires a computational complexity of 𝑂 (𝑛!), which may be impractical for real-world applications because the complexity increases factorially with the number of weights.

[0072] Efficient weight reordering based on switching activity metrics To address this challenge, computationally more feasible or practical heuristic algorithms can be implemented. Heuristic methods can approximate computational costs by reducing switching activities while maintaining a much lower computational cost. Figure 3 The optimal solution is shown. This algorithm is referred to here as a centrally based greedy heuristic sorting algorithm, which has 𝑂(𝑛 2 The computational complexity is ).

[0073] The sorting algorithm can begin by selecting an initial pivot weight from the weight sequence. The initial pivot weight can be chosen randomly or at a predetermined position in the weight sequence. In each iteration, the algorithm can select or determine the weight in the weight sequence that has the minimum Hamming distance to the current pivot weight. This process can be repeated iteratively, with the selected weight becoming the new pivot weight for the next iteration. The reordering process can continue until the weights are rearranged into a sequence or order that minimizes switching activities as much as possible within the constraints of the heuristic method.

[0074] In other words, the algorithm begins by selecting an initial pivot weight and iteratively appends the next weight with the minimum Hamming distance to the current pivot weight. The next weight becomes the pivot weight for the next iteration. The iterative process can continue until the entire sequence is reordered to minimize local bit transitions.

[0075] Figure 4 The following illustrations demonstrate how weighted reordering reduces switching activity by applying a centrally based greedy heuristic sorting algorithm according to some embodiments of the present disclosure. Figure 4 As described, this heuristic sorting algorithm can reduce the total switching activity to 22 bit transitions (e.g., total transitions = 22), which is significantly lower than... Figure 2 The default sequence shown contains 32 transitions. Although there is no theoretical minimum for the total number of transitions (e.g., as shown in the example). Figure 3The 19 transformations shown are low, but the heuristic algorithm can be used with a more manageable 𝑂 (𝑛). 2 This reduction in computational complexity makes it suitable for real-time and large-scale applications.

[0076] Figure 5 A centrally based greedy heuristic weight reordering algorithm according to some embodiments of this disclosure is illustrated. Specifically, Figure 5 A more detailed view of the classification process is depicted. The first iteration (e.g., iteration #1) can begin with an initial hub, and the algorithm selects the weights with the minimum Hamming distance from the hub element. After being selected (e.g., once selected), this element can become the new hub, and the process can be repeated. In the second iteration (e.g., iteration #2), the algorithm selects the weights with the minimum Hamming distance from the new hub. After being selected (e.g., once selected), this element can become the new hub, and the process can be repeated. This iterative sorting process progressively minimizes the Hamming distance between consecutive weights, thereby reducing switching activity across the entire weight sequence. By selecting weights that minimize the transition at each step, the algorithm ensures efficient weight reordering, thus reducing dynamic power consumption.

[0077] In some embodiments, the compiler determines the order of multiple weights by performing multiple iterations, wherein in the iterations, the compiler selects one of the multiple weights as the current pivot, selects the next weight in the order of the multiple weights that has the minimum switching activity metric for the current pivot, and updates the current pivot to the next weight.

[0078] Integrate weight reordering into model compilation. Figure 6 A model compilation process 600 with weight reordering to reduce switching power consumption is illustrated according to some embodiments of this disclosure. When compiling a neural network model such as model 602, the model compilation process 600 integrates layer-by-layer weight transformation and optimization. The model compilation process 600 outlines one or more operations performed by the compiler to produce a compiled model to be deployed on a DNN accelerator. Figure 21 An exemplary compiler is described, which can perform one or more operations of the model compilation process 600. The compiler can optimize the model by reordering weights to improve TOPS / W.

[0079] In some embodiments, the model compilation process 600 begins when the compiler receives a model definition for a neural network model 602. Model 602 may include multiple layers, such as neural network layers. These multiple layers include layers with multiple weights that are applied to multiple input activations using an array of processing elements from a neural network accelerator. Figure 17-19As shown, the processing element array can be a MAC array.

[0080] In 604, the compiler can perform model parsing. Model parsing can include parsing the architecture of model 602, such as the recognition layers and neural network operations. Model parsing can include generating a processing graph with connected nodes representing neural network operations. Model parsing can include identifying the operations to be performed for a given node in the processing graph, as well as the shapes of the inputs and outputs of the operations for a given node in the processing graph. Model parsing can include forming the connections and processing order between nodes. Model parsing allows the compiler to understand the structure and computational operations to be performed when executing model 602.

[0081] At 606, the compiler checks whether the weights and / or activations should be quantized. If yes, the model compilation process 600 can proceed to 608 via the "yes" path. If no, the model compilation process 600 can proceed to 610 via the "no" path.

[0082] In 608, the compiler can perform quantization. Quantization (e.g., `quantize()`) can include applying scaling and zeros to weights and activations to reduce precision, thereby improving storage and computational efficiency while minimizing precision loss. In some implementations, quantization is performed to produce quantized weights. After quantization, the model compilation process proceeds from 600 to 610. While quantization, such as low-bit precision quantization, is widely used for model compression, memory reduction, and throughput improvement, it can also be used for the goal of reducing switching power. Quantization can reduce value entropy and compress the dynamic range of weights, which can be combined with weight reordering to address temporal patterns of weight transitions characterized by abrupt changes and irregular switching between discrete values. Enabling quantization and performing weight reordering provides a tunable trade-off between energy efficiency and accuracy, with additional power savings exceeding those achievable through weight reordering alone.

[0083] In step 610, the compiler implements layer-by-layer weight reordering or transformation to reduce switching activity for a specific DNN architecture. Weight reordering can be applied early in the compilation process (600) to perform layer-by-layer weight transformation. By optimizing the weight ordering based on the architecture, dynamic power efficiency and computational performance can be achieved for each layer. In some cases, the compiler further optimizes or transforms the weights through weight sparsity. For a specific layer among the multiple layers of model 602, the compiler can determine the ordering of multiple weights based on a switching activity metric, and determine multiple rearranged weights by arranging the multiple weights according to that order. The compiler can select one layer from the multiple layers of model 602 to perform weight reordering.

[0084] In 612, the compiler can perform activation data stream optimization: the compiler optimizes the activation data stream to take advantage of activation sparsity and data reuse, ensuring efficient data processing with minimal memory access and improved speed.

[0085] In 614, the compiler can perform weight and activation tiling. The compiler can tile weights and activations based on the accelerator's memory limitations and allows partitioning to speed up memory access and reduce data transfer time.

[0086] At 616, the compiler can generate an intermediate representation (IR) optimized for a specific target hardware. At 618, the compiler can translate model 602 into a format suitable for low-level hardware instructions, such as a machine-readable configuration. The compiler can generate one or more machine-readable configurations to configure the processing element array (e.g., MAC array) of the neural network hardware to apply multiple rearranged weights to multiple input activations. The compiler can generate one or more machine-readable configurations for a neural network accelerator to load multiple rearranged weights onto the processing element array according to the order and apply multiple rearranged weights to multiple input activations.

[0087] In the 620, the compiler can perform scheduling optimizations. The compiler can optimize the scheduling of computations to balance data transfer and computational parallelism, ensuring efficient use of hardware resources during execution.

[0088] In 622, the compiler can perform binary generation and compilation. The compiler uses a target-specific compiler to generate accelerator-specific binaries and produces an executable that can run on a DNN accelerator, incorporating optimizations performed during the compilation process.

[0089] The model compilation process 600 can efficiently transform weights and optimize the model at the foundational level, allowing DNN accelerators to dynamically handle sparse and dense computations. By leveraging this compiler support, weight reordering can significantly improve the efficiency of DNN accelerators, including existing ones. More importantly, the compiler's reordering of weights affects the arrangement of weights during compilation but does not change their numerical values ​​or affect mathematical MAC operations. Therefore, the accuracy of the DNN remains unaffected, while the rearranged weights can lead to fewer bit transformations and reduced dynamic power consumption.

[0090] Constraints on intra-layer and inter-layer weight reordering For some DNN accelerator implementations, there may be hardware limitations in how to apply weight reordering. Intra-layer and inter-layer dependencies may need to be considered. Figure 7-9 The intra-layer dependencies are shown. Figure 10-13 Inter-layer dependencies are shown.

[0091] In many currently available DNN accelerators, fixed-dataflow architectures are designed to optimize throughput by implementing predictable and synchronous data movement across MAC processing elements. However, this dataflow rigidity, which controls how input activations and weights are fed to the MAC processing elements, can introduce significant limitations, particularly in terms of weight reordering to minimize switching activity, which impacts dynamic power consumption.

[0092] The first limitation is the inflexibility in ordering the C values ​​(channels or input channels) within a specific K (output channel) on different MAC processing elements of the MAC array across the same computation round. Each K represents a set of computations, and within each K, there are specific C values ​​that need to be processed. During each round, these C values ​​can be distributed across multiple MAC processing elements, and their order is fixed. In other words, during each round of computation, weights from a fixed set of C values ​​are distributed across the MAC processing elements in a predetermined order.

[0093] In the examples where the C values ​​are arranged as [0, 1, 2, 3] and [4, 5, 6, 7] within K=0, this specific order is consistently maintained across the MAC processing element during that round. The inability to dynamically change the order of C values ​​to minimize switching activity means that the architecture does not allow such adjustments, even if rearranging the sequence of C values ​​could reduce bit transitions and power consumption. For example, while a heuristic might identify a better order to reduce switching activity, such as by reordering [4, 5, 6, 7] to [6, 4, 5, 7], the hardware data path does not allow reordering and forces the original fixed sequence for computation in the round. This rigidity forces the system to process weights in a predetermined order, which can lead to higher switching activity due to the large Hamming distance between consecutive values.

[0094] The second constraint is ensuring that a reordering established in a specific round remains consistent across subsequent rounds. This constraint enforces order consistency across K and rounds. This means that after arranging C values ​​within a specific K during the initial round (e.g., once C values ​​are arranged within a specific K), that arrangement is replicated for other Ks in the same round, and that order is retained for future rounds. Once a specific C ordering is established in a round, such as [4, 5, 6, 7] in round 2, the same sequence is repeated for the Ks processed in that round and for future computation rounds. This design approach ensures simplified control and consistent timing across MAC processing elements and eliminates the possibility of dynamically adjusting the order based on switching costs.

[0095] For example, when the C value in K=0 is sorted as [4, 5, 6, 7] during the first round, the same order needs to be used for K=1, K=2, and so on across MAC processing elements. This arrangement remains unchanged in subsequent rounds of computation. This stringent requirement ensures synchronization and predictability across MAC processing elements and reduces rearrangement overhead, but it severely limits the ability to adjust the data flow to reduce switching activity. Even when alternative arrangements could minimize bit transitions and save power, the system is locked into this fixed pattern, resulting in excessive switching and higher dynamic power consumption, especially during repeated loading and weight processing.

[0096] Figure 7 The correct weight loading order over time is illustrated according to some embodiments of this disclosure. The correct weight loading order takes into account the constraints described above. During a given round, for example, round 1 704 or round 2 706, the C ordering across MAC processing element 702 is the same. The C ordering is maintained across round 1 704 and round 2 706. The fixed hardware-level data flow into MAC processing element 702, where weights and activations are fed into MAC processing element 702 in a synchronous, repetitive pattern, is designed for efficiency and predictability, but prevents hardware from adjusting the order of C values ​​to accommodate a more power-efficient configuration. These constraints and limitations stem from the need to maintain synchronization and consistency across MAC processing element 702 to ensure high throughput and efficient parallel processing. However, this fixed structure also introduces significant power inefficiencies because the architecture cannot dynamically adjust the data flow to minimize switching.

[0097] Figure 8 The following illustrates an incorrect weight loading order over time according to some embodiments of this disclosure. For K in a round, the C order cannot be changed, as seen in round 2 806 where K=17. Changing the C order of K in a round will result in incorrect calculations because the input activations of the same group flow to the MAC processing element 702 in a fixed order, and changing the order of K in a round will result in the input activations being multiplied by incorrect weights.

[0098] Figure 9 Some embodiments according to this disclosure are shown, in which the weight ordering is maintained in each round, but changes in the weight ordering in successive rounds. As seen in rounds 1 704, 2 906, and 3 908, the C ordering is maintained for K within a round. In a round, each MAC processing element 702 follows and processes the C values ​​in each k in a strict order. While theoretically energy-efficiently changing the C ordering between successive rounds, as seen between rounds 2 906 and 3 908, changing the C ordering introduces overhead and risks disrupting the synchronization and timing of weights and input activations fed into the MAC processing element 702.

[0099] Existing hardware constraints can introduce inter-layer dependencies. Specifically, while weight reordering achieves dynamic power reduction, applying weight reordering across multiple layers of a DNN model requires consideration of how weights, activations, and output channels are interconnected in the DNN data stream. Specifically, the inter-layer dependency between the output channel (K) of layer i and the input channel (C) of layer i+1 introduces a structural constraint, referred to here as the layer-by-layer reordering constraint, which affects where and how weight reordering can be applied in the DNN model. Figure 10-13 This highlights the interdependence between the order of input channels (IC) and output channels (OC) across layers.

[0100] Figure 10 Sample layer computation and hardware models according to some embodiments of this disclosure are shown. In a convolutional layer, activations (ACT) are multiplied by weights (WGT) to produce an output (OUT). ACT can be arranged along the input channel dimension (C). WGT can be arranged along the input channel dimension (C) and the output channel dimension (K). OUT can be arranged along the output channel dimension (K). The output channel (K) of layer i is then consumed as the input channel (C) of layer i+1. Therefore, any reordering of K in layer i is performed by reordering C in layer i+1 to maintain functional correctness.

[0101] exist Figure 10 In this model, weights (WGT) interact with activations (ACT) to produce outputs (OUT) in a specific layer. For illustrative purposes, an exemplary hardware model is depicted where the hyperparameters have specific values ​​as shown in the figure, such as M=1, P=1, R=C=4=K, H=1=w, Fx=1=Fy. It is foreseeable that other values ​​of the hyperparameters can be used in the hardware model. When optimizing or reducing weight switching, the dependencies between layers should be considered.

[0102] Figure 11 The illustration shows unaltered computations performed in layers 1 and 2 according to some embodiments of the present disclosure. In the illustration, the neural network computes where activations (ACT) and weights (WGT) are processed sequentially in layers (e.g., layers 1 and 2) without any ordering modification. Each layer performs operations independently, and the output (OUT) of one layer is directly used as the input (ACT) of the next layer. In this configuration, activations and weights follow a fixed index order for each layer.

[0103] To apply the weighted reordering transformation, consider two reordering options.

[0104] Figure 12This illustration shows some embodiments of the present disclosure in which both the input channel order and output channel order of Layer 1 are reordered, while the input channel order and output channel order of Layer 2 remain unchanged. Option I reorders the "C" (IC) and "K" (OC) dimensions in Layer 1 while keeping Layer 2 unchanged. This reordering modifies the weights (WGT) and outputs (OUT) in Layer 1, optimizing that layer, but does not propagate these optimizations to Layer 2. In Layer 1, for each K, the C ordering changes from 0123 to 0231, and the K ordering changes from 0123 to 0312. The K ordering of Layer 1 becomes the C ordering of Layer 2, meaning that the C ordering of Layer 2 depends on Layer 1. Due to the change in the IC / OC or C / K order, the reordering of K applied in Layer 1 instructs Layer 2 to adapt the C ordering to the K ordering of Layer 1.

[0105] Figure 13 The following illustrations show reordering of the input channel order of layer 1 and layer 2 according to some embodiments of this disclosure. Option II attempts to partially optimize this by reordering the "C" dimension of both layers (layer 1 and layer 2). The "K" dimension is reordered. Here, the IC order of layer 2 determines the "K" or OC order of layer 1, or more generally, the C order of layer i+1 determines the K order of layer i. This means that the reordering of each layer affects the next layer, thus enforcing a dependency where the OC or K order of one layer is consistent with the IC or C order of subsequent layers.

[0106] Figure 12 Option I shown involves reordering the C and K sorts of layer i, and not applying the reordering of C and K in layer i+1. This option can be implemented to minimize intra-layer switching activity. This reordering, for example, applying the C1 and K1 sorts in layer 1, is reflected in... Figure 12 In the output activation (OUT1) of layer 1, it now follows a reordered K, such as K1. These reordered outputs with K1 ordering are directly fed into layer 2 as its activation, where the K1 ordering is used as the C2 ordering without requiring additional reindexing hardware. Because layer 2 does not undergo any weight transformation, layer 2 simply inherits the K1 ordering from layer 1 as its input C2 ordering, maintaining correctness and alignment. This strategy of applying weights to reorder layers while keeping the next layer unchanged maintains inter-layer compatibility while still gaining the benefit of reduced switching activity in transformed layers. This strategy avoids the need to implement dynamic reordering hardware between layers and avoids the overhead of activation reformatting.

[0107] Optimal layer selection algorithm In some embodiments, option 1 can be used to implement weight reordering because it provides more degrees of freedom within a layer by allowing reordering of both C and K. This flexibility allows for better optimization opportunities for sparsity transformations and power efficiency at the layer level. Since the K ordering in each layer determines the C ordering in the next layer, it is not feasible to apply weight reordering consecutively across layers under this option. Reordering both C and K in layer i and then in layer i+1 would lead to conflicting assumptions about activation ordering and interrupt compatibility. Therefore, to maximize the power reduction potential of weight reordering while maintaining layer-to-layer correctness, a skip-layer strategy can be employed, where weight reordering is applied to layers, but their direct successors remain unchanged. The skip-layer strategy avoids the hardware challenges associated with consecutive layers and ensures that weight reordering is not applied to adjacent layers in the network.

[0108] To systematically determine which layers to apply weight reordering to, analytical models can be developed. Without the constraint that two consecutive layers can be selected for weight reordering, the analytical model can select layers based on their optimization potential. This model can strategically identify layers where weight reordering is generally beneficial (e.g., layers where weight reordering is most beneficial overall or can achieve greater power savings overall), balancing optimization with hardware feasibility.

[0109] Figure 15 An optimal layer selection algorithm 1400 according to some embodiments of this disclosure is described. The optimal layer selection algorithm 1400 can be implemented to determine where weight reordering can be most effectively applied in a DNN model. Specifically, the optimal layer selection algorithm 1400 can analytically select one or more layers(s) of the DNN model to apply weight reordering. Algorithm 1400 can identify an optimal subset of layers to apply weight reordering, subject to hardware constraints. Specifically, algorithm 1400 ensures that no two adjacent layers are selected, thereby maintaining inter-layer compatibility, as previously described. Figure 12 The subject of discussion.

[0110] In some embodiments, Algorithm 1400 examines the analytical model to perform layer selection. Algorithm 1400 may identify the optimal layers in the DNN model for applying weight reordering (e.g., reordering both K and C of the layer based on the Hamming distance between weight pairs), which aims to reduce switching activity during weight loading into the MAC array of the DNN accelerator. Algorithm 1400 may consider the number of weights per layer and one or more switching activities, selecting those layers whose weight reordering will most effectively reduce the overall power consumption of the model.

[0111] Algorithm 1400 will use the number of layers L and the switching activity score for each layer. (Based on bit transformation and weight reuse) and the number of weights in each layer As input, the switching activity score for each layer can be represented as an array of switching activity scores for each layer. When the weight set is loaded onto the circuitry of the neural network accelerator, the switching activity score can quantify the number of bit transitions or bit flips for a given layer. The number of weights for each layer can be represented as an array of weights for each layer. In some embodiments, the input also includes user-defined parameters (e.g., "selection") to influence the selection process. These inputs can be used to compute optimization metrics for each layer. For example, the product of the switching activity and the number of weights in this layer. The output of Algorithm 1400 is , which is a list of the selected layer indices.

[0112] In lines 1-2 of Algorithm 1400, when the model has a single layer ( When ), Algorithm 1400 can directly return index [0].

[0113] The third row and subsequent rows handle models with more than one layer.

[0114] In lines 3-4 of Algorithm 1400, the optimization metric for each layer is calculated. The optimization metric can be considered as the switching cost for a given layer. The optimization metric for layer i... It can be based on the switching activity score of each layer (e.g.) ) and the number of weights in each layer (e.g. One or more of the following are used for calculation. In line 4, Algorithm 1400 is based on the product of the switching activity score and the number of weights (e.g., This is used to calculate the switching cost of a given layer. Intuitively, both the layer's switching activity score and the number of weights contribute to dynamic power consumption. In some embodiments, additional weights (user-defined parameters, such as "selection") can be applied to adjust the optimization metrics.

[0115] In line 5, Algorithm 1400 can initialize two variables: and Each stores the sum of the metrics of the two layers preceding the current layer and the index of the selected layer of the two layers preceding the current layer. and These are variables that store the sum of metrics up to the previous level and the index of the selected level up to the previous level, respectively. and These are variables that store the total metric up to the current layer and the index of the selected layer up to the current layer, respectively.

[0116] Using these variables, Algorithm 1400 determines, under the constraint that consecutive layers cannot be jointly selected, whether the current layer yields a higher cumulative return than excluding the current layer. Algorithm 1400 implements a dynamically programmed formula, where it maintains two running solutions: one using... and Including layers up to i-2, another uses and This includes layers up to i-1.

[0117] In lines 6-9, for layers 1 and 2, Algorithm 1400 compares the optimization metrics of layer 1 and layer 2 (e.g., ...). ), and select layers with higher optimization metrics.

[0118] For each layer i greater than 2, Algorithm 1400 compares skipping the current layer (e.g., inheriting solutions up to i-1) with the cumulative metric including the current layer (e.g., adding the optimization metric of the current layer to solutions up to i-2). In each iteration, as shown in lines 11-16, Algorithm 1400 updates the running best solution based on which solution produces the larger total gain.

[0119] Once the final layer has been processed, Algorithm 1400 returns the index of the selected non-contiguous layer. This index maximizes the total switching reduction potential of the DNN model. This approach balances optimization opportunities and hardware feasibility, allowing weight reordering to be deployed where it is most effective.

[0120] In some embodiments, the algorithm 1400 is based on the switching activity score of each layer (e.g. ) and the number of weights in each layer (e.g. One or more of the following can be used to select a subset of layers from a plurality of layers. When weights are, for example, loaded onto an array of processing elements, the switching activity score can quantify the number of bit transitions or bit flips for a given layer. The switching cost of a given layer can be based on the switching activity score for each layer (e.g., ...). ) and the number of weights per layer (e.g., One or more of the following, such as the product of switching activity score and weight number.

[0121] In some embodiments, algorithm 1400 selects a subset of layers from a plurality of layers under the constraint of selecting only non-contiguous layers. Algorithm 1400 can select a subset of layers under constraints and based on, for example, the switching cost per layer (e.g., Use ) to make the selection.

[0122] In some embodiments, the algorithm 1400 iteratively compares the cumulative switching costs when skipping the current layer (e.g., ) and additional cumulative switching costs when selecting the current tier (e.g. To select a subset of layers from multiple layers. Cumulative switching cost. This includes, for example, the sum of all switching costs calculated for selected non-contiguous layers up to layer i-1. Cumulative switching costs. This can represent the maximum cumulative switching cost (e.g., total switching reduction potential) considering the optimal solution up to layer i-1 (i.e., excluding the current layer i). Additionally, the cumulative switching cost includes, for example, the sum of all switching costs calculated for the selected non-contiguous layers up to layer i-2 and the switching cost of the current layer i. This can be expressed as the maximum cumulative switching cost (e.g., total switching reduction potential) of the optimal solution considering layers up to i-2 (i.e., excluding layers i-1 and i-2), plus the switching cost of the current layer i. If If so, skip the current layer. Otherwise, or if If the current layer is selected, then the cumulative switching cost and other cumulative switching costs are updated accordingly after the comparison.

[0123] Optimal weight reordering within the selected layer Figure 16 A hierarchical weight reordering process 1500 according to some embodiments of this disclosure is described. The hierarchical weight reordering process 1500 illustrates an example of a structured method for reordering weights within a DNN model to reduce switching activity, with a particular focus on minimizing the Hamming distance between consecutive weight elements while respecting the inherent constraints of the data flow of the DNN accelerator. The hierarchical weight reordering process 1500 defines a systematic approach to applying weight reordering to reduce switching activity within a DNN layer while adhering to hardware-imposed constraints.

[0124] Layer-by-layer weight reordering process 1500 applications from Figure 14 Algorithm 1400 is used to generate (one or more) selected layers. .

[0125] right Each layer Repeat the operation in process 1502. In other words, the layer-by-layer weight reordering process 1500 traverses the pre-selected list of layers ( This involves reordering the layers using weights. The list of layers can be based on the combination described above. Figure 14 The optimal layer selection is derived from an analytical model that ensures that no two consecutive layers are selected.

[0126] For each selected layer, process 1502 performs L0 sorting as shown in operation 1508, and L1 sorting as shown in operation 1518.

[0127] L0 sorting can include the following operations. L0 sorting can target the K block with the highest switching activity by selecting the central K (OC) block within the current layer or The process begins. The central K-block is then analyzed to determine the optimal C (IC) ordering, aiming to reduce switching activity and align the sparsity patterns of the layer. The values ​​of M, P, and C are coordinated in a lockstep manner to ensure alignment between different parameters for efficient processing. A sorting mechanism based on a switching activity metric is then applied to minimize the Hamming distance between rows within the K-block (e.g., rows within the K-block correspond to different ICs). By reducing the Hamming distance, this operation optimizes switching efficiency and power consumption within the central K-block. After the C ordering of the central K-block is established (e.g., once the C ordering of the central K-block is established), the determined C ordering is implemented across K-blocks within the layer using L0 sorting, standardizing the C reordering pattern across the entire layer.

[0128] In operation 1504, select the central output channel (K) block. The selected central K-block exhibits the highest switching activity. This can be calculated based on previous profiling or metric estimation.

[0129] In operation 1506, the current C block is selected from the layer's C blocks. The current C block can start from the first input channel (C) block (e.g., in Each C block in this layer can have R input channels.

[0130] In operation 1508, L0 sorting determines the central K block. The optimal sorting of row C. In other words, L0 sorting minimizes the central K block. The optimal C-order is determined by the Hamming distance between adjacent rows of the weights or other suitable switching activity metrics, for example. L0 sort, minimize The Hamming distance between rows effectively reduces intra-K block switching caused by bit transitions between input channels (C) during MAC operations. The compiler determines the ordering of multiple weights, such as the C ordering of the central K block, which reduces row-to-row switching activity for multiple weights corresponding to multiple input channels of the layer.

[0131] Given weights typically constructed in the form of M×P groups, and C values ​​processed in a lockstep manner, this optimal C-sorting... Cross-architecture grouping is maintained. In operation 1510, across the current C block ( The K block in ) implements the C sorting determined in operation 1508 to ensure a consistent pattern across layers.

[0132] Operations 1508 and 1510 can be repeated individually for other C blocks in this layer. Before allowing process 1502 to proceed to operation 1514, in 1512, a loop checks whether the last C block has been reached, for example... If the final C block is not reached, process 1502 continues to return to operation 1506. Each C block can have a unique C order that results in a reduction in the maximum switching (determined in operation 1508 using the central K block selected in operation 1504).

[0133] L1 classification can include the following operations. L1 sorting can select a new central C (IC) block within the current layer, targeting the C block with the highest switching activity. The algorithm then analyzes the central C block to determine the optimal K (OC) ordering, aiming to reduce or minimize the Hamming distance between the last row of the current K block and the first row of the next K block, thus reducing the transformations required when moving between blocks. After establishing the optimal K ordering for the central C block, the algorithm implements this K ordering across the C blocks of that layer, ensuring the consistency of the reordering pattern. To complete the permutation, C and K orders are performed across Fx and Fy for the next layer, aligning activations based on the reordering structure of the current layer.

[0134] In operation 1514, the target is to switch between K blocks, and the central input channel (C) block is selected. The selected central block C exhibited the highest observed switching activity. This can be calculated based on previous profiling or metric estimation.

[0135] In operation 1516, L1 sorting determines the central block C. The optimal K-block sort. In other words, L1 sort determines the optimal K-block sort, for example, by minimizing the Hamming distance between the last row of one K-block and the first row of the next K-block, such as... L1 sorting reduces switching activity between input channel C accesses across consecutive K blocks, i.e., K-block switching. The compiler can determine the sorting of multiple weights, such as the K-sorting of the central C block. This reduces switching activity between the last row of multiple weight rows (e.g., the last row of a K block) and the first row of multiple other weight rows in the layer (e.g., the last row of another K block). Rows of a K block can correspond to multiple input channels of the layer. Additional rows of additional K blocks can correspond to multiple additional input channels of the layer.

[0136] In operation 1518, the C blocks across layers uniformly implement the identified or determined K sorting. .

[0137] In operation 1520, the identified or determined C sorting and the identified or determined K ordering cross-layer weights and Implementation.

[0138] In operation 1522, to maintain cross-layer correctness, the determined K-order is then aligned with the input channel sorting (C-ordering) of the next layer and the output channel sorting (K-ordering) of the current layer. This is mapped to the active layout of the next layer. This maintains functional equivalence without introducing additional reordering hardware.

[0139] After execution process 1502, the layer-by-layer weight reordering process 1500 is performed. The next selected layer repeats process 1502, continuing until... The remaining specified layers are processed. Procedure 1502 determines the optimal weight ordering along the C and K dimensions of the weights for the selected discontinuous layers. Procedure 1502 can efficiently select and rearrange layer parameters to reduce handover and power consumption without affecting adjacent layers, utilizing Hamming distance minimization techniques at the L0 and L1 levels to maximize the effectiveness of weight reordering under the constraints of existing hardware. This two-level ordering process enables weight reordering to minimize both intra-K-block (via L0 ordering) and cross-K-block (via L1 ordering) handover under realistic architectural constraints such as fixed channel grouping and inter-layer alignment. By strengthening consistency within each layer and maintaining cross-layer compatibility, the algorithm achieves meaningful energy savings without implementing hardware modifications or accuracy trade-offs.

[0140] Switching Perception Architecture Based on the guiding principles behind weight reordering to reduce switching activity, DNN accelerators can be made more efficient by adopting different architectural frameworks. Specifically, this framework can achieve power-efficient DNN accelerators by selecting a configuration with a high M (e.g., the number of MAC processing elements per MAC processing element) and a low P (e.g., the number of MAC processing elements in parallel in the MAC array). In this configuration, bit transitions can be minimized by processing weights sequentially within the same filter, resulting in inherently lower switching activity. This customized approach can reduce dynamic power consumption based on user-specific region and power constraints. Furthermore, the described weight reordering algorithm can be applied on top of these configurations, providing further switching reductions without any hardware modifications. This two-layer strategy ensures maximum energy efficiency for machine learning applications in edge and client computing environments. The switching-aware architecture framework with integrated weight reordering can achieve dynamic power savings and significantly reduce power consumption across a variety of DNN models. These results demonstrate the effectiveness of this framework in reducing bit transitions and optimizing dynamic power efficiency, making it a general solution for DNN accelerators.

[0141] Insights gained from weight reordering optimization extend to hardware architecture design. While weight reordering operates at the software level to reduce switching activity by reordering weights within a fixed data stream, it can also optimize the underlying hardware configuration itself to influence switching behavior, even when total computational throughput remains constant. Switching-aware architecture design frameworks can help select accelerator configurations optimized for low-level transitions (e.g., MPR configurations). In some spatial DNN accelerators, the number of MAC processing units is typically determined by three architectural parameters: M, P, and R in an MPR configuration. Designers can choose combinations of M, P, and R to maintain the overall computational budget. However, the effectiveness of weight reordering in reducing dynamic power consumption under different configurations reveals that the same total number of operations can lead to very different switching characteristics, depending on how data is spatially partitioned and accessed.

[0142] This difference arises because switching activity is closely related to the locality and correlation of the weights processed by each MAC processing unit. In a configuration where MAC processing units process weights associated with a single filter across multiple input channels (e.g., the same output channel K), the weights tend to exhibit natural correlation; they evolve smoothly, and their values ​​are typically close in magnitude or sign. This spatial proximity of the weight distribution results in a smaller Hamming distance between consecutive values, thus reducing switching activity. Conversely, when MAC processing units are assigned to rapidly and consecutively process weights from different filters (e.g., across multiple K), the likelihood of abrupt transitions between different values ​​increases, leading to high-bit switching and thus increasing dynamic power.

[0143] By strategically selecting an MPR configuration that minimizes switching activity, a more power-efficient architecture can be achieved without changing the total number of MAC operations. For example, consider two configurations with different M and P values ​​but the same MAC count: M=1 and P=16, and M=16 and P=1, regardless of the R value. In the 1-16 configuration, one MAC unit processes 16 different ICs from 16 different K values, resulting in significantly higher switching activity. This is because each MAC unit is receiving weight values ​​that are adjacent to each other in the IC dimension, and since these weights come from the same filter, their values ​​can vary greatly. This high variability leads to frequent bit transitions, thus increasing switching power. In the first configuration, each MAC processing element processes weights from 16 different filters (K values) on one round of input channels. Since each weight comes from a different filter, there is no guarantee of continuity or smoothness between them. The result is a highly irregular switching distribution with frequent large Hamming distances. On the other hand, in the 16-1 configuration, 16 MAC units process ICs associated with the same K value in parallel, inherently resulting in lower switching activity. Here, each MAC unit processes the weights associated with a single filter within the same K value in a given round, and then in the next round, it processes the next set of ICs from the subsequent K value. Since the weights within a single filter tend to have smoother value changes, this configuration introduces fewer bit transitions, thus reducing switching power. The lack of adjacency between significantly different weights contributes to this reduced switching, as each MAC unit is exposed to more similar weights in successive operations. In the second configuration, each MAC processing element processes multiple weights from the same filter in a round, which generally results in smoother transitions and lower cumulative switching. While the arithmetic throughput is the same in both cases, the second configuration exhibits a more power-efficient switching profile due to better weight correlation.

[0144] This behavior is further confirmed by observations of structured pruning. This phenomenon reflects the principles of structured sparsity and pruning techniques, where pruning in larger groups (e.g., 4:8) preserves functional properties better than a more granular (1:2) scheme. In structured pruning, 1:2 or 2:4 pruning patterns (where smaller subsets of weights are removed) tend to have a more detrimental effect on accuracy compared to larger pruning ratios (e.g., 4:8). This is because larger pruning windows provide a more global view, allowing the identification of potentially redundant or less important weights without significantly impacting model performance. The reason is that broader grouping captures more global context, thus enabling better preservation of structure and relevance.

[0145] The switching-aware hardware architecture framework follows this guiding principle: by selecting an MPR configuration that favors depth-level reuse (high M) and reduces inter-filter weight shuffling (low P), the accelerator can naturally align computation with filter locality, thereby minimizing unnecessary switching activity. In configurations with M=16 and P=1, a wider view of the IC within the same K value reduces abrupt weight changes, allowing for smoother data flow with fewer bit transitions and more efficient power consumption. This basic reasoning illustrates that by selecting a configuration with fewer MAC processing elements in the MAC array (e.g., M=16 and P=1) on highly parallel configurations (e.g., M=16 and P=16), the system can achieve a naturally lower switching activity profile. This configuration choice can leverage weight adjacency and similarity within the same filter to create an architecture with inherently lower switching power. This flexibility allows for a custom approach in architecture design, where configurations are selected based on their inherent switching behavior to meet area and power consumption constraints.

[0146] Another consideration is that not all switching activities are equally costly. In spatial accelerators, MAC array-level switching, especially switching across multiple rows or columns, incurs greater energy overhead than local switching triggers within small functional units. Therefore, minimizing high-amplitude Hamming transitions across block boundaries (e.g., between consecutive rows of weights assigned to different MAC units or between output filters) is beneficial. The switch-aware hardware architecture framework leverages this by supporting configurations that allow weight transitions to remain within the same filter window for as long as possible, reducing cross-block switching and maintaining more stable signal coverage. Ultimately, the switch-aware hardware architecture framework provides a principled approach to directly integrating power awareness into the architecture design process. It enables designers to explore the MPR design space not only for performance or area efficiency but also for dynamic power minimization, leveraging the inherent structure and activation reuse of neural network weights. By embedding switching behavior into the cost model of architecture selection, the switch-aware framework bridges algorithm-hardware co-design, enabling more energy-aware DNN accelerators without sacrificing functional accuracy or architectural simplicity.

[0147] In designing DNN accelerators, switching activity is a critical factor affecting power consumption and overall efficiency. The switching-aware architecture framework emphasizes that even with the same number of MAC units (as determined by the M, P, and R values ​​in the MPR configuration), different configurations can lead to significant variations in the level of switching activity. In some experiments, high variability in switching efficiency has been observed across different models (e.g., MobileNetV3-large, ResNet50, Inception-V3, and GoogleNet) with different MPR configurations. Despite having the same number of MAC operations, different MPR configurations exhibit different switching characteristics based on the distribution of M, P, and R. This switching variability can be used to customize DNN accelerators to meet user-specific requirements, balancing area constraints and power efficiency.

[0148] Compiler-managed weight reordering adds an extra layer of optimization to this switch-aware architecture selection framework. After selecting the optimal configuration based on switching characteristics (e.g., once the optimal configuration is selected), a weight reordering algorithm can be applied to further reduce switching activity. This reordering can be achieved without any additional area or energy overhead, effectively providing a "free" benefit. The algorithm rearranges weights to minimize bit transitions, complements the inherent efficiency of the selected configuration, and maximizes power savings. The combination of using a switch-aware framework to select an inherently low-switching MPR configuration and applying weight reordering for further optimization results in efficient DNN accelerator designs well-suited for energy-sensitive applications in client and edge computing.

[0149] This handover-aware design framework, enhanced by weight reordering in the compiler, provides a general toolset for achieving high performance and low power consumption within constrained regions. By selecting a configuration that naturally minimizes handover and then leveraging weight reordering optimization for even greater power savings, developers can meet the specific power and area targets of various AI applications, including imaging, video, and speech processing. This two-layer approach of reducing handover activity ensures that DNN accelerators can achieve optimal performance with minimal energy and area costs, reinforcing the value of weight reordering as a key enabler for efficient and scalable AI hardware solutions.

[0150] Implementing DNNs on processors or neural network hardware accelerators The significant improvements in DNN model size and accuracy, coupled with the rapid growth in the computing power of execution platforms, have led to the adoption of DNN applications, even in resource-constrained mobile and edge devices with limited power availability. DNN models can be executed by DNN accelerators (or, as referred to herein, neural network hardware accelerators), for example, for training or inference. A DNN accelerator can be or includes one or more data processing units or DPUs. A DPU can also be referred to as a computation block or computation chip. A DPU has highly specialized hardware circuitry modules that perform neural network operations. A DPU may include one or more processing engines capable of performing neural network operations or computational operations. A processing engine may include one or more processing units to perform arithmetic operations associated with neural network operations, such as multiplication and multiplicative accumulation operations. A DPU may include one or more PPEs that can perform neural network operations, such as scaling, adding biases, and applying activation functions.

[0151] Here, and as those skilled in the art understand, a tensor is a mathematical object that includes scalars, vectors, matrices, and even higher-dimensional data structures. At its most basic level, a tensor can be a single number, called a scalar. When extended to one dimension, a tensor can be a vector, which is a set of numbers. Further extended to two dimensions, a tensor can be a matrix, which is a grid of numbers. Beyond these, tensors can exist in multiple dimensions, representing complex data structures that can be manipulated and transformed in various ways by DNN accelerators. In the context of neural networks, tensors can be used to store multidimensional data. Neural networks involve tensor operations. Examples of operations can include addition, subtraction, multiplication, convolution, shaping, transposition, slicing and indexing, broadcasting, etc. These operations manipulate and transform tensors to perform neural network tasks such as training and inference.

[0152] Figure 16 A DNN system 1600 according to some embodiments of the present disclosure is illustrated. The entire DNN system 1600 or a portion thereof may be implemented in one or more computing devices, such as… Figure 23 The computing device 2300 is included. The DNN system 1600 can generate and execute DNNs, such as transformer-based neural networks, CNNs, etc. Figure 16As shown, the DNN system 1600 includes a DNN module 1601 and a DNN accelerator 1602. In other embodiments, the DNN system 1600 may include alternative configurations, different or additional components. For example, the DNN system 1600 may include multiple DNN modules or multiple DNN accelerators. Furthermore, the functionality of the components attributed to the DNN system 1600 may be performed by different components or different systems included in the DNN system 1600. In some embodiments, the DNN module 1601 and the DNN accelerator 1602 may include different types of processing units or be implemented by different types of processing units. In the example, the DNN module 1601 may be implemented by one or more central processing units (CPUs). The DNN accelerator 1602 may also be referred to as a neural network hardware accelerator, neural processing unit, AI accelerator, or AI processor. The DNN module 1601 and the DNN accelerator 1602 may be implemented in the same chip or as separate chips.

[0153] DNN module 1601 facilitates the generation and deployment of DNNs. In some embodiments, DNN module 1601 can generate and train DNNs. For example, DNN module 1601 can define the hierarchical architecture of the DNN. DNN module 1601 can also determine the intrinsic parameters of the DNN through the DNN training process. DNN module 1601 can also determine one or more hyperparameters that define how the DNN is trained. An example hyperparameter is the sparsity rate, which defines the sparsity level of one or more deep learning tensors of the DNN.

[0154] DNN module 1601 may, for example, compress the DNN during or after training. In some embodiments, DNN module 1601 may prune weights in one or more layers of the DNN by turning non-zero value weights to zero. DNN module 1601 may prune weights based on a target weight sparsity rate. The weight sparsity rate may be the ratio of the number of zero-value weights to the total number of weights. In the example where DNN module 1601 prunes weights during DNN training, DNN module 1601 may prune the layer weights after one or more epochs to achieve the target sparsity rate. DNN module 1601 may prevent the pruned weights from changing values ​​during the remainder of the training process. Alternatively, DNN module 1601 may allow the pruned weights to change values ​​such that the pruned zero-value weights may have non-zero values ​​after further training. DNN module 1601 may prune the layer weights again after one or more additional epochs.

[0155] DNN module 1601 can deploy trained, compressed, or validated DNNs for deep learning applications. In some embodiments, DNN module 1601 can distribute trained, compressed, or validated DNNs to devices or systems that can use the DNNs to perform tasks (e.g., image classification, motion planning, etc.), with the DNNs being trained for those tasks. In other embodiments, DNN module 1601 can use DNN accelerator 1602 to facilitate DNN deployment. For example, DNN module 1601 can receive data from devices or systems coupled to DNN system 1600 and input the received data (or data generated by DNN module 1601, for example, based on the received data) into the DNN. In some embodiments, DNN module 1601 can control the execution process of the trained, compressed, or validated DNNs.

[0156] DNN module 1601 can compile instructions executable by DNN accelerator 1602 to perform DNN operations according to the model definition of the DNN. DNN module 1601 can generate instructions (e.g., configuration descriptors, low-level machine instructions, etc.) that control the operation of DNN accelerator 1602 during DNN execution. The instructions may correspond to one or more data processing workloads passed from DNN module 1601 to DNN accelerator 1602, which will be executed by DNN accelerator 1602. DNN module 1601 can act as a compiler for the DNN to be deployed to and executed by DNN accelerator 1602. DNN module 1601 can compile the DNN and generate configuration descriptors and / or low-level machine instructions, based on which the DNN can be executed. Instructions can be used to configure or control the processing units of processing engine 1670 to perform one or more deep neural network operations. Instructions can be used to configure or control post-processing engine 1680 to perform one or more operations, such as applying activation functions. Figure 20 Some aspects of the DNN module 1601 are described and illustrated in the document.

[0157] DNN module 1601 can receive the output of DNN from DNN accelerator 1602. DNN module 1601 can transmit the output of DNN (or the result of DNN module 1601 processing the output of DNN) to a device or system.

[0158] The DNN accelerator 1602 executes DNN operations based on instructions (configuration descriptors and / or low-level machine instructions) provided by the DNN module 1601. For example, the DNN accelerator 1602 can execute a DNN by running deep learning operations within the DNN. The process of executing deep learning operations is also referred to as the process of performing deep learning operations or implementing deep learning operations. The execution of the DNN can be used to train the DNN or to use the DNN to perform AI and / or inference tasks.

[0159] like Figure 16 As shown, the DNN accelerator 1602 includes a memory 1610, a direct memory access (DMA) engine 1620, and a data processing unit 1630 (referred to separately as "data processing unit 1630"). In other embodiments, the DNN accelerator 1602 may include alternative configurations, different, or additional components. For example, the DNN accelerator 1602 may include more than one memory 1610 or DMA engine 1620. As another example, the DNN accelerator 1602 may include a single data processing unit 1630. Furthermore, the functionality of the components attributable to the DNN accelerator 1602 may be implemented by different components or different systems included in the DNN accelerator 1602. The components of the DNN accelerator 1602 may be implemented using hardware, software, firmware, or some combination thereof.

[0160] Memory 1610 stores data associated with deep learning operations performed by DNN accelerator 1602. Example deep learning operations include convolution (also known as "convolution operation"), layer normalization operation, SoftMax operation, matrix multiplication operation, pooling operation, element-wise operation, activation function, other types of deep learning operations, or some combination thereof.

[0161] In some embodiments, memory 1610 may store data used by data processing unit 1630 for DNN execution. Memory 1610 may store weights, such as the weights of convolutional layers determined by training the DNN. Memory 1610 may also store the inputs and / or outputs of DNN layers, such as data generated by data processing unit 1630 by performing deep learning operations in the DNN.

[0162] Memory 1610 may store instructions (e.g., configuration descriptors, low-level machine instructions, etc.) that can be executed by DNN accelerator 1602, such as instructions that can be executed by data processing unit 1630. Memory 1610 may be the main memory of DNN accelerator 1602. In some embodiments, memory 1610 includes one or more dynamic random access memories (DRAMs). In some embodiments, cache 1612 may be a cache of memory 1610. Cache 1612 may include one or more SRAMs. Cache 1612 may provide faster data / memory access than memory 1610. Cache 1612 may store frequently accessed data. The capacity of cache 1612 is smaller than the capacity of memory 1610.

[0163] DMA engine 1620 facilitates data transfer between local memory 1640 and memory 1610 of data processing unit 1630. For example, DMA engine 1620 can read data from memory 1610 and write the data to local memory 1640 of data processing unit 1630. As another example, DMA engine 1620 can read data from local memory 1640 of data processing unit 1630 and write the data to memory 1610. DMA engine 1620 provides DMA features that allow data processing unit 1630 to initiate data transfer between local memory 1640 and memory 1610 and to perform other operations while the data transfer is in progress. In some embodiments, DMA engine 1620 can read tensors from memory 1610 and modify the tensors in a manner optimized for data processing unit 1630 before writing the tensors to local memory 1640 of data processing unit 1630.

[0164] Data processing unit 1630 performs deep learning operations within a DNN. For example, data processing unit 1630 can execute a DNN layer by running one or more deep learning operations within the DNN layer. Data processing unit 1630 can execute one layer or a portion of a layer at a time. In some embodiments, the operations of the DNN layer can be run in parallel by multiple data processing units 1630. For example, multiple data processing units 1630 can each perform a data processing workload, or a portion of a data processing workload for deep learning operations. Data can be shared among data processing units 1630. Data processing unit 1630 may also be referred to as a computation block or computation slice.

[0165] Data processing unit 1630 may be capable of performing various types of deep learning operations, such as convolution, layer normalization, SoftMax operations, pooling, element-wise operations, linear operations, non-linear operations, matrix multiplication (MatMul), etc. Deep learning operations performed by data processing unit 1630 include tensor operations, i.e., operations whose input is a tensor or whose output is a tensor. In the example, data processing unit 1630 receives an input tensor and one or more convolutional kernels, and performs a convolution with the input tensor and the convolutional kernels. The result of the convolution can be an output tensor, which can be further computed, for example, by data processing unit 1630 or another data processing unit 1630.

[0166] exist Figure 16 In some embodiments, each data processing unit 1630 includes a local memory 1640, a loading module 1660, a processing engine 1670, a post-processing engine 1680, and an output module 1690. The data processing unit 1630 may include a data processing pipeline comprising the loading module 1660, processing engine 1670, post-processing engine 1680, and output module 1690. Some or all components of the data processing unit 1630 may be implemented on the same chip. In other embodiments, the data processing unit 1630 may include alternative configurations, different, or additional components. Furthermore, the functionality of the components belonging to the data processing unit 1630 may be implemented by different components included in the data processing unit 1630, different data processing units 1630, another component of the DNN accelerator 1602, or different systems. The components of the data processing unit 1630 may be implemented using hardware, software, firmware, or some combination thereof.

[0167] Local memory 1640 is located locally within the corresponding data processing unit 1630. Figure 16 In one embodiment, local memory 1640 is internal to data processing unit 1630. In other embodiments, local memory 1640 may be external to data processing unit 1630. Local memory 1640 may include one or more SRAMs. The capacity of local memory 1640 (e.g., 1.5-2 megabytes) may be much smaller than the capacity of memory 1610. Data in local memory 1640 may be transferred to or from memory 1610 or cache 1612, for example, via DMA engine 1620. In some embodiments, data in local memory 1640 may be transferred to or from the local memory 1640 of another data processing unit 1630. Local memory 1640 may store data received, used, or generated by loading module 1660, processing engine 1670, post-processing engine 1680, or output module 1690. Examples of data may include input activations, weights, output activations, low-level machine instructions, configuration descriptors, etc.

[0168] In some embodiments, local memory 1640 may store tensors to be processed by processing engine 1670 or post-processing engine 1680. The tensors may be input tensors for deep learning operations. Local memory 1640 may also store tensors generated by processing engine 1670 or post-processing engine 1680. The tensors may be output tensors for deep learning operations. The layout of the data points of the tensors in local memory 1640 may depend on the format in which the tensors are stored. In some embodiments, local memory 1640 may store tensors in various formats, including Z-major, X-major, and Y-major formats. For tensors with Z-major format, local memory 1640 may store data points with the same (x, y) coordinates contiguously. For example, data points with the same (x, y) coordinates may be stored in a sequence of memory addresses in local memory 1640. For tensors with ZXY or ZYX format, local memory 1640 may store data points with the same (x, y) coordinates contiguously. For example, data points with the same (x, y) coordinates may be stored in a sequence of memory addresses in local memory 1640. For tensors with an X-major format, local memory 1640 can continuously store data points with the same (y, z) coordinates. For tensors with a Y-major format, local memory 1640 can continuously store data points with the same (x, z) coordinates.

[0169] In some embodiments, local memory 1640 may store dense tensors (e.g., dense activation tensors, dense weight tensors, etc.), sparse tensors (e.g., sparse activation tensors, sparse weight tensors, etc.), and so on. A dense tensor may be a tensor in which zero-value elements (if any) have not been removed. A dense tensor can be converted into a sparse tensor by removing one or more zero-value elements from it. A sparse tensor may also be referred to as a compressed tensor or a packed tensor. The process of converting a dense tensor into a sparse tensor can be called sparse coding. Sparse coding can also generate sparse tensors. Each element in a sparse tensor may correspond to a different element in a dense tensor and indicate whether an element in the dense tensor is zero. A sparse tensor may indicate the position of an element in a dense tensor. A sparse tensor may include a sparse bitmap, where each element is a bit. A sparse tensor can be converted into a dense tensor by a densification process, in which one or more zeros can be added to the sparse tensor based on the sparse tensor.

[0170] In some embodiments, local memory 1640 includes one or more SRAMs. Local memory 1640 may be byte-addressable, and each memory address identifies a single byte (eight bits) stored. In some embodiments, local memory 1640 may include a memory bank. The number of data banks in local memory 1640 may be 16, 64, 128, 356, 512, 1624, 2048, or other numbers. A memory bank may include multiple memory cells. In the example, a data bank may include 8, 16, 64, or different numbers of memory cells. A memory bank or a memory cell within a memory bank may have a memory address. In the example, a memory cell may store a single byte, and data larger than a single byte may be stored in memory cells with contiguous memory addresses, i.e., adjacent memory cells. For example, a memory cell may store an integer in INT8 format, while storing a 16-bit number in FP16 or BF16 format may require two memory cells. In some embodiments, 16 bits may be transferred from local memory 1640 in a single read cycle. In other embodiments, 16 bits can be transferred from local memory 1640 in multiple read cycles (e.g., two cycles).

[0171] Loading module 1660 loads data from local memory 1640 into processing engine 1670 or post-processing engine 1680. Loading module 1660 can load data from local memory 1640 into one or more data buffers of processing engine 1670. Loading module 1660 can read tensors from local memory 1640. Tensors may include sparse activation tensors, sparse weight tensors, activation sparse tensors, weight sparse tensors, etc. In some embodiments, loading module 1660 can load data based on a sparsity pattern. Loading module 1660 can select different data to be delivered to processing engine 1670 in different sparsity patterns.

[0172] Processing engine 1670 performs neural network operations on the DNN. Figure 17 An exemplary processing engine 1670 is described and illustrated in the document.

[0173] Post-processing engine 1680 processes the output of processing engine 1670. Post-processing engine 1680 may include one or more post-processing elements. In some embodiments, the post-processing elements in post-processing engine 1680 may be arranged in an arrangement having rows and columns (e.g., an array arrangement). In some embodiments, post-processing engine 1680 computes an activation function. Post-processing engine 1680 may receive the output of processing engine 1670 as input to the activation function. In addition to or as an alternative to the activation function, post-processing engine 1680 may perform other types of post-processing on the output of processing engine 1670. For example, post-processing engine 1680 may apply a bias to the output of processing engine 1670. For example, post-processing engine 1680 may perform scaling on the output of processing engine 1670. In some embodiments, post-processing engine 1680 may be bypassed for certain neural network operations.

[0174] Output module 1690 evicts data from processing engine 1670 and / or post-processing engine 1680. Output module 1690 may write data to local memory 1640. The evicted data may be tensors, such as the output tensors of neural network operations. In some embodiments, output module 1690 may evict data at the unit level of processing engine 1670. For each processing unit, output module 1690 may evict the output of processing elements in the processing unit based on the row index or column index of each processing unit. For example, output module 1690 may evict data from the processing unit using a series of cycles. Output module 1690 may evict the output of some processing elements in each cycle. The order of the cycles may be configured based on configuration parameters indicating the operating mode of load module 1660. The evicted data, such as tensors, may be further loaded into memory 1610, for example, via DMA engine 1620. Additionally or alternatively, the evicted data may be loaded into processing engine 1670 by load module 1660 for further computation, such as for performing deep learning operations in the next layer.

[0175] Figure 17 A processing engine 1670 according to some embodiments of the present disclosure is shown. The processing engine 1670 may be included as part of a data processing unit, for example... Figure 16 The data processing unit 1630. The processing engine 1670 may include one or more processing units 1702. In some embodiments, the processing units 1702 may be arranged in one or more rows and / or one or more columns in the processing engine 1670. In some embodiments, the processing units 1702 may be arranged as one or more sets or arrays of processing units 1702 performing different operations. The processing engine 1670 may have one or more arrays of multiply-accumulate circuit modules (e.g., processing units 1702) optimized to perform computational operations such as MatMul and convolution.

[0176] Each processing unit (e.g., processing unit 1702) may include one or more processing elements. In some cases, a processing unit includes a single processing element. In some cases, a processing unit includes multiple processing elements. Processing elements may be arranged in an array. Processing elements may be arranged in rows and / or columns. In some cases, a processing unit may include one or more processing elements performing the same operation. In some cases, a processing unit may include one or more processing elements performing different operations. In some cases, at least some of the processing elements in a processing unit may be arranged to perform operations in parallel. In some cases, at least some of the processing elements in a processing unit may be arranged to perform operations serially.

[0177] Processing elements can perform arithmetic operations associated with neural network or DNN operations. In some cases, one or more processing elements can be arranged in an array comprising rows and columns. Examples of processing elements may include multiplication units, division units, scaling units, addition units, accumulator units, subtractor units, logarithm units, exponentiation units, multiply-accumulate (MAC) units, shift units, square root units, etc. Processing elements in a processing unit can be arranged to perform arithmetic operations on an input vector to generate an output vector (parallel processing), sometimes referred to as vector processing. Processing elements in a processing unit can also perform scalar operations.

[0178] Processing engine 1670 may include controller 1704, which may configure circuit modules of one or more processing units 1702 to perform arithmetic operations. In some cases, controller 1704 may configure one or more processing units 1702 (or individual processing elements within processing units 1702) to perform operations in a specific order or manner. In some cases, controller 1704 may configure one or more processing units 1702 (or individual processing elements within processing units 1702) based on instructions loaded in instruction buffer 1706 (e.g., configuration descriptors and / or low-level machine instructions). Controller 1704 may include a program counter to determine the instructions loaded into instruction buffer 1706 to be executed by one or more processing units 1702 (or individual processing elements within processing units 1702).

[0179] Instructions loaded in instruction buffer 1706 (e.g., configuration descriptors and / or low-level machine instructions) can signal which processing unit 1702 (or individual processing elements within processing unit 1702) is to execute or perform one or more operations. Instruction buffer 1706 may include one or more register files or one or more memory cell arrays.

[0180] Data can be generated by Figure 16 The controller 1704 and / or loading module 1660 loads data into the data buffer 1708. The data can then be used by the processing unit 1702. Data generated by the processing unit 1702 can be output from the data buffer 1708 to the output module 1690. Figure 16 The local memory 1640. The data buffer 1708 may include one or more register files, or one or more memory cell arrays.

[0181] Data buffer 1708 may include one or more of the following: one or more input data buffers and one or more output data buffers. Data buffer 1708 may include one or more weight / parameter buffers. Data buffer 1708 may store operands of one or more processing elements of processing unit 1702. Data buffer 1708 may store generated outputs of one or more processing elements of processing unit 1702.

[0182] Instructions loaded in instruction buffer 1706 (e.g., configuration descriptors and / or low-level machine instructions) can signal which data stored in data buffer 1708 will be processed by processing unit 1702 (or a separate processing element within processing unit 1702). In some cases, processing unit 1702 (or a separate processing element within processing unit 1702) can read data from data buffer 1708 at a default location of processing unit 1702 or a separate processing element within processing unit 1702.

[0183] After processing unit 1702 generates output data, instructions loaded into instruction buffer 1706 (e.g., configuration descriptors and / or low-level machine instructions) can signal where the output data will be stored in data buffer 1708. In some cases, processing unit 1702 (or a separate processing element within processing unit 1702) can write data to data buffer 1708 at a default location of processing unit 1702 or a separate processing element within processing unit 1702.

[0184] Figure 16 The loading module 1660 can load data into a specific location in the data buffer 1708. Figure 16 The output module 1690 can eject data from the data buffer 1708 for storage. Figure 16 In the local memory 1640 and / or memory 1610.

[0185] Figure 18A sparse processing unit 1802 according to some embodiments of the present disclosure is illustrated. The sparse processing unit 1802 illustrates an exemplary implementation of the processing unit 1702. In some embodiments, the processing engine 1670 may include sparsity acceleration logic for facilitating and supporting sparsity acceleration. For example, each processing unit 1702 in the processing engine 1670 may implement a component of the sparse processing unit 1802.

[0186] The sparse processing unit 1802 may include a sparsity controller 1804 and a MAC array 1810. The sparse processing unit 1802 may include a weight data buffer 1806 for storing weight data and an activation data buffer 1808 for storing input activation data. The sparse processing unit 1802 may include an accumulator store 1812 for storing accumulated data and generating output activation data. The sparsity controller 1804 may receive one or more of the following: weighted sparse data and input activation sparse data.

[0187] In some embodiments, the sparsity controller 1804 accelerates computation in the MAC array 1810 based on sparsity in activations, sparsity in weights, or both, to provide bilateral sparsity acceleration. The sparsity controller 1804 may include a storage unit for storing sparse tensors, which may be generated by... Figure 16 The loading module 1660 loads the data into the storage unit. The sparse tensor can be an activation sparse tensor, a weighted sparse tensor, or a combined sparse tensor.

[0188] An activation sparse tensor can be a sparse tensor of an activation tensor and has the same number of elements as the activation tensor. Elements in an activation sparse tensor can indicate whether the corresponding element in the activation tensor is zero. For example, a zero value in an activation sparse tensor indicates that the corresponding element in the activation tensor is zero. A one value in an activation sparse tensor indicates that the corresponding element in the activation tensor is non-zero.

[0189] A weighted sparse tensor is a sparse tensor of weight tensors and has the same number of elements as the weight tensor. Elements in a weighted sparse tensor indicate whether the corresponding element in the weight tensor is zero. For example, a zero value in a weighted sparse tensor indicates that the corresponding element in the weight tensor is zero. A one value in a weighted sparse tensor indicates that the corresponding element in the weight tensor is non-zero.

[0190] The sparsity controller 1804 can generate a combined sparse tensor using an activation sparse tensor and a weight sparse tensor. For example, the sparsity controller 1804 can multiply the elements of the activation sparse tensor with the corresponding elements of the weight sparse tensor to compute the elements of the combined sparse tensor. The positions of the three elements in their respective sparse tensors may match. In some embodiments, each element in the sparse tensor may be a bit, and the sparse tensor may be referred to as a sparse bitmap.

[0191] The sparsity controller 1804 can use sparse tensors to identify the activations and weights that will be used by the MAC unit in MAC operations. In an embodiment where the processing engine 1670 operates in combined sparsity mode, the sparsity controller 1804 can identify the activations and weights corresponding to the non-zero value elements of the combined sparse tensor. In an embodiment where the processing engine 1670 operates in activation sparsity mode, the sparsity controller 1804 can identify the activations and weights corresponding to the non-zero value elements of the activation sparse tensor. In an embodiment where the MAC array 1810 operates in weighted sparsity mode, the sparsity controller 1804 can identify the activations and weights corresponding to the non-zero value elements of the weighted sparse tensor. The sparsity module can be bypassed in dense mode because no sparsity acceleration is performed.

[0192] Figure 19 Sparse computation in a sparse processing unit 1802 according to some embodiments of the present disclosure is illustrated. The sparse processing element 1900 may be a unit component of a processing unit, such as processing unit 1702 in processing engine 1670, or... Figure 18 The sparse processing unit 1802. In other words, the processing unit may have a grid or array of sparse processing elements, wherein an instance is shown as sparse processing element 1900.

[0193] exist Figure 19 In one embodiment, the sparse processing element 1900 includes a MAC unit 1905, an activation register file 1910, a weight register file 1920, an output register file 1950, and a sparsity accelerator 1960. The MAC unit 1905 includes a multiplier 1930 and an adder 1940. In other embodiments, the sparse processing element 1900 may include fewer, more, or different components.

[0194] Activation register file 1910 stores the activation operand. Activation register file 1910 can be... Figure 18 The activation data buffer 1808 is a part of the weight register file 1920, which stores the weight operands. The weight register file 1920 may be a part of the weight data buffer 1806. The activation operands and weight operands can be retrieved from memory (e.g., from...). Figure 16 The memory (1640) is loaded into the activation register file 1910 and the weight register file 1920, respectively.

[0195] The sparsity accelerator 1960 receives a sparse bitmap 1915 corresponding to the sparse tensors in the weight register file 1920. The sparse bitmap 1915 can be obtained from... Figure 18The sparsity controller 1804 generates the sparse bitmap. When the MAC unit 1905 operates in combined sparsity mode, the sparse bitmap 1915 can be a combined sparse bitmap. When the MAC unit 1905 operates in active sparsity mode, the sparse bitmap 1915 can be an active sparse bitmap. When the MAC unit 1905 operates in weighted sparsity mode, the sparse bitmap 1915 can be a weighted sparse bitmap. The sparse bitmap 1915 can have the same size (e.g., the same number of elements) as the active operand or the weight operand, or a larger size.

[0196] Using sparse bitmap 1915, sparsity accelerator 1960 selects, for example, four activations from activation register file 1910 and four weights from weight register file 1920. Sparsity accelerator 1960 transmits the selected activations and weights to multiplier 1930. These selected data elements correspond to non-zero value elements of sparse bitmap 1915. The four selected activations and four selected weights can form four activation-weight pairs. Multiplier 1930 can compute a product based on each activation-weight pair, and thus compute a total of four products. These four products can be provided to adder 1940. Although... Figure 19 A single multiplier 1930 is shown, but the MAC unit 1905 may include multiple multipliers capable of performing multiple multiplication operations simultaneously.

[0197] Adder 1940 accumulates the four products and computes the cell-level internal partial sum. The four unselected elements of the dense tensor are not processed to save power and time, which will not affect the value of the cell-level internal partial sum. For example, when the dense tensor is a dense activation tensor, the weights corresponding to the unselected activations are zero, so the product of the unselected activations and weights will all be zero and will not contribute to the cell-level internal sum or other partial sums computed by the sparse unit. Similarly, when the dense tensor is a dense weight tensor, the activations corresponding to the unselected weights are zero, so the product of the unselected weights and activations will all be zero and will not contribute to the cell-level internal sum or other partial sums computed by the sparse unit. In other embodiments, MAC unit 1905 can operate in dense mode, where sparse bitmap 1915 is not used and sparsity accelerator 1960 is inactive. MAC unit 1905 can process all activations in the activation operand and all weights in the weight operand.

[0198] The cell-level internal partial sums can be stored in the output register file 1950. In some embodiments, the cell-level internal partial sums can be used multiple times. For example, the activation operand can represent N data blocks in the input tensor of the convolution, where N is an integer greater than 1. Instead of processing all N data blocks to compute N cell-level internal partial sums, the cell-level internal partial sums are computed once and used as N cell-level internal partial sums N times in the convolutional layer.

[0199] In some embodiments, sparse processing element 1900 receives one or more internal portions of a processing element level from one or more other processing elements of the processing unit. Adder 1940 or accumulator ( Figure 19 (Not shown) One or more processing element-level internal sums can be accumulated with the processing element-level internal sum of sparse processing element 1900, and the accumulated result (i.e., multi-processing element internal sum) is stored in output register file 1950. One or more other processing elements in the processing unit with a MAC array can be in the same column as sparse processing element 1900 in the sparse processing unit. Multi-cell internal sums can be column-level internal sums. In some embodiments, the processing element-level internal sum or multi-cell internal sum of sparse processing element 1900 can be sent to one or more other processing elements in the processing unit for further accumulation.

[0200] Figure 20 A DNN module 1601 according to some embodiments of the present disclosure is illustrated. The DNN module 1601 includes an interface module 2010, a training module 2020, a validation module 2040, a compiler 2050, and a data storage 2060. In other embodiments, the DNN module 1601 may include alternative configurations, different or additional components. Furthermore, the functionality of the components belonging to the DNN module 1601 may be performed by different components, different modules, or systems included in the DNN module 1601.

[0201] Interface module 2010 facilitates communication between DNN module 1601 and other modules or systems. For example, interface module 2010 establishes communication between DNN module 1601 and external data storage to receive data that can be used to train the DNN or input into the DNN to perform tasks. As another example, interface module 2010 supports DNN module 1601 in distributing the DNN to other systems, such as computing devices configured to apply the DNN to perform tasks.

[0202] Training module 2020 trains the DNN using a training dataset. Training module 2020 forms the training dataset. In an example where training module 2020 trains the DNN to recognize objects in images, the training dataset includes training images and training labels. The training labels describe the ground truth classification of objects in the training images. In some embodiments, each label in the training dataset corresponds to an object in a training image. In an example where training module 2020 trains a transformer-based neural network to predict the next token, the training dataset may include a large library of token sequences. In some embodiments, a portion of the training dataset can be used for initial training of the DNN, while the remainder of the training dataset can be reserved as a validation subset used by validation module 2040 to validate the performance of the trained DNN. The portion of the training dataset excluding the tuning and validation subsets can be used to train the DNN.

[0203] The Training Module 2020 also determines the hyperparameters used to train the DNN. Hyperparameters are variables that specify the DNN training process. Hyperparameters are different from parameters inside the DNN (e.g., filter weights). In some embodiments, hyperparameters include variables that determine the DNN architecture, such as the number of hidden layers. Hyperparameters also include variables that determine how the DNN is trained, such as batch size, number of epochs, etc. Batch size defines the number of training samples to be processed before updating the DNN parameters. The batch size is equal to or less than the number of samples in the training dataset. The training dataset can be divided into one or more batches. The number of epochs defines the number of times the entire training dataset is traversed throughout the entire network. The number of epochs defines the number of times the deep learning algorithm works across the entire training dataset. An epoch means that each training sample in the training dataset has a chance to update the parameters inside the DNN. An epoch can include one or more batches. The number of epochs can be 1, 5, 10, 50, 100, 500, 1600, or even larger.

[0204] The training module 2020 can define the architecture of the DNN, for example, based on some hyperparameters. In some cases, the training module 2020 can receive a model definition that defines or specifies the architecture of the DNN. The architecture of the DNN may include multiple layers. Examples of layers may include convolutional layers, pooling layers, fully connected layers, normalized layers, SoftMax or logit layers, etc. After the training module 2020 defines the architecture of the DNN, it feeds a training dataset into the DNN. The training dataset includes multiple training samples. The training module 2020 modifies the parameters internal to the DNN (“internal parameters of the DNN”) to minimize the error between the labels of the training objects generated by the DNN and the ground truth labels of the objects. The internal parameters include the weights used in the DNN layers. In some embodiments, the training module 2020 uses a cost function to minimize the error.

[0205] The training module 2020 can train the DNN for a predetermined number of epochs. The number of epochs is a hyperparameter that defines how many times the deep learning algorithm will work across the entire training dataset. An epoch means that each sample in the training dataset has a chance to update the internal parameters of the DNN. After the training module 2020 has completed the predetermined number of epochs, it can stop updating the parameters in the DNN. The DNN with updated parameters is called the trained DNN.

[0206] The validation module 2040 validates the accuracy of the trained DNN. In some embodiments, the validation module 2040 inputs samples from the validation dataset into the trained DNN and uses the DNN's output to determine model accuracy. In some embodiments, the validation dataset may consist of some or all of the samples from the training dataset. Additionally or alternatively, the validation dataset may include additional samples in addition to those in the training set. In some embodiments, the validation module 2040 may determine an accuracy score that measures the precision, recall, or a combination of precision and recall of the DNN. The validation module 2040 may use the following metrics to determine the precision score: precision = TP / (TP + FP) and recall = TP / (TP + FN), where precision may be how many of the total number of predictions (TP + FP or false positives) the DNN correctly predicts (TP or true positives), and recall may be how many of the total number of objects having said attribute (TP + FN or false negatives) the DNN correctly predicts (TP). F-score (F-score = 2 * PR / (P + R)) unifies precision and recall into a single metric.

[0207] The verification module 2040 can compare the accuracy score with a threshold score. In an example where the verification module 2040 determines that the accuracy score of the DNN is less than the threshold score, the verification module 2040 instructs the training module 2020 to retrain the DNN. In one embodiment, the training module 2020 can iteratively retrain the DNN until a stopping condition is met, such as an indication that the DNN may be sufficiently accurate, or that multiple training rounds have already occurred.

[0208] Compiler 2050 compiles information associated with the DNN, which can be used to enable or configure it. Figure 16The DNN accelerator 1602 executes the neural network operations of the DNN. This information may include a model definition, one or more processing graphs, one or more data processing workloads generated from the one or more processing graphs, and executable instructions (e.g., workload descriptors, configuration descriptors, and / or low-level machine instructions) that can be executed by the DNN accelerator 1602. The model definition may include one or more neural network operations to be performed by the DNN. In some embodiments, the compiler 2050 may generate a processing graph representing the DNN. The graph may include nodes and edges. Nodes may represent specific neural network operations in the DNN. Edges may connect two nodes and represent a connection between two corresponding neural network operations. In the example, an edge may encode a tensor flowing from one neural network operation to another. The tensor may be the output tensor of the first neural network operation and the input tensor of the second neural network operation. Edges may encode one or more properties of the tensor, such as size, shape, storage format, etc.

[0209] The compiler 2050 can preprocess and / or modify the processing graph to identify opportunities to streamline it, thereby reducing the overhead of compiled configuration descriptors. Due to the specific nature of the data processing pipeline in the DPU, the compiler 2050 can follow a set of rules or patterns when generating one or more configuration descriptors for one or more nodes of the processing graph. The compiler 2050 can traverse the processing graph to generate configuration descriptors according to this set of rules or patterns. Components of the DNN accelerator 1602 (e.g., Figure 16 The processing engine 1670 and post-processing engine 1680 can use configuration descriptors to execute DNNs.

[0210] Data storage 2060 stores received, generated, used, or associated data with DNN module 1601. For example, data storage 2060 stores datasets used by training module 2020 and validation module 2040. Data storage 2060 may also store data generated by training module 2020 and validation module 2040, such as hyperparameters used to train the DNN, intrinsic parameters of the trained DNN (e.g., weights), data used for sparsity acceleration (e.g., sparse bitmaps), etc. Data storage 2060 may store configuration parameters, configuration descriptors, instructions generated by compiler 2050, etc. Data storage 2060 may include one or more memories. Figure 20 In one embodiment, data storage 2060 is a component of DNN module 1601. In other embodiments, data storage 2060 may be external to DNN module 1601 and may communicate with DNN module 1601 via a network.

[0211] Figure 21A compiler 2050 according to some embodiments of the present disclosure is shown. The compiler 2050 includes a neural network analyzer 2102, quantization 2120, layer selection 2130, weight reordering 2140, a configuration descriptor generator 2108, and a scheduler 2110. The compiler 2050 can perform... Figure 6 The model compilation process is 600. Compiler 2050 can execute... Figure 14 Algorithm 1400. Compiler 2050 can execute it. Figure 15 The hierarchical weight reordering process 1500. Compiler 2050 can execute it. Figure 22 Method 2200.

[0212] Neural network analyzer 2102 can analyze a DNN (e.g., a model definition) and determine how a neural network hardware accelerator can utilize the components of the DNN accelerator to implement the DNN. Neural network analyzer 2102 can receive a neural network model definition for the DNN. The neural network model definition can specify one or more layers of the neural network. For example, the neural network model definition can specify the layers of the neural network and how data should flow through these layers. A layer can be specified by the neural network operations performed by that layer. Examples of layers may include fully connected (linear) layers, convolutional layers, recurrent layers, long short-term memory networks, gated recurrent unit layers, max pooling layers, average pooling layers, batch normalization layers, normalization layers, exit layers, activation layers, embedding layers, etc. The layer can be specified by one or more of the following: input size, hidden size, output size, etc. The layer can be specified by one or more parameters of the neural network operations (e.g., for a convolutional layer, one or more parameters may include kernel size, padding, stride, etc.).

[0213] In some cases, the neural network analyzer 2102 can determine the processing graph based on the neural network model definition. The processing graph can include connected nodes. Connected nodes can represent neural network operations to be performed by one or more data processing units or other components of the DNN accelerator, and the order in which these neural network operations are performed. Edges connecting nodes can represent data flow between neural network operations. Examples of neural network operations can include: computational operations, convolution, filtering, pooling, arithmetic, matrix multiplication, applying activation functions (clamping functions, exponential functions, sigmoid functions, power functions, square root functions, etc.), and so on. An edge connecting a node to another node following that node can indicate that the output generated by that node will be provided as input to that other node. The processing graph can include one or more neural network operations to be performed by one or more DPUs (of the neural network hardware accelerator).

[0214] The neural network analyzer 2102 can determine one or more data processing workloads to be executed by the DPU (of the neural network hardware accelerator) based on a processing graph and / or a neural network model definition. One or more data processing workloads may correspond to and / or include one or more neural network operations within the processing graph. For example, a data processing workload may include one or more neural network operations executed according to one or more configurations. Data processing workloads can be executed by a data processing pipeline of a data processing unit, such as a processing engine or a post-processing engine of the data processing unit. In some cases, neural network operations can be transformed into data processing workloads. In some cases, neural network operations can be transformed into multiple data processing workloads. In some cases, one or more neural network operations can be transformed into one or more data processing workloads. Neural network operations can be executed by a data processing pipeline of a data processing unit, such as a processing engine or a post-processing engine of the data processing unit. Neural network operations can be executed by one or more data processing units or one or more parts of a data processing unit according to the execution order represented by the processing graph. The neural network operations of a data processing workload can be executed by a data processing unit according to one or more configurations of the neural network operations. In some cases, the neural network model definition includes a processing graph.

[0215] The neural network analyzer 2102 can perform Figure 6 604.

[0216] Quantitative 2120 can be executed Figure 6 608.

[0217] Layer selection 2130 can be executed. Figure 14 Algorithm 1400 is used to determine the selected layer.

[0218] Weight reordering 2140 can be performed on each selected layer. Figure 15 Process 1502.

[0219] Layer selection 2130 and weight reordering 2140 are executed together. Figure 6 610.

[0220] The configuration descriptor generator 2108 can generate one or more instructions (e.g., one or more configuration descriptors, one or more workload descriptors, low-level machine instructions, one or more machine-readable configurations, etc.) based on a data processing graph.

[0221] Scheduler 2110 coordinates when and which DPUs should load configuration descriptors generated by configuration descriptor generator 2108 to execute data processing workloads. Data processing workloads can be allocated to DPUs (in neural network hardware accelerators) by scheduler 2110, and scheduler 2110 can coordinate the provision of appropriate configuration descriptors (or one or more portions of configuration descriptors) to the DPUs. In some cases, scheduler 2110 can determine a plan capable of load balancing the execution of data processing workloads. Scheduler 2110 can determine a plan to ensure that data processing workloads are executed according to the processing graph. Scheduler 2110 can determine a plan to load configuration descriptors or portions of configuration descriptors onto the DPUs at appropriate times. In some cases, scheduler 2110 can be part of compiler 2050. In some cases, scheduler 2110 can be... Figure 16 and 21 It is part of the DNN module 1601.

[0222] The configuration descriptor generator 2108 and the scheduler 2110 can be executed together. Figure 6 The numbers 616, 618, 620, and 622.

[0223] Refer back Figure 16-21 The DNN system 1600 illustrates one implementation of a processing system designed to accelerate DNN execution. The architecture of the DNN accelerator 1602 can vary depending on the application requirements of the processor. The architecture can vary based on the number of DPUs, the number of processing engines, the number of processing units, the number of post-processing engines, the structure of the data processing pipeline in the DPU, support for vector processing, support for sparse patterns, the type or set of processing elements, the amount of memory, and buffer size, among other factors. The underlying hardware implementation of the process can include other computing techniques, such as in-memory computing (including analog in-memory computing and digital in-memory computing).

[0224] Exemplary methods of compilers Figure 22 A flowchart illustrating a method 2200 executable by a compiler according to some embodiments of the present disclosure is shown. Method 2200 can be executed by a compiler such as compiler 2050 to compile a neural network.

[0225] In 2202, the compiler receives a model definition for a neural network model that includes multiple layers. The multiple layers have a single layer with multiple weights applied to the activations of multiple inputs, for example, using an array of processing elements from a neural network accelerator.

[0226] In 2204, the compiler determines the order of multiple weights based on the switching activity metric.

[0227] In 2206, the compiler determines multiple rearranged weights by sorting multiple weights according to their order.

[0228] In 2208, the compiler generates one or more machine-readable configurations to configure a neural network accelerator to apply multiple rearranged weights to multiple input activations. The compiler can generate one or more machine-readable configurations for a neural network accelerator to load multiple rearranged weights according to their order and apply these rearranged weights to multiple input activations.

[0229] In some cases, multiple rearranged weights are loaded onto the processing element array of the neural network accelerator according to a sorting order to reduce switching activity in the circuit modules of the processing element array. In some embodiments, the processing element array is a MAC array.

[0230] Exemplary computing device Figure 23 This is a block diagram of an apparatus or system (e.g., an exemplary computing device 2300) according to some embodiments of the present disclosure. One or more computing devices 2300 may be used to implement the functions described herein in conjunction with the accompanying drawings. Figure 23 The various components shown may be included in computing device 2300, but any one or more of these components may be omitted or duplicated to suit the application. In some embodiments, some or all of the components included in computing device 2300 may be attached to one or more motherboards. In some embodiments, some or all of these components are manufactured onto a single system-on-a-chip (SoC) die. Additionally, in various embodiments, computing device 2300 may not include... Figure 23 The computing device 2300 may include one or more components as shown, and may include interface circuit modules for coupling to one or more components. For example, the computing device 2300 may not include the display device 2306, but may include display device interface circuit modules (e.g., connectors and driver circuitry) to which the display device 2306 may be coupled. In another set of examples, the computing device 2300 may not include the audio input device 2318 or the audio output device 2308, but may include audio input or output device interface circuit modules (e.g., connectors and support circuitry modules) to which the audio input device 2318 or the audio output device 2308 may be coupled.

[0231] Computing device 2300 may include processing device 2302 (e.g., one or more processing devices, one or more processing devices of the same type, or one or more processing devices of different types). Processing device 2302 may include an electronic circuit module that processes electronic data from data storage elements (e.g., registers, memory, resistors, capacitors, qubit units) to convert the electronic data into other electronic data that can be stored in registers and / or memory. Examples of processing device 2302 may include CPU, GPU, quantum processor, machine learning processor, AI processor, neural network processor, AI accelerator, application-specific integrated circuit (ASIC), analog signal processor, analog computer, microprocessor, digital signal processor, field-programmable gate array (FPGA), tensor processing unit (TPU), neural network hardware accelerator, DNN accelerator (e.g., such as...). Figure 16-19 The DNN accelerator 1602 and NPU shown are examples of this.

[0232] Computing device 2300 may include memory 2304, which itself may include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high-bandwidth memory (HBM), flash memory, solid-state memory, and / or hard disk drive. Memory 2304 includes one or more non-transitory computer-readable storage media. In some embodiments, memory 2304 may include memory that shares a die with processing device 2302.

[0233] In some embodiments, memory 2304 includes one or more non-transitory computer-readable media storing instructions executable to perform operations described herein in conjunction with the accompanying drawings. Exemplary components, such as DNN module 1601 and compiler 2050, are depicted and can be encoded as instructions and stored in memory 2304. Memory 2304 may store instructions encoding one or more exemplary portions, such as DNN module 1601, one or more portions of DNN module 1601, compiler 2050, and one or more portions of compiler 2050. Instructions stored in one or more non-transitory computer-readable media may be executed by processing device 2302. Memory 2304 may store instructions causing processing device 2302 to perform one or more methods, procedures, or algorithms described and illustrated herein, such as... Figure 6 The model compilation process 600 Figure 14 Algorithm 1400 Figure 15 The hierarchical weight reordering process 1500 and Figure 22 Method 2200.

[0234] In some embodiments, memory 2304 may store data, such as data structures, binary data, bits, metadata, files, blobs, etc., as shown in the accompanying drawings and described herein. Memory 2304 may store data related to… Figure 6 The model compilation process 600 Figure 14 Algorithm 1400 Figure 15 The hierarchical weight reordering process 1500 and Figure 22 Method 2200 related data.

[0235] In some embodiments, memory 2304 may store one or more DNNs (and / or portions thereof). Memory 2304 may store training data used to train (trained) DNNs. Memory 2304 may store instructions for performing operations associated with training the DNNs. Memory 2304 may store input data, output data, intermediate outputs, and intermediate inputs of one or more DNNs. Memory 2304 may store one or more parameters used by one or more DNNs. Memory 2304 may store information encoding how the nodes of one or more DNNs are interconnected. Memory 2304 may store instructions to perform one or more operations of one or more DNNs. Memory 2304 may store model definitions specifying one or more operations of the DNNs. Memory 2304 may store instructions generated by a compiler based on the model definitions, such as configuration descriptors or model blobes.

[0236] In some embodiments, computing device 2300 may include communication device 2312 (e.g., one or more communication devices). For example, communication device 2312 may be configured to manage wired and / or wireless communications to transmit data to and from computing device 2300. The term “wireless” and its derivatives may be used to describe circuits, apparatuses, systems, methods, technologies, communication channels, etc., which can transmit data using modulated electromagnetic radiation over a non-solid medium. This term does not imply that the associated apparatus does not contain any wiring, although in some embodiments they may not contain any wiring. Communication device 2312 may implement any of a variety of wireless standards or protocols, including but not limited to Institute of Electrical and Electronics Engineers (IEEE) standards, including Wi-Fi (IEEE 1702.10 series), IEEE 1702.16 standards (e.g., IEEE 1702.16-2005 amendments), Long Term Evolution (LTE) projects, and any amendments, updates, and / or revisions (e.g., Advanced LTE projects, Ultra Mobile Broadband (UMB) projects (also known as “3GPP2”), etc.). Broadband wireless access (BWA) networks compliant with IEEE 1702.16 are commonly referred to as WiMAX networks. This acronym stands for WiMAX, an acronym for global interoperability of microwave access, and is a certification mark for products that have passed conformance and interoperability testing according to the IEEE 1702.16 standard. Communication device 2312 can operate according to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. Communication device 2312 can operate according to Enhanced Data GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). Communication device 2312 can operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolved Data Optimization (EV-DO) and its derivative protocols, as well as any other wireless protocols designated as 3G, 4G, 5G, and above. In other embodiments, the communication device 2312 may operate according to other wireless protocols. The computing device 2300 may include an antenna 2322 to facilitate wireless communication and / or receiving other wireless communications (e.g., radio frequency transmissions). The computing device 2300 may include receiver circuitry and / or transmitter circuitry. In some embodiments, the communication device 2312 may manage wired communication, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication device 2312 may include multiple communication chips.For example, the first communication device 2312 may be dedicated to short-range wireless communication such as Wi-Fi or Bluetooth, while the second communication device 2312 may be dedicated to long-range wireless communication such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, etc. In some embodiments, the first communication device 2312 may be dedicated to wireless communication, and the second communication device 2312 may be dedicated to wired communication.

[0237] The computing device 2300 may include a power supply / power circuit module 2314. The power supply / power circuit module 2314 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuit modules for coupling components of the computing device 2300 to an energy source (e.g., DC power, AC power, etc.) that is separate from the computing device 2300.

[0238] The computing device 2300 may include a display device 2306 (or a corresponding interface circuit module, as described above). The display device 2306 may include any visual indicator, such as a head-up display, computer monitor, projector, touch screen display, liquid crystal display (LCD), light-emitting diode display, or flat panel display.

[0239] The computing device 2300 may include an audio output device 2308 (or a corresponding interface circuit module, as described above). The audio output device 2308 may include any device that generates auditory indicators, such as a speaker, headphones, or earphones.

[0240] The computing device 2300 may include an audio input device 2318 (or a corresponding interface circuit module, as described above). The audio input device 2318 may include any device that generates a signal representing sound, such as a microphone, microphone array, or digital instrument (e.g., an instrument with a Musical Instrument Digital Interface (MIDI) output).

[0241] Computing device 2300 may include GPS device 2316 (or a corresponding interface circuit module, as described above). As is known in the art, GPS device 2316 can communicate with satellite-based systems and can receive the location of computing device 2300. The computing device 2300 may include a sensor 2330 (or one or more sensors). The computing device 2300 may include corresponding interface circuit modules (as described above). The sensor 2330 can sense physical phenomena and convert them into electrical signals that can be processed by, for example, the processing device 2302. Examples of sensors 2330 may include: capacitive sensors, inductive sensors, resistive sensors, electromagnetic field sensors, light sensors, cameras, imagers, microphones, pressure sensors, temperature sensors, vibration sensors, accelerometers, gyroscopes, strain sensors, humidity sensors, distance sensors, time-of-flight sensors, pH sensors, particle sensors, air quality sensors, chemical sensors, gas sensors, biosensors, ultrasonic sensors, scanners, etc.

[0242] The computing device 2300 may include another output device 2310 (or a corresponding interface circuit module, as described above). Examples of other output devices 2310 may include audio codecs, video codecs, printers, wired or wireless transmitters for providing information to other devices, haptic output devices, gas output devices, vibration output devices, lighting output devices, home automation controllers, or additional storage devices.

[0243] The computing device 2300 may include another input device 2320 (or a corresponding interface circuit module, as described above). Examples of other input devices 2320 may include an accelerometer, gyroscope, compass, image capture device, keyboard, cursor control device such as mouse, stylus, touchpad, barcode reader, quick response (QR) code reader, any sensor, or radio frequency identification (RFID) reader.

[0244] The computing device 2300 can have any desired form factor, such as a handheld or mobile computer system (e.g., a cellular phone, smartphone, mobile internet device, music player, tablet computer, laptop computer, netbook computer, personal digital assistant (PDA), personal computer, remote control, wearable device, helmet, glasses, footwear, electronic clothing, etc.), desktop computer system, server or other networked computing component, printer, scanner, monitor, set-top box, entertainment control unit, vehicle control unit, digital camera, digital video recorder, Internet of Things device, or wearable computer system. In some embodiments, the computing device 2300 can be any other electronic device that processes data.

[0245] Select Example Example 1 provides an apparatus for compiling a neural network model to be executed on a neural network accelerator, comprising: a processor; and a memory for storing instructions that, when executed by the processor, cause the processor to: receive a model definition of the neural network model comprising multiple layers, the multiple layers including layers having multiple weights to be applied to multiple activations; determine an ordering of the multiple weights based on a switching activity metric; determine multiple rearranged weights by arranging the multiple weights according to the ordering; and generate one or more machine-readable configurations for the neural network accelerator to load the multiple rearranged weights according to the ordering and apply the multiple rearranged weights to the multiple activations.

[0246] Example 2 provides the device of Example 1, wherein the neural network accelerator includes an array of processing elements for applying the plurality of weights to the plurality of activations, and the array of processing elements is a multiplicative accumulation array.

[0247] Example 3 provides a device from Example 1 or 2, wherein the switching activity metric between a pair of weights among the plurality of weights is the Hamming distance.

[0248] Example 4 provides a device for any of Examples 1-3, wherein the instructions cause the processor to determine the order of the plurality of weights by: selecting one of the plurality of weights as the current hub; selecting the next weight with the minimum switching activity metric for the current hub from the order of the plurality of weights; and updating the current hub to the next weight.

[0249] Example 5 provides a device for any of Examples 1-4, wherein the instructions further cause the processor to: select a subset of the plurality of layers based on one or more of a switching activity score for each layer and a number of weights for each layer, the switching activity score quantizing the number of bit transitions for a given layer.

[0250] Example 6 provides a device of any one of Examples 1-5, wherein the instructions further cause the processor to: select a subset of the layers among the plurality of layers under the constraint of selecting only non-contiguous layers.

[0251] Example 7 provides a device of any one of Examples 1-6, wherein the instructions further cause the processor to: select a subset of the layers by traversing the plurality of layers and comparing the cumulative switching cost when the current layer is skipped with the additional cumulative switching cost when the current layer is selected.

[0252] Example 8 provides a device of any one of Examples 1-7, wherein the instructions cause the processor to determine the order of the plurality of weights in such a way that determining the order of the plurality of weights reduces switching activity between rows of the plurality of weights corresponding to the plurality of input channels of the layer.

[0253] Example 9 provides a device of any one of Examples 1-8, wherein the instructions cause the processor to determine the order of the plurality of weights in such a way that determining the order of the plurality of weights reduces switching activity between the last row of the rows of the plurality of weights and the first row of the additional rows of the plurality of other weights of the layer, the rows corresponding to the plurality of input channels of the layer, and the additional rows corresponding to the plurality of additional input channels of the layer.

[0254] Example 10 provides a device from any of Examples 1-9, wherein the plurality of weights are quantized.

[0255] Example 11 provides one or more non-transitory computer-readable media storing instructions for compiling a neural network model to be executed on a neural network accelerator, which, when executed by a processor, cause the processor to: receive a model definition of the neural network model comprising multiple layers, the multiple layers including layers having multiple weights to be applied to multiple activations; determine an ordering of the multiple weights based on a switching activity metric; determine multiple rearranged weights by arranging the multiple weights according to the ordering; and generate one or more machine-readable configurations for the neural network accelerator to load the multiple rearranged weights according to the ordering and apply the multiple rearranged weights to the multiple activations.

[0256] Example 12 provides one or more non-transitory computer-readable media of Example 11, wherein the neural network accelerator includes an array of processing elements for applying the plurality of weights to the plurality of activations, and the array of processing elements is a multiplicative accumulation array.

[0257] Example 13 provides one or more non-transitory computer-readable media of Example 11 or 12, wherein the switching activity metric between a pair of weights among the plurality of weights is the Hamming distance.

[0258] Example 14 provides one or more non-transitory computer-readable media of any one of Examples 11-13, wherein the instructions cause the processor to determine the order of the plurality of weights by: selecting one of the plurality of weights as the current pivot; selecting the next weight with the minimum switching activity metric for the current pivot in the order of the plurality of weights; and updating the current pivot to the next weight.

[0259] Example 15 provides one or more non-transitory computer-readable media of any one of Examples 11-14, wherein the instructions further cause the processor to: select a subset of the plurality of layers based on one or more of a switching activity score for each layer and a number of weights for each layer, the switching activity score quantifying the number of bit transitions for a given layer.

[0260] Example 16 provides one or more non-transitory computer-readable media of any one of Examples 11-15, wherein the instructions further cause the processor to: select a subset of the layers among the plurality of layers under the constraint of selecting only non-contiguous layers.

[0261] Example 17 provides one or more non-transitory computer-readable media of any one of Examples 11-16, wherein the instructions further cause the processor to: select a subset of the layers by traversing the plurality of layers and comparing the cumulative switching cost when the current layer is skipped with an additional cumulative switching cost when the current layer is selected.

[0262] Example 18 provides one or more non-transitory computer-readable media of any of Examples 11-17, wherein the instructions cause the processor to determine the ordering of the plurality of weights in such a way that determining the ordering of the plurality of weights reduces switching activity between rows of the plurality of weights corresponding to the plurality of input channels of the layer.

[0263] Example 19 provides one or more non-transitory computer-readable media of any one of Examples 11-18, wherein the instructions cause the processor to determine the ordering of the plurality of weights in such a way that determining the ordering of the plurality of weights reduces switching activity between the last row of a row of the plurality of weights and the first row of another row of a plurality of other weights of the layer, the row corresponding to a plurality of input channels of the layer, and the other rows corresponding to a plurality of other input channels of the layer.

[0264] Example 20 provides one or more non-transitory computer-readable media of any of Examples 11-19, wherein the plurality of weights are quantized.

[0265] Example 21 provides a method for compiling a neural network model to be executed on a neural network accelerator, comprising: receiving a model definition of the neural network model including multiple layers, the multiple layers including layers having multiple weights to be applied to multiple activations; determining an order of the multiple weights based on a switching activity metric; determining multiple rearranged weights by arranging the multiple weights according to the order; and generating one or more machine-readable configurations for the neural network accelerator to load the multiple rearranged weights according to the order and apply the multiple rearranged weights to the multiple activations.

[0266] Example 22 provides the method of Example 21, wherein the neural network accelerator includes an array of processing elements for applying the plurality of weights to the plurality of activations, and the array of processing elements is a multiplicative accumulation array.

[0267] Example 23 provides the method of Example 21 or 22, wherein the switching activity metric between a pair of weights among the plurality of weights is the Hamming distance.

[0268] Example 24 provides a method for any of Examples 21-23, wherein determining the order of the plurality of weights includes: selecting one of the plurality of weights as the current pivot; selecting the next weight with the minimum switching activity metric for the current pivot from the order of the plurality of weights; and updating the current pivot to the next weight.

[0269] Example 25 provides a method of any of Examples 21-24, further comprising selecting a subset of the plurality of layers based on one or more of a switching activity score for each layer and a number of weights for each layer, wherein the switching activity score quantifies the number of bit transitions for a given layer.

[0270] Example 26 provides a method of any of Examples 21-25, and further includes selecting a subset of the layers among the plurality of layers under the constraint of selecting only non-continuous layers.

[0271] Example 27 provides a method from any of Examples 21-26, further comprising selecting a subset of the multiple layers by traversing the multiple layers and comparing the cumulative switching cost when the current layer is skipped with an additional cumulative switching cost when the current layer is selected.

[0272] Example 28 provides a method of any of Examples 21-27, wherein determining the order of the plurality of weights includes: determining the order of the plurality of weights, which reduces switching activity between rows of the plurality of weights corresponding to the plurality of input channels of the layer.

[0273] Example 29 provides a method of any one of Examples 21-28, wherein determining the order of the plurality of weights includes: determining the order of the plurality of weights such that it reduces switching activity between the last row of a row of the plurality of weights and the first row of another row of a plurality of other weights of the layer, the row corresponding to a plurality of input channels of the layer, and the other row corresponding to a plurality of other input channels of the layer.

[0274] Example 30 provides a method from any of Examples 21-29, wherein the plurality of weights are quantized.

[0275] Example 31 provides an apparatus including components for performing a method according to any one of Examples 21-30.

[0276] Example 32 provides a computer program product including instructions that, when executed by a processor, cause the processor to perform a method according to any one of Examples 21-30.

[0277] Example 33 provides a machine-readable storage containing machine-readable instructions that, when executed, cause a computer to implement a method according to any one of Examples 21-30.

[0278] Example 34 provides a computer program including instructions that, when executed by a processing device, cause the processing device to perform a method according to any one of Examples 21-30.

[0279] Example 35 provides a computer-implemented system including one or more processors and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform a method according to any one of Examples 21-30.

[0280] Variations and other notes Although the operations of the example methods shown in the figures and described with reference to the figures are depicted as each occurring once in a specific order, it will be appreciated that some operations can be performed in any suitable order and repeated as needed. Furthermore, the operations shown in the figures can be combined or may include more or less detail than described.

[0281] The various implementations described here can refer to AI, machine learning, and deep learning. Deep learning may be a subset of machine learning. Machine learning may be a subset of AI. When referring to deep learning models, machine learning models can be used instead if they are suitable for a specific application. When referring to deep learning models, digital signal processing systems can be used instead if they are suitable for a specific application.

[0282] The foregoing description of the illustrated embodiments of this disclosure, including the content described in the abstract, is not intended to be exhaustive or to limit this disclosure to the precise forms disclosed. While specific implementations and examples of this disclosure have been described herein for illustrative purposes, various equivalent modifications are possible within the scope of this disclosure, as will be recognized by those skilled in the art. These modifications can be made to this disclosure based on the detailed description above.

[0283] For illustrative purposes, specific figures, materials, and configurations have been set forth to provide a thorough understanding of the illustrative implementation. However, it will be apparent to those skilled in the art that this disclosure may be practiced without specific details, and / or may be practiced only through some of the described aspects. In other instances, well-known features have been omitted or simplified to avoid obscuring the illustrative implementation.

[0284] Furthermore, reference has been made to the accompanying drawings, which form part of this specification, in which illustrative embodiments that may be implemented are shown. It should be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of this disclosure. Therefore, the following detailed description should not be construed as limiting.

[0285] Various operations can be described as multiple discrete actions or operations in a manner most conducive to understanding the disclosed subject matter. However, the order of description should not be construed as implying that these operations are necessarily sequentially related. In particular, these operations may not be performed in the order presented. The described operations may be performed in a different order than the described embodiments. In other embodiments, various additional operations may be performed, or the described operations may be omitted.

[0286] For the purposes of this disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). When referring to a measurement range, the term "between" includes both ends of the measurement range.

[0287] For the purposes of this disclosure, "A is less than or equal to a first threshold" is equivalent to "A is less than a second threshold," provided that the first and second thresholds are set in such a way that for any value of A, both statements lead to the same logical result. For the purposes of this disclosure, "B is greater than a first threshold" is equivalent to "B is greater than or equal to a second threshold," provided that the first and second thresholds are set in such a way that for any value of B, both statements lead to the same logical result. This description uses the phrases "in one embodiment" or "in an embodiment," each of which can refer to one or more of the same or different embodiments. The terms "comprising," "including," "having," etc., used with respect to embodiments of this disclosure are synonymous. This disclosure may use perspective-based descriptions, such as "upper," "lower," "top," "bottom," and "side," to interpret various features of the drawings; however, these terms are merely for ease of discussion and do not imply a desired or required direction. The drawings are not necessarily drawn to scale. Unless otherwise stated, the use of sequential adjectives such as "first," "second," and "third" to describe common objects merely indicates that different instances of the same object are referenced and does not imply that the objects so described must be in a given order in time, space, hierarchy, or any other manner.

[0288] In the following detailed description, various aspects of the illustrative implementation will be described using terms that are commonly used by those skilled in the art to communicate the substance of their work to others skilled in the art.

[0289] The terms “substantially,” “near,” “approximately,” “close to,” and “about” generally refer to a value within + / - 20% of the target value or as described herein or known in the art. Similarly, terms indicating the orientation of various elements, such as “coplanar,” “perpendicular,” “orthogonal,” “parallel,” or any other angle between elements, generally refer to a value within + / - 5-20% of the target value or as described herein or known in the art.

[0290] Furthermore, the terms "comprise," "comprising," "include," "have," or any other variations thereof are intended to cover non-exclusive inclusion. For example, a method, process, or apparatus that includes a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to the method, process, or apparatus. Additionally, the term "or" refers to an inclusive "or," not an exclusive "or." Each of the systems, methods, and apparatuses disclosed herein has several innovative aspects, none of which is solely responsible for all the desired properties disclosed herein. Details of one or more implementations of the subjects described herein are set forth in the specification and accompanying drawings.

Claims

1. An apparatus for compiling a neural network model to be executed on a neural network accelerator, comprising: processor; as well as Memory for storing instructions that, when executed by the processor, cause the processor to: Receive a model definition of the neural network model comprising multiple layers, the multiple layers including layers having multiple weights to be applied to multiple activations; The order of the multiple weights is determined based on the switching activity metric; Multiple rearranged weights are determined by arranging the multiple weights according to a sorting order; as well as Generate one or more machine-readable configurations for the neural network accelerator to load the plurality of rearranged weights according to a sorting order and apply the plurality of rearranged weights to the plurality of activations.

2. The device according to claim 1, wherein, The neural network accelerator includes an array of processing elements for applying the plurality of weights to the plurality of activations, and the array of processing elements is a multiplicative cumulative array.

3. The device according to claim 1 or 2, wherein, The switching activity metric between a pair of weights among the plurality of weights is the Hamming distance.

4. The device according to claim 1 or 2, wherein, The instruction causes the processor to determine the order of the multiple weights in the following manner: Select one of the multiple weights as the current pivot; Select the next weight that has the minimum switching activity metric for the current hub from the sorting of the multiple weights; as well as Update the current central pivot to the next weight.

5. The device according to claim 1 or 2, wherein, The instructions further cause the processor to: A subset of the multiple layers is selected based on one or more of the switching activity score for each layer and the number of weights for each layer, wherein the switching activity score quantifies the number of bit transitions for a given layer.

6. The device according to claim 1 or 2, wherein, The instructions further cause the processor to: Under the constraint of selecting only non-continuous layers, a subset of the layers is selected.

7. The device according to claim 1 or 2, wherein, The instructions further cause the processor to: A subset of the layers is selected by traversing the multiple layers and comparing the cumulative switching cost when the current layer is skipped with the additional cumulative switching cost when the current layer is selected.

8. The device according to claim 1 or 2, wherein, The instruction causes the processor to determine the order of the multiple weights in the following manner: Determining the order of the multiple weights reduces switching activity between rows of the multiple weights corresponding to the multiple input channels of the layer.

9. The device according to claim 1 or 2, wherein, The instruction causes the processor to determine the order of the multiple weights in the following manner: Determining the order of the plurality of weights reduces switching activity between the last row of the plurality of weights and the first row of another row of the plurality of other weights of the layer, the row corresponding to the plurality of input channels of the layer, and the other rows corresponding to the plurality of other input channels of the layer.

10. The device according to claim 1 or 2, wherein, The multiple weights are quantized.

11. A method for compiling a neural network model to be executed on a neural network accelerator, comprising: Receive a model definition of the neural network model comprising multiple layers, the multiple layers including layers having multiple weights to be applied to multiple activations; The order of the multiple weights is determined based on the switching activity metric; Multiple rearranged weights are determined by arranging the multiple weights according to a sorting order; as well as Generate one or more machine-readable configurations for the neural network accelerator to load the plurality of rearranged weights according to a sorting order and apply the plurality of rearranged weights to the plurality of activations.

12. The method according to claim 11, wherein, The neural network accelerator includes an array of processing elements for applying the plurality of weights to the plurality of activations, and the array of processing elements is a multiplicative cumulative array.

13. The method according to claim 11 or 12, wherein, The switching activity metric between a pair of weights among the plurality of weights is the Hamming distance.

14. The method according to claim 11 or 12, wherein, Determining the order of the multiple weights includes: Select one of the multiple weights as the current pivot; Select the next weight that has the lowest switching activity metric for the current hub from the ranking of the multiple weights; and Update the current central pivot to the next weight.

15. The method according to claim 11 or 12, further comprising: A subset of the multiple layers is selected based on one or more of the switching activity score for each layer and the number of weights for each layer, wherein the switching activity score quantifies the number of bit transitions for a given layer.

16. The method according to claim 11 or 12, further comprising: Under the constraint of selecting only non-continuous layers, a subset of the layers is selected.

17. The method according to claim 11 or 12, further comprising: A subset of the layers is selected by traversing the multiple layers and comparing the cumulative switching cost when the current layer is skipped with the additional cumulative switching cost when the current layer is selected.

18. The method according to claim 11 or 12, wherein, Determining the order of the multiple weights includes: Determining the order of the multiple weights reduces switching activity between rows of the multiple weights corresponding to the multiple input channels of the layer.

19. The method according to claim 11 or 12, wherein, Determining the order of the multiple weights includes: Determining the order of the plurality of weights reduces switching activity between the last row of the plurality of weights and the first row of another row of the plurality of other weights of the layer, the row corresponding to the plurality of input channels of the layer, and the other rows corresponding to the plurality of other input channels of the layer.

20. The method according to claim 11 or 12, wherein, The multiple weights are quantized.

21. One or more non-transitory computer-readable media storing instructions for compiling a neural network model to be executed on a neural network accelerator, which, when executed by a processor, cause the processor to perform the method according to any one of claims 11-20.

22. A computer program product comprising instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 11-20.

23. A machine-readable storage device comprising machine-readable instructions that, when executed, cause a computer to perform the method according to any one of claims 11-20.

24. A computer program comprising instructions that, when executed by a processing device, cause the processing device to perform the method according to any one of claims 11-20.

25. A computer-implemented system comprising one or more processors and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the method according to any one of claims 11-20.

Images