Runtime configurable register file for artificial intelligence workloads

Abstract

Systems and methods for performing artificial intelligence (AI) inference are disclosed that include programming an AI accelerator circuit to solve an AI problem using multiple layer-specific register file (RF) size distributions, where the AI ​​accelerator circuit comprises processing elements (PEs) having respective associated RFs, the RFs being individually divided into K sub-banks of size B bytes, where B and K are integers, and the RFs include circuitry for individually allocating the sub-banks to one of input features (IFs), output features (OFs), or filter weights (FLs), where programming the multiple layer-specific RF size distributions includes taking into account sparse data within the layer; and causing the AI ​​accelerator circuit to run the AI ​​problem including applying the layer-specific RF size distributions at runtime.

Description

[Technical Field]

[0001] [Cross-reference of related applications] This application claims the benefit of U.S. Nonprovisional Application No. 17 / 530,156, filed on 18 November 2021, entitled “Runtime Configurable Register Files for Artificial Intelligence Workloads,” thereby incorporating, for all purposes, the entirety of said U.S. Nonprovisional Application by reference.

[0002] This specification relates to the field of artificial intelligence, and more specifically, to runtime configurable register files for artificial intelligence workloads, but not exclusively. [Background technology]

[0003] Artificial intelligence is a subfield of computer science in which computers or circuits are programmed to learn from data and update their algorithms based on this learning. One popular type of artificial intelligence (AI) circuit is the neural network (NN). When an NN has multiple convolutional layers between its input and output layers, it may be called a deep neural network (DNN). One popular species of DNN is the convolutional neural network (CNN). To achieve performance advantages, AI circuits can be implemented in hardware accelerators, which may be, for example, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or some other hardware platform. Accelerators can be used to offload AI tasks to hardware circuits, where the AI ​​tasks can be executed faster than on a general-purpose processor.

[0004] An accelerator can operate on multiple input and output tensors, such as input features (IF) and output features (OF), and filter weights (FL). These may be stored in a dedicated register file, which may be a high-speed memory circuit associated with each processing element in the AI ​​accelerator circuit. The register file (RF) is much faster to access than higher-level memory such as static random access memory (SRAM). In at least some existing systems, RF is statically allocated among IF, OF, and FL. For example, each tensor may be allocated a 64-byte register. Static register allocation can lead to inefficiencies in memory management in at least some cases. [Brief explanation of the drawing]

[0005] This disclosure will be best understood from the following detailed description when read in conjunction with the accompanying drawings. In accordance with standard industry practice, it is emphasized that various features are not necessarily depicted to scale and are used merely for illustrative purposes. Where a scale is explicitly or implicitly indicated, it provides only one exemplary example. In other embodiments, the dimensions of various features may be arbitrarily increased or decreased to clarify the discussion. Furthermore, the various block diagrams shown herein disclose only one exemplary arrangement of logical elements. These elements may be rearranged in different configurations, and elements shown in one block may, in appropriate circumstances, be moved to a different block or configuration.

[0006] [Figure 1] These are block diagrams of hardware circuits according to various embodiments.

[0007] [Figure 2] These are block diagrams of subcircuits according to various embodiments.

[0008] [Figure 3A]Block diagram of selected elements of a static RF ecosystem according to various embodiments.

[0009] [Figure 3B] Diagram of an alternative schedule generator according to various embodiments.

[0010] [Figure 4] Block diagram of two register files showing the difference between a fixed-capacity register file and a dynamic register file according to various embodiments.

[0011] [Figure 5] Block diagram showing selected aspects of an elastic register file scheme according to various embodiments.

[0012] [Figure 6] Graph showing the relative hardware costs of different configurations according to various embodiments.

[0013] [Figure 7] Graph showing the percent reduction in total SRAM load access from using an exemplary elastic register file according to various embodiments.

[0014] [Figure 8] Block diagram of selected elements of a system-on-chip (SoC) according to various embodiments.

[0015] [Figure 9] Diagram showing machine learning according to "textbook" problems associated with real-world applications according to various embodiments.

[0016] [Figure 10] Flowchart of a method that can be used to train a neural network according to various embodiments.

[0017] [Figure 11] This is a flowchart illustrating how to use a neural network to classify objects according to various embodiments.

[0018] [Figure 12] A block diagram showing selected elements of an analyzer engine according to various embodiments.

[0019] [Figure 13] This is a block diagram of a circuit programming ecosystem according to various embodiments.

[0020] [Figure 14] This is a flowchart illustrating a method for programming hardware circuits according to various embodiments. [Modes for carrying out the invention]

[0021] Outline This specification provides flexible or elastic RF, or other circuits that may benefit from elastic registers, within AI accelerator circuits. In some existing systems, the register file (RF) is allocated to each processing element (PE) which is divided among three distinct tensors (e.g., IF, OF, and FL). If each tensor is allocated 64 bytes, for example, the total RF is 192 bytes. Since the accelerator is a hardware circuit, the RF has a fixed configuration with a fixed division among three registers for the three tensors.

[0022] Some existing systems have sought to make better use of RF space by, for example, dividing RF into non-uniform sizes, e.g., 128 bytes for IF and 32 bytes each for OF and FL. For example, FPGAs can be programmed to provide hardware circuitry at speeds comparable to those achieved in ASICs. FPGAs can be programmed using non-uniform register files (e.g., the sizes of IF, FL, and OF registers do not need to be the same). This may result in better data utilization in some layers but counterproductive in others. Again, since the accelerator is hardware circuitry, the register file cannot be modified at runtime to account for, for example, data sparsity, data stationarity, or tensor shape within a given layer.

[0023] However, these factors can be known in advance, and different register file configurations can provide performance advantages at different layers. For example, if the IF is highly stationary at layer 2, it may be advantageous to provide larger registers (e.g., 128 bytes) for the IF at that layer. However, if the IF is not stationary at layer 3, a register configuration that was very efficient at layer 2 may become very inefficient at layer 3.

[0024] Therefore, it is desirable to provide the system with a flexible register file allocation for each layer. Given a flexible register file, the register configuration can be optimized on a layer-by-layer basis before the AI ​​problem is loaded into the hardware accelerator. AI system designers know the data sparsity, tensor shape, and data stationarity that occur in each layer during the design phase. Based on these factors, designers can optimize memory usage by scheduling registers to have more or less capacity for a given layer. In general, data with high stationarity can better utilize larger registers, while sparse data can better utilize smaller registers.

[0025] To provide flexible registers, hardware accelerators may be provided with elastic register files. These contain register files divided into multiple subbanks, each with a given number of bytes. Input multiplexers and output demultiplexers are connected to the inputs and outputs of each register bank. This allows the system programmer to individually select tensors (i.e., one of IF, FL, or OF) for each subbank. The system designer can create a layer-specific register schedule that takes into account the data shape and structure of each layer. This register schedule can be loaded into the accelerator circuit before the AI ​​network is executed, and the accelerator can then apply the schedule to each layer as it is activated.

[0026] The teachings herein can be embodied in various illustrative implementations. One example is a method comprising the steps of: generating a plurality of layer-specific register schedules for a deep learning neural network, wherein at least two layer-specific register schedules are distinct from one another, the layer-specific register schedules are for dividing a register file into a plurality of tensor-specific registers, the register file comprising a plurality of discrete subbanks, each of which comprises one or more of the subbanks; and programming a configuration register to provide the layer-specific register schedules; programming an AI hardware circuit using the plurality of layer-specific register schedules; and instructing the AI ​​hardware circuit to begin.

[0027] An example of the aforementioned plurality of tensor-specific registers is also disclosed, which include registers for input features (IF), output features (OF), and filter weights (FL).

[0028] An example is also disclosed in which the layer-specific register schedule is for multiple register files, and the schedule for the multiple register files is the same within the layer.

[0029] An example is also disclosed in which the register file is associated with each PE of the AI ​​hardware circuit.

[0030] One example is also disclosed in which the step of generating a layer-specific register schedule includes providing smaller registers for tensors having sparse data in the layer compared to tensors having non-sparse data in the layer.

[0031] In one example, the step of generating a layer-specific register schedule is disclosed, which includes a step of providing surplus capacity for tensors with high stationarity within the layer.

[0032] In one example, the step of generating a layer-specific register schedule also includes a step of considering the tensor shape within the layer.

[0033] One example is a method for performing AI inference, comprising the steps of: programming an AI accelerator circuit to solve an AI problem using multiple layer-specific register file (RF) size allocations, wherein the AI ​​accelerator circuit comprises a PE having each associated RF, the RF being individually divided into K subbanks of size B bytes, where B and K are integers, and the RF includes a circuit for individually allocating the subbanks to one of input features (IF), output features (OF), or filter weights (FL); the step of programming the multiple layer-specific RF size allocations comprises the step of considering sparse data within the layers; and causing the AI ​​accelerator circuit to execute the AI ​​problem, which includes the step of applying the layer-specific RF size allocations at runtime.

[0034] An example is disclosed in which the PE is a multiply-accumulate unit (MAC).

[0035] An example is also disclosed in which B is one of 1, 2, 4, 8, 16, 32, 64, or 128.

[0036] An example is also disclosed in which B is between 1 and 128.

[0037] An example of the AI ​​circuit being a DNN is also disclosed.

[0038] An example of the aforementioned AI circuit being a CNN is also disclosed.

[0039] The step of programming the aforementioned layer-specific RF size distributions includes a step of considering steady-state data within a particular layer, the steady-state data including data that rarely changes within a particular layer, as is also disclosed.

[0040] One example is a plurality of substantially identical processing element circuits, where the plurality of PE circuits are configured to provide discrete numerical operations for the AI ​​accelerator circuit to execute an AI algorithm; a plurality of register files, which are communicably coupled to and associated with each of the PE circuits, where the register files are configured to store at least two kinds of data and have a total capacity C divided into K subbanks of B bytes each. TOT The device is an AI accelerator circuit or the like, having bytes, the K subbanks each having input and output multiplexer circuits configured to selectively assign each subbank to one of the at least two types of data; and a control circuit configured to change the subbank assignments for different layers of the neural network of the AI ​​accelerator at runtime.

[0041] An example is also disclosed in which the PE circuit is a multiply-accumulate (MAC) unit.

[0042] An example is also disclosed in which the aforementioned PE circuits are substantially identical to each other in hardware.

[0043] An example is also disclosed in which the control circuit has an input multiplexer and an output demultiplexer for each of the subbanks.

[0044] An example is disclosed in which the data for at least two species includes data for three species.

[0045] An example of the three types of data mentioned above, including input features (IF), output features (OF), and filter weights (FL), is also disclosed.

[0046] An example is also disclosed in which the register file includes at least one dedicated subbank for each of the at least two types of data.

[0047] An example is also disclosed in which the dedicated subbank does not have input and output multiplexers.

[0048] An example where B=1 is also disclosed.

[0049] An example where B=4 is also disclosed.

[0050] An example where B=8 is also disclosed.

[0051] An example where B=16 is also disclosed.

[0052] An example where B=32 is also disclosed.

[0053] Examples of such data are also disclosed, including tensor inputs and / or outputs for the AI ​​algorithm.

[0054] The aforementioned neural network is a CNN, and one example is also disclosed.

[0055] An example of the aforementioned CNN being a DNN is also disclosed.

[0056] An example is also disclosed that further includes a counter and glue logic circuit for maintaining state data relating to the active layer and the DNN.

[0057] An example is also disclosed in which the control circuit is for assigning the subbanks according to the layer-level attributes of the hidden layers of the DNN.

[0058] An example of the aforementioned control circuit, which takes data sparsity into consideration when allocating the subbanks, is also disclosed.

[0059] An example of the control circuit described above is disclosed, which is intended to take into account the layer-level tensor dimension when allocating the subbanks.

[0060] An example of the aforementioned AI accelerator circuit being an ASIC is also disclosed.

[0061] An example of the aforementioned AI accelerator circuit being an FPGA is also disclosed.

[0062] An example of the aforementioned AI accelerator circuit, which is an intellectual property (IP) block, is also disclosed.

[0063] An example of one or more tangible non-temporary storage media storing one or more masks or instructions for creating or implementing the aforementioned AI accelerator circuit is also disclosed.

[0064] An example of a device is also disclosed comprising: a processing element circuit configured to perform calculations using multiple input and / or output types; a register file communicatively coupled to the PE circuit and containing multiple hardware subregisters; and a runtime programmable selection circuit for assigning the subregisters of the register file to each of the input and / or output types.

[0065] An example of the aforementioned PE circuit, which is used to perform mathematical operations for AI problems, is also disclosed.

[0066] An example is also disclosed in which the PE circuit is a multiply-accumulate (MAC) unit.

[0067] An example is also disclosed that further comprises multiple PE circuits, each associated with a different register file.

[0068] An example is also disclosed in which the aforementioned multiple PE circuits are substantially identical to one another.

[0069] An example of the selection circuit is also disclosed, which includes an input multiplexer and an output demultiplexer.

[0070] The aforementioned input and / or output types include, for example, three types of input and / or output values, and such examples are also disclosed.

[0071] An example of the register file is also disclosed, which includes K sub-registers of a common size of B bytes.

[0072] An example where B=1 is also disclosed.

[0073] An example where B=4 is also disclosed.

[0074] An example where B=8 is also disclosed.

[0075] An example where B=16 is also disclosed.

[0076] An example where B=32 is also disclosed.

[0077] An example is also disclosed in which the register file includes at least one dedicated sub-register for each of the input and / or output types.

[0078] An example is also disclosed in which the dedicated sub-register does not have a selection circuit.

[0079] Examples of the aforementioned input and / or output types are also disclosed, including tensor inputs and / or outputs for AI problems.

[0080] An example of the aforementioned PE circuit is disclosed, which is intended to provide a CNN for the aforementioned AI problem.

[0081] An example of the aforementioned CNN being a DNN is also disclosed.

[0082] An example is also disclosed that further includes a counter and glue logic circuit for maintaining state data relating to the active layer and the DNN.

[0083] An example is also disclosed which further includes a control circuit for programming the selection circuit at runtime.

[0084] An example of the aforementioned control circuit, which takes data sparsity into consideration when allocating the sub-registers, is also disclosed.

[0085] An example of the aforementioned control circuit is disclosed, which takes into account the layer-level tensor dimension when allocating the sub-registers.

[0086] Examples of the aforementioned input and / or output types are also disclosed, including an input feature (IF) tensor, an output feature (OF) tensor, and a filter weight (FL) tensor.

[0087] An example of the aforementioned device, which is an AI accelerator circuit, is also disclosed.

[0088] An example of the aforementioned AI accelerator circuit being an ASIC is also disclosed.

[0089] An example of the aforementioned AI accelerator circuit being an FPGA is also disclosed.

[0090] An example of the aforementioned AI accelerator circuit being an IP block is also disclosed.

[0091] An example of one or more tangible non-temporary storage media storing one or more masks or instructions for creating or implementing the aforementioned AI accelerator circuit is also disclosed.

[0092] An example of a method for performing AI inference is also disclosed, comprising the steps of: receiving input data; providing the input data to the input layer of a DNN circuit, where the DNN circuit comprises a PE having each register file, each register file containing K banks of B-byte subregisters divisible among input feature (IF), output feature (OF), and filter weight (FL) tensors; programming each register file for the hidden layer of the DNN using layer-level allocation among the IF, OF, and FL, where the layer-level allocation takes into account the tensor shapes within the layer; and providing the inference as output.

[0093] An example is disclosed in which the PE is a multiply-accumulate unit (MAC).

[0094] An example where B=1 is also disclosed.

[0095] An example where B=4 is also disclosed.

[0096] An example where B=8 is also disclosed.

[0097] An example where B=16 is also disclosed.

[0098] An example where B=32 is also disclosed.

[0099] An example of the aforementioned DNN being a CNN is also disclosed.

[0100] An example is also disclosed that further includes a step to consider data sparsity within the layers.

[0101] An example of an apparatus comprising means for carrying out the above method is also disclosed.

[0102] The means for carrying out the above method also includes, for example, an AI accelerator circuit.

[0103] An example of the aforementioned AI accelerator circuit being an ASIC is also disclosed.

[0104] An example of the aforementioned AI accelerator circuit being an FPGA is also disclosed.

[0105] An example of the aforementioned AI accelerator circuit being an IP block is also disclosed.

[0106] An example of one or more tangible non-temporary storage media storing one or more masks or instructions for creating or implementing the aforementioned AI accelerator circuit is also disclosed.

[0107] The means for carrying out the method described above also include, for example, a processor and memory.

[0108] The memory also discloses, in one example, a machine-readable instruction that, when executed, causes the device to perform the method.

[0109] When executed, an example of at least one computer-readable medium containing instructions for implementing the method or realizing the device as described above is also disclosed.

[0110] Further examples include providing one or more tangible, non-temporary, computer-readable media storing instructions for a deep neural network (DNN) accelerator circuit, wherein the instructions include generating a plurality of layer-specific register schedules for the DNN accelerator circuit, where at least two of the layer-specific register schedules are distinct from each other, and the layer-specific register schedules are for dividing a register file into a plurality of tensor-specific registers, the register file comprising a plurality of discrete subbanks, each of which comprises one or more of the subbanks; transmitting the plurality of layer-specific register schedules, along with a deep learning problem, to a neural network hardware accelerator; and instructing the DNN accelerator circuit to begin execution.

[0111] An example of the aforementioned plurality of tensor-specific registers is also disclosed, which include registers for input features (IF), output features (OF), and filter weights (FL).

[0112] An example is also disclosed in which the layer-specific register schedule is for multiple register files, and the schedule for the multiple register files is the same within the layer.

[0113] An example is also disclosed in which the register file is associated with each processing element of the neural network accelerator circuit.

[0114] One example is also disclosed in which generating a layer-specific register schedule provides smaller registers for a tensor having sparse data in a layer compared to a tensor having non-sparse data in the layer.

[0115] One example is also disclosed in which generating a layer-specific register schedule provides surplus capacity for tensors with high stationarity within the layer.

[0116] One example is also disclosed in which generating a layer-specific register schedule involves considering the tensor shape within the layer.

[0117] The following disclosure provides many different embodiments or examples for implementing different features of this disclosure. Specific examples of components and arrangements are described below for the sake of brevity of this disclosure. These are, of course, merely examples and are not intended to be limiting. Furthermore, this disclosure may repeat reference numbers and / or letters in various examples. This repetition is for the sake of simplification and clarity and does not, in essence, define the relationships between the various embodiments and / or configurations discussed. Different embodiments may have different advantages, and certain advantages are not necessarily required for all embodiments.

[0118] A DNN operates by propagating output values ​​from one layer to the next, using the output value of the previous layer as the input value in the following layer. A more detailed explanation of how a DNN works is shown in Figures 9 to 12 below. In Figure 9, the inputs and outputs of each layer may be tensors, which are N-dimensional arrays of values ​​(where "N" is an integer), as will be explained in more detail below. Generally, a hardware platform or hardware accelerator that provides a CNN may include a bank of processing elements (PEs). A PE may be, for example, a multiply-accumulate (MAC) circuit that performs discrete convolution operations for each neuron in each layer. The MAC accesses the tensors,

number

number

number

number

[0119] These tensors may be stored in the main memory structure, or in multiple layers of memory such as DRAM or SRAM, or in one or more layers of a cache, to ensure that the MAC circuit operates at true hardware speed, while the values ​​for each layer may be loaded into hardware RFs associated with the MAC unit. For example, there may be one RF or set of RFs for each MAC unit, or one set of RFs for each group of n MAC units. These RFs are very fast storage locations, similar to hardware registers in a general-purpose central processing unit (CPU). MAC units can access registers in a single or small number of clock cycles, while higher-level caches or memory may be accessible over tens, hundreds, or thousands of clock cycles.

[0120] Throughout the remainder of this specification, exemplary embodiments are used as examples in which one register file is allocated for each MAC unit. These examples can be extended to other configurations. This specification provides dataflow-aware and sparsity-aware elastic capacities for input features (IF) or input activations, output features (OF) or output activations, and filter weights (FL). For example, each MAC unit has a total capacity C TOT It may have a register file, the total capacity of which may be elastically or dynamically divided among IF, OF, and FL.

[0121] In existing systems, RF capacity is divided into three separate registers, such as the IF register, OF register, and FL register. These may have a fixed capacity of, for example, 64 bytes or some other value (e.g., between 4 and 256 bytes). However, fixed register capacities can lead to inefficiencies, as will be explained below.

[0122] Therefore, this specification provides improvements to AI accelerator circuits including RF with dynamic or elastic capacitance to achieve increased efficiency. This may be achieved by dividing the RF into a plurality of K subbanks or subregisters, each having a capacitance of B bytes, where K and B are both integers. Input and output multiplexers (e.g., 3-to-1 and 1-to-3 multiplexers) are used to select which kind of tensor (i.e., variable) is assigned to each subbank. Theoretically, the best embodiment may be when B=1 and K is equal to the total size of the RF. This configuration provides the ability to dynamically allocate individual bytes of the RF to arbitrarily different tensors. In real-world use cases, B=1 may not be feasible due to the number of mux required, along with the associated costs in space and circuit power. Therefore, design trade-offs may encourage the adoption of other values ​​for B, such as integers between 2 and 128 bytes, in particular any one of 2, 4, 8, 16, 32, 64, or 128 bytes, as an illustrative and non-restrictive example.

[0123] With the RF divided into K separate subbanks, tensor allocation can be modified at runtime. For example, the RF may have a nominal capacity of 64 bytes per tensor, totaling 192 bytes (64 bytes each for IF, OF, and FL). If this is an elastic RF with K=48 (e.g., B=4), each of the 48 separate 4-byte subbanks may be dynamically allocated at runtime to one of IF, OF, or FL. In a very balanced layer, each tensor may receive its nominal 64 bytes, or close to it. However, in the extreme case of stationarity in IF, for example, only 4 bytes each could be allocated to OF and FL, leaving 184 bytes for IF. This allows loading large amounts of IF data into the IF register, thereby saving access to higher-level memory. This enables efficient data organization in DNN inference accelerators.

[0124] Neural networks represent a rapidly developing aspect of AI. In recent years, neural networks have seen a significant increase in the number of proposed inference algorithms and hardware platforms capable of accelerating them. The network layers for the underlying deep learning inference algorithms appear in many possible tensor shapes, and their dimensions can continuously change within very short time spans.

[0125] For example, the sequence of activation and weighted data organization within a network layer, often referred to as a "schedule" or "data flow," heavily depends on the layer dimensions, the underlying hardware architecture, and the level of sparsity in the data. Sparsity refers to the fact that some values ​​in an array can be zero, and these zero-value elements can be optimized out.

[0126] The schedule in data flow can vary significantly based on the network, hardware platform, and input dataset under consideration. Given the highly variable network layer dimensional profile, hardware platform constraints, and the sparsity content of the input dataset, it is advantageous to build flexible DNN accelerator hardware that can support efficient data organization scheduling.

[0127] In some current-level flexible-scheduling DNN accelerators, the hardware provides the ability to generate schedules for different DNN data flows, such as weight stationary, output stationary, and no local reuse, as exemplary and non-limiting examples. These address different network layer dimensions. However, some of the schedules generated by the schedule generator may be suboptimal from a data organization standpoint, as the same schedule can be used for all layers in the neural network.

[0128] In one type of design, various tensors (IF, FL, and OF) reside in their own private physical register files. In many cases, sparsity and stationarity factors result in one or more register files being completely unused. This is because each individual register file has a predetermined capacity that is statically fixed by the hardware. Many designs have been used to mitigate utilization imbalances, such as storing all different types of data in a single monolithic structure. However, reading and writing from this large global buffer often consumes a lot of power and imposes limitations on the chip's operating frequency.

[0129] Depending on the layer dimension, the generated optimal schedule may prioritize computation cycles or computational utilization using the corresponding negative effect on RF capacity utilization. If RF is not 100 percent utilized, memory capacity is wasted, and the amount of data reuse may be suboptimal.

[0130] The architecture described herein provides an elastic register file, a hardware solution that enables capacity borrowing between unused capacity in the IF, FL, and / or OF register files, further reducing data movement and improving scheduling performance. By including a configurable register file feature, the scheduler can leverage this feature to produce schedules with a better data movement profile by saving the number of accesses to higher-level memory hierarchies. Advantageously, the scheduler can change the distribution between different layers of the DNN. Thus, the scheduler can optimize the design at runtime.

[0131] Therefore, the elastic register file provides a hardware technique that facilitates the effective use of available capacity that would otherwise be wasted by borrowing unused RF capacity from one RF and distributing it to another RF. Thus, although the IF, FL, and OF register files have static dedicated capacity in hardware, this hardware technique unlocks the possibility of increasing the capacity of any of these register files through capacity borrowing from one RF with unused capacity to another RF that can use additional capacity. This promotes a higher overall level of data reuse across all RFs, resulting in fewer read data accesses to cache, SRAM, or other higher-level memory.

[0132] Since efficient data organization solutions promote energy efficiency in DNN accelerators, this specification provides techniques for granting schedulers the ability to process network layers of any dimension with different levels of sparsity in the data.

[0133] As an example, a ResNet-50 network may have a res2_branch1 layer. The capacity of this layer may be 128 bytes. In this example, the IF dimension is 56 × 56 × 54. The FL is 1 × 1 × 64 × 256. The OF is 56 × 56 × 256. The scheduler can optimize FL data movement from SRAM / RF by only 50 percent, achieving a twofold reduction in FL memory traffic. This results in significant energy savings due to the reduction in overall SRAM / RF memory traffic. Due to the increase in IF register file capacity, the system uses fewer SRAM accesses for FL.

[0134] However, statically increasing RF storage capacity negatively impacts that area and reduces the operating frequency of the DNN accelerator. The runtime-configurable register files described herein utilize capacity borrowing within the IF, FL, and OF register files to achieve reduced data movement and higher efficiency at higher operating frequencies. It can achieve these advantages without statically increasing dedicated RF capacity in hardware.

[0135] The elastic RF described herein offers numerous advantages over existing systems. For example, it enables an increase in RF capacity between RFs with static, dedicated storage capacity. It utilizes capacity by borrowing unused capacity within individual RFs to reduce overall data movement and improve performance. This offers advantages over DNN accelerators where IF, FL, and OF register files are implemented as separate, dedicated physical structures, each with statically fixed capacity in hardware. Such systems do not offer the possibility of sharing unused capacity with other RFs.

[0136] More advantageously, this specification provides a schedule-aware system. The elastic RF can increase the storage capacity of the RF involved in active calculations by borrowing unused capacity based on the DNN data flow. This is determined by the schedule and therefore can allow more data to be brought to the RF that holds the steady-state data. This achieves a higher level of data reuse.

[0137] By facilitating more advanced data reuse, this system allows the schedule generator to select the optimal schedule from a data organization perspective. This optimized schedule helps minimize load memory traffic between SRAM and RF storage closest to the computing resources.

[0138] A further advantage is that this system is sparsity-aware. The level of sparsity in the data can alter the schedule for a given network layer. This system can support such variability in scheduling based on the level of sparsity in the data, while achieving better performance in terms of data organization compared to some existing systems that are sparsity-unaware.

[0139] More advantageously, this specification provides a system for implementing the use of previously wasted RF storage capacity. This system enables the allocation of the entire RF capacity across a wide range of network layer dimensions and levels of data sparsity. This helps to provide greater data reuse within DNN accelerators. For active steady-state schedules where the IF resides in RF for longer periods, expanded capacity for the IF register file can be borrowed from any unused capacity in the FL or OF register file. For weighted steady-state schedules, sparse capacity can be borrowed from the IF or OF. Similarly, for output steady-state schedules, both IF and FL capacities can be increased simultaneously by borrowing from the OF, thereby allocating RF capacity that could have previously been wasted.

[0140] Even more advantageously, the configuration registers within the system can be programmed via software that allows for changing the capacities of IF, OF, and FL on a layer-by-layer basis.

[0141] Furthermore, this specification reduces SRAM / RF traffic for IF and FL data. In experimental implementations, SRAM / RF traffic was reduced by only 33.3 percent and 98.4 percent compared to fixed static registers.

[0142] The foregoing may be used to construct or embody some illustrative implementations in accordance with the teachings herein. Some illustrative implementations are included herein as non-limiting examples of these teachings.

[0143] Here, a system and method for runtime configurable register files for AI workloads is described with more specific reference to the attached diagrams. Note that throughout the diagrams, certain reference numbers may be repeated to indicate that a particular device or block is referenced multiple times across several diagrams. In other cases, similar elements may be given new numbers in different diagrams. None of these implementations are intended to require any particular relationships between the various embodiments disclosed. In certain examples, a category or class of elements may be referenced by a reference number ("Widget 10"), while individual types or examples of elements may be referenced by a hyphenated number ("First Specific Widget 10-1" and "Second Specific Widget 10-2").

[0144] Figure 1 is a block diagram of hardware circuit 100 according to various embodiments. Hardware circuit 100 can be, for example, an ASIC, FPGA, or other circuit. Hardware circuit 100 can also be implemented as an IP block that can be integrated into other designs or as some other modular form factor. Hardware circuit 100 may be designed to provide an AI accelerator that performs DNN operations for inference or other computations. Hardware circuit 100 is a logic diagram of a DNN accelerator architecture that includes hierarchical memory that feeds multiple PEs, which in this example are MACs. Hardware circuit 100 may be implemented in many different embodiments and form factors.

[0145] In this example, MAC bank 108 includes multiple substantially identical (in-hardware) MAC units such as MAC 0 112-0, MAC 1 112-1, MAC 2 112-2 to MAC N 112-N. Each MAC unit may be hardware coded to perform multiply-accumulate operations. In other embodiments, the computing circuitry may be programmed to perform some other mathematical operation. Furthermore, the teachings herein may be adapted to other architectures, including common CPU or GPU computing architectures that may benefit from elastic register file allocation.

[0146] In this example, RF bank 116 contains a register file, where there is a one-to-one association between the register file and MAC units. For example, RF 0 120-0 is associated with MAC 0 112-0. RF 1 120-1 is associated with MAC 1 112-1. RF 2 120-2 is associated with MAC 2 112-2. RF N 120-N is associated with MAC N 112-N.

[0147] The hierarchical memory architecture in this example includes cache 124, SRAM 128, and DRAM 132. In various implementations, some or all of these memory levels may be omitted, or different memory architectures may be used.

[0148] The configuration register 110 may be used to configure the MAC bank 108 and the RF bank 116. In some embodiments, the RF bank 116 includes registers with elastic runtime configurable memory capacity. In that case, the configuration register 110 may be used to program the RF bank 116 for each layer. In other examples, the RF bank 116 may be programmed using the RF architecture for the entire DNN.

[0149] The internal counter and glue logic 122 may be used to program the state machine, propagate data layer by layer in the neural network, track the position of the neural network (e.g., which layer is being computed), and perform other logic to provide the overall structure for larger mathematical operations performed by discrete MAC units.

[0150] When MAC bank 108 operates on various data, MAC 112 may cause the associated IF, FL, and / or OF tensors to be loaded into the associated RF 120. This data can be loaded from cache 124, SRAM 128, DRAM 132, or other sources.

[0151] The input circuit 104 may be programmed to receive inputs such as input values ​​or input problems to be calculated. When the neural network calculates the inference, the result may be sent to the output circuit 140, which can then send the output to an external destination.

[0152] Data movement (particularly between different levels of memory, such as between SRAM128 and RF120) can be expensive compared to performing calculations. Data movement is expensive both in terms of power and time. Therefore, within neural network technology, there has been a shift to allocating storage in the form of RF at the memory hierarchy closest to the calculation. For example, because data movement is expensive, some existing architectures may have a MAC112 that operates directly on cache 124, or SRAM128 if cache 124 is not present. This provides greater flexibility and reduces the need to move memory values ​​between different memory levels in the hierarchy. Therefore, in one example, the entire IF, FL, and OF data may be stored in a single monolithic off-chip DRAM132 or a single monolithic on-chip SRAM128.

[0153] The capacities of DRAM132 and SRAM128 are shared among IF, FL, and OF data within the nearest RF120 for calculating MAC112, whereas existing methods allow for static allocation of IF, FL, and OF data with a fixed distribution at design time.

[0154] In some existing architectures, the physical implementation of the accelerator architecture has RF storage implemented as a separate physical structure with dedicated storage capacity allocated to one of the IF, FL, and OF data. This may be the opposite of a monolithic structure that stores all IF, FL, and OF data together in DRAM132 or SRAM128, for example. In some cases, even the storage buffer that holds the IF, FL, and OF data is implemented as a separate physical structure with a fixed capacity.

[0155] Within existing structures, several advantages have been realized by migrating from monolithic RF structures to dedicated RF structures for each of the IF, FL, and OF data. The expensive nature of adding multiple read and write ports to a monolithic RF is one factor driving the adoption of static dedicated register files for each tensor. For example, this would require at least three read ports and three write ports for the monolithic RF if the system needed to access the IF, FL, and OF data simultaneously. In terms of area, clock period, and read energy, this has proven to be prohibitively expensive in some cases. Moreover, read and write RFs with a high number of ports need to be custom-made or are not readily available, and the RF with the largest number of ports available from a standard RF compiler is the 2R2W RF.

[0156] Existing DNN accelerator architectures can support fixed schedules. The RF storage capacity and the capacity of the intermediate-level storage buffers that should store IF, OF, and FL data may be statically fixed and unchangeable during operation or at runtime. Using a fixed schedule eliminates any need to modify storage capacity at runtime.

[0157] However, fixed hardware and fixed-schedule DNN accelerators may be suboptimal in terms of handling network layers of arbitrary dimensions measured via data movement from SRAM / RF. For example, Table 1 below shows the loss of optimality for different schedule stationarity. [Table 1] Table 1. Iso-RF DNN Accelerator Architecture

[0158] Table 1 shows the total number of SRAM accesses as a function of the fixed hardware of a DNN accelerator and the fixed-schedule dataflow it supports. The main diagonal of the table (where the hardware architecture and scheduled dataflow coincide) represents the most optimal, while the off-diagonal elements represent suboptimal. This highlights the need to design flexible-scheduled DNN dataflow accelerators that include flexible underlying hardware that can be leveraged by the schedule generator to produce more optimal or near-optimal schedules.

[0159] Several existing systems have dealt with aspects of designing flexible DNN accelerators. However, these focus on designing flexible data distribution models to enable flexible scheduling. For example, some systems may provide flexible PE computation kernels to support variable-shape tensor data processing in DNN accelerators. However, these systems do not utilize unused capacity in static dedicated register file storage for IF, OF, and FL data.

[0160] Furthermore, these systems may not be sparsity-aware and are therefore suboptimal in handling sparse data. The level of sparsity in the data can alter the schedule for a given network layer. For example, Table 2 shows the impact of sparse data on the example network "Mobilenet_v2_deploy". [Table 2] Table 2. Impact of sparse data

[0161] As shown in Table 2, the dense schedules for IF, FL, and OF register files are almost fully utilized, while for sparse data schedules, the register files are not fully utilized. Fixed dedicated capacity RF implementations may not be able to utilize unused capacity to bring in additional data from the outer round to improve the reuse factor.

[0162] However, if an elastic register file is provided instead of the hardware circuit 100, the capacity can be shared among various input and output tensors to account for the sparsity of the data, different tensor dimensions, and different stationarity, as described herein.

[0163] Figure 2 is a block diagram of subcircuit 200. Subcircuit 200 is a logic diagram of a selected configuration of MAC units such as MAC112 selected from MAC bank 108 in Figure 1.

[0164] In this example, register file 202 is divided into IF map 204, FL (filter weights) 208, and OF map 212. IF map 204 provides an input tensor to MAC unit 216. Specifically, multiplier 220 receives the input feature tensor from IF map 204. Multiplier 220 also receives scalar weights (which are a special zero-dimensional case of tensors) as filter 208. Multiplier 220 calculates the product of the IF map and the filter weights.

[0165] The adder 224 calculates the sum, i.e., the sum of the OF tensor 212, along with the product of the input feature tensor and scalar weights. This sum is then returned to the OF map 212 and stored.

[0166] AI accelerators such as the hardware circuit 100 in Figure 1 can achieve substantial speed advantages by providing a bank of MAC units such as those shown here. In this example, register file 202 is shown as a conceptual register file. In a more general sense, register file 202 simply represents a data source that can be used by MAC unit 216. This can be implemented as physical registers of fixed or flexible capacity or monolithic data structures such as SRAM or DRAM.

[0167] As shown above, the MAC unit 216 may achieve efficiency advantages by having a register file 202 with flexible register capacity, and unused capacity in a particular part of the register file may be shared with other parts of the register file.

[0168] Embodiments of this specification include hardware that modifies the capacities of IF map 204, OF map 212, and / or filter 208 via an elastic register file. This allows borrowing of unused capacity between RFs at one or more levels of the nearest memory hierarchy for computation. It should be noted that this technique may also be adapted to software methods, including software methods for problems other than the AI ​​or DNN methods disclosed herein. In general, any hardware or software method that can benefit from a flexible register file in which a portion of the registers can be lent or borrowed may benefit from the teachings herein. Any such structure is intended to be included within the scope of this specification. In some embodiments, the elastic registers are allocated between a set of fixed values, such as the three tensors (IF, OF, FL) shown in the examples herein, or between other tensors or inputs and outputs. In other embodiments, the elastic registers may be adapted for use by general-purpose data and methods.

[0169] It should be noted that there are existing software-programmable registers used to configure DNN accelerators for neural network layers. The configuration registers may be supersets of such registers.

[0170] This realizes the advantages associated with existing systems where the amount of storage for IF, OF, and FL is initially fixed. Elastic register files can modulate the capacity of RF storage allocated to IF, OF, and FL data. DNN accelerators that support activated steady, weighted steady, and output steady schedules can significantly benefit from this elastic register file technique.

[0171] Preferences can be assigned to desired tensors. For example, preferences or additional weights can be assigned to IF vs OF vs FL in terms of storage capacity. Stationary data flows, or in other words, data that reside in RF for longer durations, may be allocated higher capacity, while other faster-changing data flows may be allocated lower capacity. For activated stationary schedules, the elastic RF system borrows any unused capacity in the FL and OF register files and allocates higher capacity to IF data. For weighted stationary schedules, FL data is allocated higher capacity storage through capacity borrowing from the IF and OF register files. In the case of output stationary schedules where both activated and weighted have the same preference, the elastic RF technique can allocate equally weighted storage capacity to both IF and FL data by borrowing any unused capacity from the OF register file.

[0172] Therefore, Elastic RF achieves efficient data movement by facilitating advanced data reuse across a wide sample of the schedule (e.g., activation, weights, and output stationary). Furthermore, since the schedule for the network layer depends on the level of sparsity in the data, Elastic RF techniques can improve data organization efficiency even in the presence of sparsity in the weight and activation data.

[0173] The architectures described herein address the trend of deploying an increasing number of DNN accelerators on energy-constrained devices. DNN accelerators can perform inference for mobile computing edges for a variety of AI applications, including imaging, video, and audio applications, as exemplary and non-exclusive examples. Efficient power management schemes can be critical in battery-operated edge devices. Recent trends indicate that data movement can replace the cost of computing itself as a control factor in such devices. Therefore, enabling efficient data organization techniques through advanced data reuse can significantly improve the energy and power efficiency of DNN accelerators at the current level of technology.

[0174] Embodiments of the elastic RF scheme described herein may depend on the type of dataflow of the DNN schedule generated by the schedule generator. This may be in the form of a software compiler and may be programmed to the DNN accelerator via configuration registers. In one embodiment, identifiers in the form of flags or knobs that enable elastic RF features are introduced within the schedule generator. For different types of network layer DNN dataflow, the software may program specific register fields to specify the amount of used and unused storage capacity in the IF, OF, and FL register files. In some cases, additional pins may be provided for connection to host CPU control / status registers.

[0175] Figure 3A is a block diagram of selected elements of the static register file ecosystem 300. This can be compared with Figure 3B, which is a block diagram of selected elements of the elastic register file ecosystem.

[0176] Referring to Figure 3A, the ecosystem 300 includes a schedule generator 304. The schedule generator 304 accepts a hardware input 308, which represents a statically allocated dedicated register file capacity. The schedule generator 304 also receives a network input 312, which is used to provide schedules such as schedule A 316. The network input 312 is an input to the DNN, and as an exemplary and non-limiting example, its width (W), height (H), input channels (C), output channels (K), and filter width (F) are shown. w ), filter height (F h ), and may include the layer dimension in the form of stride (S).

[0177] From the hardware input 308, the schedule generator 304 obtains the static dedicated IF, FL, and OF register file capacities for the accelerator. Based on this, the schedule generator 304 creates schedule A 316, which is a schedule applied to the entire network. In other words, schedule A 316 is applied to all layers of the network and cannot be changed at runtime.

[0178] Figure 3B discloses an elastic register file ecosystem 302, which includes an alternative schedule generator 320. The schedule generator 320 is configured to provide elastic RF features to a neural network. The network input 328 may be identical or substantially identical to the network input 312 in Figure 3A. As previously stated, the schedule generator 320 provides W, H, C, K, F w F hNetwork inputs 328 such as S may be considered. However, hardware input 324 is different from hardware input 308 in Figure 3A. In this case, the schedule generator 320 becomes aware of the elastic RF features available in hardware. This includes the ability to borrow unused RF capacity in an IF, FL, or OF register file and allocate the borrowed capacity to any of the other IF, FL, or OF register files to increase their capacity. The elastic RF features give the schedule generator 320 the ability to generate schedules that are dataflow-aware and sparsity-aware, and RF capacity is allocated to the RF that holds steady data by borrowing surplus RF capacity that was previously unused by other registers. For example, if the IF is steady and the FL and OF are not fully utilized, capacity can be borrowed from the FL and / or OF and allocated to the IF to make better use of the steady data. Then more steady data can be loaded into the IF, and computational efficiency is increased as there is less data movement.

[0179] Therefore, the schedule generator 320 can generate schedules B 332 and C 336 along with any other schedules that may be necessary. The schedule generator 320 may assign different schedules to each layer in the neural network depending on the stationarity and / or sparsity of the data in that layer. In an exemplary example, the schedule generator 320 may generate as many schedules as there are layers in the neural network. This provides better data movement performance compared to schedule A 316 in terms of SRAM data access, due to the higher level of data reuse enabled by the elastic register file.

[0180] Figure 4 is a block diagram of two register files illustrating the difference between a fixed-capacity register file and a dynamic register file.

[0181] The fixed-capacity register file 404 includes an input activation register 408, a weight register 412, and an output activation register 416. In the case of the fixed-capacity register file 404, the input activation register 408 has a fixed capacity C IF The weight register 412 has a fixed capacity C FL The output activation register 416 has a fixed capacity C OF The total byte capacity of the register file is C TOT = C IF + C FL + C OF

[0182] In a general use case, the registers 408, 412, and 416 are stored hierarchically closest to the computing unit (e.g., MAC, etc.). Their storage capacities are static and dedicated. Regardless of the network layer dimension and the data flow of the schedule, the storage capacities allocated to IF, OF, and FL are statically assigned and remain fixed. In the case of an FPGA, these can be dynamically allocated during the burn-in of the FPGA kernel, but once the FPGA is programmed, the register file size remains fixed for the entire neural network operation.

[0183] The dynamic register file 408 represents the concept of an elastic register. In the case of the dynamic register file 408, the total capacity remains the same. In other words, C TOT for the fixed-capacity register file 404 may be the same as C TOT for the dynamic register file 408. However, the register allocation may be different. Each register can be C IF for IF or the input activation tensor, C OF for OF or the output activation tensor, and C FL ​It may have nominal capacities such as α, β, and γ. The variables α, β, and γ may represent the amount actually used for a particular layer, and are decimal values ​​between 0 and 1.0 (e.g., 0 = completely unused, 1 = completely used). Therefore, (1-α) may represent the capacity available to be "lent" to other tensors. For example, if IF uses 25% of its nominal capacity (α=0.25), then 75% ((1-α)=0.75) may be available to be lent to either OF or FL. Therefore, register 420 is α*C IF Using bytes, register 424 is β*C FL Using bytes, register 428 is γ*C OF Uses bytes. The available capacity to be borrowed by another register (typically a register that is already fully utilized, i.e., one or more of α, β, or γ are 1.0) is (1-α)*C IF +(1-β)*C FL +(1-γ)*C OF This "sparse" capacity can be allocated as needed among the IF, FL, and OF registers, and the granularity is determined by the size of each subbank.

[0184] If each register has a nominal capacity of 64 bytes, then C TOT It is 192 bytes. As shown below, there is a trade-off between the granularity for partitioning the register file and the size of the circuit and power consumption. For example, each byte can be a unit, in which case C IF The minimum value is 1 byte, giving programmers essentially unrestricted access to reprogram the sharing of register file bytes for each layer. However, 1-byte granularity can result in exorbitant size and power consumption for some use cases. Therefore, different granularities such as 2 bytes, 4 bytes, 8 bytes, 16 bytes, 32 bytes, 64 bytes, or some other measure may be used.

[0185] Using 4 bytes as an exemplary use case, each register file 420, 424, and 428 has a minimum capacity of 4 bytes. Therefore, C IF This must be at least 4 bytes for the input activation register 420. FL It must be at least 4 bytes for weight register 424. OF The output activation register 428 must be at least 4 bytes. The remaining sub-registers (e.g., 4-byte blocks) can be allocated as needed in 4-byte chunks. These can be borrowed or lent to other register files to account for the data paths, stationarity, and sparsity of each layer. Therefore, once 4 bytes are reserved for the IF, the rest of the register file can be allocated to other register files as needed for the layers.

[0186] Since the granularity is 4 bytes in this exemplary example, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, or 60 bytes can be leased to other register files for their calculations. On the other hand, if IF has high stationarity with respect to this layer and can benefit from more than 64 bytes, it may borrow additional bytes from other register files, again in increments of 4 bytes.

[0187] In some embodiments, different register files may have different granularities and, therefore, different allocation sizes. However, as shown below, the advantages of specific hardware can be realized by using common hardware such that the register file block essentially has an array of identical byte groups (i.e., sub-registers) that can be allocated according to the requirements of three different variables and their tensors.

[0188] The elastic RF storage scheme provides two partial capacities for each of the IF, OF, and FL register files. There is a used capacity portion and an unused capacity portion available for borrowing by other register files. The used capacity percentages for IF, FL, and OF may be denoted by α, β, and γ, respectively. Therefore, the total unused storage capacity for the IF, FL, and OF register files can be expressed as follows: [1-α]*C IF +[1-β]*C FL +[1-γ]*C OF

[0189] Unused portions are available to be borrowed, either partially or entirely, by any other register file. Tables 3 and 4 below illustrate the borrowings. In this case, a 192-byte RF is assumed, with each tensor having a nominal size of 64 bytes. [Table 3] Table 3. Data distribution for example data flow (Formulaic) [Table 4] Table 4. Data allocation for example data flow (byte allocation)

[0190] In this example, each register file has 64 bytes, with α=1.0, β=0.5, and γ=0.5. This results in 32 bytes of unused capacity from both the FL and OF register files being borrowed by the IF register file to increase its capacity from 64 bytes to 128 bytes.

[0191] Table 3 shows the relative allocation of IF, OF, and FL register file storage capacity for various types of scheduled data flows. For output stationarity families of schedules where activation and weight reside within the RF for equal durations, unused storage capacity may be allocated equally to activation and weight. Unused capacity of RF volume [1-α]*C IF +[1-β]*C FL +[1-γ]*C OF The capacity is distributed equally between activation and weight. When the schedule is activation steady, the Elastic RF system allocates preference to the activation storage capacity, with the entire unused RF capacity being borrowed by the IF register file. On the other hand, for weighted steady schedules, the Elastic RF scheme prioritizes weights and allocates the integrity of the unused RF capacity to the FL register file.

[0192] In the example above, a 64-byte register file is used as an example for each register. For α=0.5, β=0.5, γ=0.5, Table 3 shows the IF, FL, and OF register capacity storage for output, activation, and weighted steady-state schedules. The numbers shown above should be understood as specific examples, and the concept of elastic RF is sufficiently extendable to cover any values ​​of C, α, β, and γ. In particular, while the elastic register file scheme is presented herein as a feature of AI systems, this scheme can be extended to any hardware architecture that would benefit from the elastic register file scheme.

[0193] Figure 5 is a block diagram showing a selected embodiment of the elastic register file hardware scheme. In this case, the configuration register or register 502 controls the register subbank or group of register subbanks 504. For example, register subbanks 504-0, 504-1 to 504-N are shown herein. Again, as a specific example, each register subbank 504 may provide 4 bytes of available storage. Other sizes of register subbanks, such as 1, 2, 4, 8, 16, 32, 64, or 128 bytes, can be provided as exemplary and non-limiting examples.

[0194] The register subbank 504 can be divided as needed among IF, FL, and OF (or other tensors or general data) to realize the benefits of this specification. Each register bank 504 contains a register file 516 having a specified number of bytes available for its register file, for example, 4 bytes in this case. In a static register file with C=64, 16 fixed register banks 504 would be wired to IF, another 16 to FL, and another 16 to OF. However, in this case, a flexible register file allocation is provided. Each register file 516 is connected to an input multiplexer 508 and an output multiplexer 512. The input mux 508 receives signals from each of IF, FL, and OF. Similarly, the output mux 512 is wired to provide its signals to each of IF, FL, and OF. In this example, both the input mux508 and the output mux512 receive a common selection input from the configuration register 502, which may provide an encoding to select the correct tensor for the register file. Therefore, if the input multiplexer 508 is programmed to receive an IF, the output multiplexer 512 is also programmed to deliver an IF.

[0195] In the case of a DNN, at least one register file 516 is allocated to each tensor. In some embodiments, one or more register files providing minimum capacity may be wired to one each of IF, OF, and FL. This can save space and power costs for three surplus multiplexers, where it is known that at least one register file 516 is always allocated to each tensor.

[0196] Other register files can be dynamically allocated at runtime on a layer-by-layer basis according to the stationarity, sparsity, and data demand of a particular layer. Note that group 504 of the register bank together forms a set of registers for a specific computing unit, such as a single MAC. In other words, register subbanks 504-0 to 504-N may form an elastic "register file" for a single MAC.

[0197] From a practical standpoint, it is efficient to divide each RF capacity into K banks, each with a capacity of C / K. K is an indicator of the discrete quantum of the RF storage capacity increment that can be leased to one of the other register files, depending on the scheduled data flow. A smaller value of K results in lower hardware overhead associated with the elastic RF scheme. Fewer banks mean fewer encoders and decoders are required as hardware overhead for register file read and write paths. However, this also results in greater granularity of control over the allocation of leaseable RF capacity. In this example, the step size of the leaseable RF capacity increment is large due to the small value of K.

[0198] On the other hand, having a larger value of K suggests the ability to divide individual capacities C into much finer-grained subbanks, thereby enabling greater control over the total available RF capacity allocation. The programmer can then select individual banks of much finer-grained storage capacities. However, this comes at the cost of higher hardware area overhead, as more banks require larger encoder and decoder areas for RF read and write paths.

[0199] The configuration register 502 may be programmed via software depending on the scheduled data flow for the selected DNN data flow. It may also depend on the total number of bits in the elastic RF register, which can be expressed as 2*(3*K). This includes K banks for IF, OF, and FL respectively, where each bank of bits indicates the polarity of the data within the individual RF bank.

[0200] The configuration register 502 may provide encoded bit values ​​to select the appropriate input / output pairing for each subbank 504. In the case of the example DNN, there are three possible choices (e.g., IF, OF, and FL). In one example, a bit pair value of "00" indicates that the bank is used to store output activation data (OF). A bit pair value of "01" indicates input activation data (IF). A bit pair value of "10" indicates weight / filter data (FL). Other bit encodings may also be used. A suitable multiplexer may be inserted on the RF bank write and read paths, and selection signals for the corresponding bit pair values ​​for that RF bank may be taken from the configuration register 502.

[0201] Figure 6 is a graph showing the relative hardware costs of different configurations for different values ​​of K. The value K=1 corresponds to a baseline implementation (e.g., static register files), as found in several existing systems. Increasing the value of K corresponds to the number of banks, each of the IF, FL, and OF register files. This indicates the granularity of the partition and the granularity of the loanable subbanks. As the number of banks K increases, there is generally a linear increase in the relative hardware cost and the number of 3:1 multiplexers. Increasing the number of banks increases the number of 3:1 multiplexers added to the data path, which ultimately limits the scalability of the design. Therefore, while it is theoretically desirable to have a large value of K to maximize the loanable RF storage capacity allocation, practical considerations determine that the relative hardware costs incurred when implementing the elastic RF scheme should also be taken into account. Higher values ​​of K result in larger surface area usage and higher power consumption. K can be treated as a design-time choice that can be used by software to determine how to utilize unused capacity, given the ability to partition into K banks. The value of K may be selected by the system designer in accordance with the system design considerations. Realization of efficiency gain

[0202] Figure 7 is a graph showing the percentage reduction in total SRAM load access from using the example elastic register file. [Table 5] Table 5. FL SRAM Access Reduction

[0203] Table 5 shows the percentage reduction in the total number of SRAM load accesses using Elastic RF (sum of activated SRAM load accesses and weighted SRAM load accesses). The first column shows the scheduled data flow type and the values ​​of C and K. The columns "Inside," "Outside," and "#Entry IF / FL / OF," qualifiers "Baseline" and "Elastic RF," refer to the compiler-generated schedules for hardware that does not use and uses the Elastic RF technique, respectively. For brevity in the explanation, the first term is the output dimension variable, the second term is the blocking coefficient, and the third term is the partitioning coefficient. For example, OX / 1 / 8 inside and OX / 8 / 1 outside indicate that there are eight such identical PEs acting on eight independent X points, each PE (e.g., MAC) having 1X point, while there are seven such outer rounds acting on seven loops that extend in time. #Entry IF / FL / OF indicates the number of IF, FL, and OF entries in the RF.

[0204] Several experimental results are disclosed. Case A: Activation steady-state schedule (rows 1 and 2 in Table 5)

[0205] In the baseline scheme, where the capacities of the IF, FL, and OF register files are fixed at C=64B, the FL and OF RFs bear the allocated capacity, while the IF RF cannot be expanded to accommodate additional IF points. This is mitigated in the Elastic RF scheme, where the IF RF capacity is increased to accommodate 128B through capacity borrowing, while the FL RF stores 32B. The increase in OY points in the inner loop from OY / 2 / 1 (28 OY points in total) in the baseline schedule to OY / 4 / 14 (56 OY points in total) in the Elastic RF-supported scheduling results in a subsequent reduction in the OY outer loop from OY / 2 / 1 to OY / 1 / 1. Having more activation points in the inner loop (1*2*32=64 in the baseline vs. 1*4*32=128 in the Elastic RF) enhances the effectiveness of the activated steady-state schedule by improving the degree of activation reuse. This, in turn, reduces weight load memory traffic from SRAM by 50% because weights must be brought to the PE fewer times. The power / energy efficiency of the flexible DNN accelerator is significantly improved due to the reduction in weight data movement from SRAM to the computing unit. A similar analysis for the K=8 case is shown. For this activated steady-state schedule, no further memory traffic reduction is achieved by increasing the number of banks from K=2 to K=8. Case B: Weighted steady-state schedule (rows 3 and 4 in Table 5)

[0206] Similar analyses are shown for two types of K (K=2 and K=8) for weighted steady-state scheduling. For K=2 (Weight_1), the capacity of each RF bank is 64B / 2=32B, while for K=8, the capacity of each RF bank is equal to 64 / 8=8B. For smaller values ​​of K=2, the capacity of individual RF banks is large, so the system cannot achieve finer granularity of control over total RF capacity management. (Baseline: IF=8B, weight=64B is updated to Elastic RF: IF=8B, weight=96B, with 128-(96+8)=24B unused between IF and FL RF). The quantization of RF capacity increments occurs at multiples of 32B, which is the individual RF bank size for K=2. On the other hand, for K=8 (Weight_2), the size of individual RF banks is 8B, which allows the weighted RF capacity to be increased to 120B so that the entire 128B RF capacity is shared between IF and weight. Having more weight points within the inner loop (96B for K=2 vs. 120B for K=8) improves the efficiency of the steady-state weight schedule using more advanced weight data reuse within the PE, resulting in a 33.3% and 46.7% reduction in IF SRAM load access for K=2 and K=8, respectively (the baseline outer loop OC / 4 / 1 is reduced to the elastic RF outer loop OC / (8 / 3) / 1 and OC / (15 / 32) / 1 for the K=2 and K=8 cases, respectively).

[0207] The downside to increasing the value of K is having a smaller RF bank size, which increases the region overhead associated with the encoder and decoder on the RF write and read paths (Weight_2, which generates greater hardware overhead compared to the Weight_1 schedule). The additional timing overhead caused by the multiplexer in the RF read and write paths was impractical in the baseline implementation. However, if the RF read and write are not in the critical path, and the critical path is in the DNN accelerator sum-of-products data path unit, the timing overhead is zero. In the worst-case scenario, if the RF read and write are in the critical path, there is minimal degradation at the maximum achievable frequency of the DNN accelerator calculations. Case C: Output steady-state schedule (rows 5 and 6 in Table 5)

[0208] Finally, for the output steady-state schedule, IF and FL are treated identically and allocated equal RF storage capacity borrowed from unused capacity. For the K=4 case (Output_1) and the K=8 case (Output_2), there is a significant saving (98.4%) in IF SRAM load access, which is achieved due to the entire IF moving into the inner loop and made possible by elastic RF. The system is able to allocate additional storage capacity in finer-grained chunks, which was not possible in the K<4 case. For K<4, elastic RF does not realize the gain from SRAM load access reduction over the baseline implementation due to the large-grained bank size which does not allow for the allocation of additional IF and FL inner loop storage capacity.

[0209] Figure 7 shows graph 700 of elastic RF schemes applied to several realistic layer dimensions from ResNet-50 and Inception networks, along with reductions in activation and weight SRAM access.

[0210] As the level of sparsity in the data changes, the schedule for the layers changes, and therefore, sparsity awareness can be reflected as schedule awareness that Elastic RF can achieve.

[0211] In summary, elastic RF can be beneficial to the network layer due to its wide range of width (OX), height (OY), input channel (IC), output channel (OC), filter width (FX), filter height (FY), and stride (S), as well as the sparsity of the degree of variation in the data. Elastic RF can ensure a higher storage capacity allocation between IF, FL, and OF RF by borrowing unused RF capacity to achieve greater reuse in either IF, FL, or OF data.

[0212] Figure 8 is a block showing selected elements of an exemplary SoC800. At least some of the teachings herein may be embodied on or paired with the SoC800. The SoC800 may include or be paired with an Advanced Reduced Instruction Set Computer Machine (ARM) component. For example, the SoC800 may include or be paired with any ARM core such as A-9, A-15, etc. This architecture represents, as an exemplary example, a hardware platform that may be useful in devices such as Android® phones or tablets, iPhone® (any version), iPad®, Google Nexus, Microsoft Surface, and other tablets and smartphones. The SoC800 can also be integrated, for example, into PCs, servers, video processing components, laptop computers, notebook computers, netbooks, or touch-enabled devices.

[0213] Similar to the hardware platform QB00 described above, the SoC800 may include multiple cores 802-1 and 802-2. In this exemplary example, the SoC800 also includes an L2 cache control 804, a GPU 806, a video codec 808, a liquid crystal display (LCD) interface 810, and an interconnect 812. The L2 cache control 804 may include a bus interface unit 814 and an L2 cache 816. The liquid crystal display (LCD) interface 810 may be associated with a Mobile Industry Processor Interface (MIPI) / HDMI® link coupled to an LCD.

[0214] The SoC800 may include a subscriber identification module (SIM) interface 818, a boot ROM 820, a synchronous dynamic random access memory (SDRAM) controller 822, a flash controller 824, a serial peripheral interface (SPI) director 828, suitable power control 830, dynamic RAM (DRAM) 832, and flash 834. In addition, one or more embodiments may include one or more communication functions, interfaces, and features such as Bluetooth®, a 3G modem, a global positioning system (GPS), and an instance of 802.11 Wi-Fi®.

[0215] Designers of integrated circuits such as the SoC800 (or other integrated circuits) may use intellectual property blocks (IP blocks) to simplify system design. IP blocks are modular, standalone hardware blocks that can be easily integrated into a design. Because IP blocks are modular and standalone, integrated circuit (IC) designers only need to "drop in" the IP block to use its functionality. System designers can then create the appropriate connections to inputs and outputs.

[0216] IP blocks are often "black boxes." In other words, system integrators using IP blocks may not know, and may not need to know, the specific specifications of the IP block. In fact, IP blocks may be supplied as proprietary third-party units without the system integrator having any insight into the IP block's design.

[0217] For example, a system integrator designing a SoC for a smartphone may use, in addition to the processor core, IP blocks such as memory controllers, non-volatile memory (NVM) controllers, Wi-Fi®, Bluetooth®, GPS, 4G or 5G networks, audio processors, video processors, image processors, graphics engines, GPU engines, security controllers, and many other IP blocks. In many cases, each of these IP blocks has its own embedded microcontroller.

[0218] In an exemplary example, the SoC800 also includes an AI accelerator circuit 825. The AI ​​accelerator circuit 825 may be closely coupled to the SoC800. The programming module 827 may include the logic, software, or firmware necessary to program the AI ​​accelerator circuit 825. An example of such a configuration is shown in Figure 13 below.

[0219] Figures 9–11 illustrate selected elements of an AI system or architecture. In these figures, a basic neural network is used as a representative embodiment of an AI or machine learning architecture or engine. This should be understood as a non-limiting example, and other machine learning or AI architectures are available, including, for example, symbolic learning, robotics, computer vision, pattern recognition, statistical learning, speech recognition, natural language processing, deep learning, convolutional neural networks, recurrent neural networks, object recognition, etc.

[0220] Figure 9 illustrates machine learning according to the “textbook” problem associated with a real-world application. In this case, the neural network 900 is tasked with recognizing characters. For simplicity, the neural network 900 is tasked only with recognizing single digits in the range of 0 to 9. These are provided as the input image 904. In this example, the input image 904 is an 8-bit grayscale image with 28x28 pixels. In other words, the input image 904 is a square with a width of 28 pixels and a height of 28 pixels. Each pixel has a value between 0 and 255, where 0 represents white or colorless, 255 represents black or full color, and values ​​in between represent various shades of gray. This provides a clear problem space for demonstrating the operating principle of the neural network. Only selected elements of the neural network 900 are shown in this figure; its real-world applications can be more complex and may include additional features such as the use of multiple channels (for example, in the case of a color image, there may be three separate channels for red, green, and blue). To meet the demands of a particular problem, additional layers or functions of complexity may be provided in a neural network or other AI architecture. In practice, the architecture described herein is sometimes referred to as the "hello-world" problem in machine learning and is provided merely as one example of how the machine learning or AI functions described herein may be implemented.

[0221] In this case, the neural network 900 includes an input layer 912 and an output layer 920. In principle, the input layer 912 receives input such as an input image 904, and in the output layer 920, the neural network 900 "ignites" a perceptron, which indicates which character the neural network 900 believes is represented by the input image 904.

[0222] There are several hidden layers 916 between the input layer 912 and the output layer 920. The number of hidden layers 916 depends on the problem to be solved, the available computational resources, and other design factors. Generally, the more hidden layers 916 there are, and the more neurons there are per hidden layer, the more accurate the neural network 900 can become. However, adding hidden layers and neurons also increases the complexity of the neural network and its demand on computational resources. Therefore, some design skill is required to determine the appropriate number of hidden layers 916 and the number of neurons that should be represented in each hidden layer 916.

[0223] The input layer 912 in this example contains 784 "neurons" 908. Each neuron in the input layer 912 receives information from a single pixel of the input image 904. Since the input image 904 is a 28x28 grayscale image, it has 784 pixels. Therefore, each neuron in the input layer 912 holds 8 bits of information taken in from one pixel of the input layer 904. This 8-bit value is the "activation" value for that neuron.

[0224] Each neuron in the input layer 912 has a connection to each neuron in the first hidden layer of the network. In this example, the first hidden layer has neurons labeled 0 to M. Each of the M+1 neurons is connected to all 784 neurons in the input layer 912. Each neuron in the hidden layer 916 contains a kernel or transfer function, which will be described in more detail below. The kernel or transfer function determines how large a "weight" should be assigned to each connection from the input layer 912. In other words, a neuron in the hidden layer 916 can think of some pixels as being more important to its function than others. Based on this transfer function, each neuron independently calculates an activation value, which may be, for example, a decimal number between 0 and 1.

[0225] Each neuron in this layer is also connected to each neuron in the next layer, which has 0 to N neurons. Similar to the previous layer, each neuron has a transfer function that assigns a specific weight to each of its M+1 connections and calculates its own activation value. In this way, the values ​​propagate along the hidden layer 916 until they reach the final layer, which has P+1 neurons labeled 0 to P. Each of these P+1 neurons has a connection to each neuron in the output layer 920. The output layer 920 contains neurons known as perceptrons that calculate activation values ​​based on their weighted connections to each neuron in the final hidden layer 916. The final activation value calculated in the output layer 920 can be thought of as the "probability" that the input image 904 is a value represented by the perceptrons. For example, if the neural network 900 is working perfectly, perceptron 4 will have a value of 1.00, while each other perceptrons will have a value of 0.00. This theoretically represents perfect detection. In practical applications, detection is generally not expected to be perfect, and it is desirable that perceptron 4 has a value close to 1, while the other perceptrons have values ​​close to 0.

[0226] Conceptually, neurons in the hidden layer 916 can correspond to "features." For example, in the case of computer vision, the task of recognizing characters can be divided into recognizing features such as loops, lines, curves, or other features that make up the characters. Recognizing each loop, line, curve, etc., can be further divided into recognizing smaller elements that make up the feature (e.g., line or curve segments). By moving through the hidden layers from left to right, it is often expected and desirable that each layer recognizes "construction blocks" that make up features for the next layer. In practical applications, achieving this effect is a non-trivial problem and may require a higher level of sophistication in programming and training than fairly represented in this simplified example.

[0227] The activation value for a neuron in the input layer is the value obtained from the corresponding pixel in the bitmap. The activation value (a) for each neuron in subsequent layers is calculated according to a transfer function, which takes into account the respective "strength" of its connection to each neuron in the preceding layer. The transfer can be described as the sum of weighted inputs (i.e., the activation values ​​(a) received from each neuron in the preceding layer multiplied by a weight representing the strength of the neuron-to-neuron connection (w)) plus a bias value.

[0228] A common operation for kernels is convolution, in which case the neural network may be called a "convolutional neural network" (CNN). Networks with multiple hidden layers between the input and output layers may be called deep neural networks. In current convention, the convolutional DNN (known as CNN) is the most commonly used type of AI circuit or program.

[0229] In the case of CNNs, convolution can be performed in software (such as on a general-purpose computer or GPU-based hardware) or in special hardware. For example, a multiply-accumulate unit (MAC unit) can perform a formal

number

number

number

number

[0230] The fundamental data structure of a CNN is a tensor. A tensor is an n-dimensional structure of values, where n indices are required to address a particular value. Scalars, vectors, and matrices are special cases of tensors. A scalar is a 0-dimensional tensor, or a single value. A vector is a 1-dimensional tensor, which can be addressed via a single index (for example,

number

number

number

number

[0256]

[0256]

[64]

[12] ;

[0231] The fundamental properties of a tensor include its rank, axes, and shape. Tensor rank refers to the number of dimensions of the tensor. For example, a 2-dimensional tensor (known as a matrix) has rank 2. Axes are distinct dimensions. For example, a rank 2 tensor has axis 0 and axis 1. In common use, these may also be referred to as the "x" and "y" axes. A 3-dimensional tensor has "x", "y", and "z" axes. Higher-rank tensors generally do not have common names for their axes, and these axes may be indicated by their order.

[0232] The tensor shape is a measure of the length of each axis. For example, a rank 3 tensor with 256 elements on axis 0, 256 elements on axis 1, and 64 elements on axis 2 is:

number

number

number

number

[0233] When calculating convolution, weights may be used, for example, to "select" regions of interest in the pixmap that correspond to "features" represented by neurons. Positive weights may be used to select this region, with higher positive magnitudes representing a higher probability that pixels (if activation values ​​come from the input layer) or sub-features (if activation values ​​come from the hidden layer) in that region correspond to features. Negative weights may be used to actively "exclude" surrounding regions or sub-features (for example, to mask brighter values ​​on edges), which may be used, for example, to remove noise on the edges of features. Pixels or sub-features isolated from features may have a weight of zero, for example, meaning that those pixels should not contribute to the inspection of the feature.

[0234] Bias (b) may be used to set a threshold for detecting features. For example, a large negative bias indicates that a feature should only be detected if it is strongly detected, while a large positive bias makes it much easier to detect the feature.

[0235] A biased weighted sum yields a number of arbitrary sign and magnitude. This real number can then be normalized to a final value between 0 and 1, which (conceptually) represents the probability that the feature represented by this neuron was detected from the input received from the previous layer. The normalization may include functions such as a step function, sigmoid, piecewise linear function, Gaussian distribution, linear function or regression, or the popular "normalized linear unit" (ReLU) function. In the examples herein, the sigmoid function notation (σ) is used as an illustrative example, but it should be understood to represent any normalization function or algorithm used to calculate the final activation value in a neural network.

[0236] The transfer function for each neuron in a single layer yields a scalar value. For example, the activation value for neuron "0" in layer "1" (the first hidden layer) can be described as follows:

Number

[0237] In this case, it is assumed that layer 0 (input layer 912) has 784 neurons. Here, the previous layer has "n" neurons, and the function can be generalized as follows:

Number

[0238] A similar function is used to calculate the activation value of each neuron in layer 1 (the first hidden layer), which is weighted by the strength of the connection of that neuron to each neuron in layer 0 and biased by some threshold. As discussed above, the sigmoid function shown here is intended to represent any function that normalizes the output to a value between 0 and 1.

[0239] The complete transfer function for layer 1 (having k neurons in layer 1) can be described as follows in matrix notation:

Number

[0240] More compactly, the complete transfer function for layer 1 can be described as follows in vector notation:

Number

[0241] Neural connections and activation values ​​are propagated throughout the network's hidden layers 916 until the network reaches the output layer 920. In the output layer 920, each neuron is a "bucket" or classification, and its activation value represents the probability that an input object should be classified to its perceptron. Classifications can be mutually exclusive or polynomial. For example, in computer vision for character recognition, a character may be best assigned only one value, or in other words, a single character is not expected to be both "4" and "9" at the same time. In that case, the neurons in the output layer 920 are binary perceptrons. Ideally, only one value exceeds the threshold, figuratively "igniting" the perceptron, and that value is selected. In cases where multiple perceptrons ignite, the one with the highest probability may be selected. The result is that only one value (in this case, "4") should be ignited, while the rest should be "dark". In fact, if the neural network were theoretically perfect, the "4" neurons would have an activation value of 1.00, while each of the other neurons would have an activation value of 0.00.

[0242] In the case of a multinomial perceptron, more than one output may be fired. For example, a neural network may determine that a particular document has high activation values ​​for perceptrons corresponding to several departments, such as accounting, information technology (IT), and human resources. On the other hand, it may have low activation values ​​for perceptrons for legal, manufacturing, and transportation. In the case of a multinomial classification, a threshold may be defined, and any neuron in the output layer with a probability exceeding the threshold may be considered a "match" (e.g., the document is related to those departments). Neurons below the threshold are considered to be non-matches (e.g., the document is not related to those departments).

[0243] The weights and biases of a neural network function as parameters, or "controls," where features in the previous layer are detected and recognized. When the neural network is first initialized, the weights and biases may be assigned randomly or pseudo-randomly. Therefore, since the weights and bias controls are garbage, the initial output is expected to be garbage. In the case of "supervised" learning algorithms, the network is improved by providing a "training" set, which contains objects with known outcomes. Since the correct answer for each object is known, the training set can be used to iteratively move the weights and biases away from garbage values ​​and towards more useful values. A "validation set" can be used to verify the success of the training. The validation set has known values ​​like the training set, and the trained network can be run against the validation set and the results are measured.

[0244] Common methods for improving values ​​include "gradient descent" and "backpropagation". An example of gradient descent involves calculating a "cost" function, which measures the error in the network. For example, in the example, four perceptrons ideally have a value of "1.00", while other perceptrons have an ideal value of "0.00". The cost function takes the difference between each output and its ideal value, squares this difference, and then takes the sum of all differences. Each training example has its own calculated cost. Initially, the cost function is very large because the network does not yet know how to classify objects. As the network is trained and improved, the cost function value is expected to become smaller as the weights and biases are adjusted toward more useful values.

[0245] For example, if 100,000 training examples are working, we can calculate the average cost (e.g., a mathematical average) across all 100,000 training examples. This average cost provides a quantitative measure of how "poorly" the neural network is performing its detection job.

[0246] Therefore, the cost function can be considered as a single, very complex equation, where the inputs are the network parameters (weights and biases). The network may have thousands or even millions of parameters, and the cost function may have thousands or even millions of input variables. The output is a single value representing a quantitative measurement of the network's error. The cost function can be expressed as follows:

number

[0247] Here,

number

number

[0248] Symbolically solving such problems is infeasible and, in some cases, may not even be possible, even with significant computational power. Instead, neural networks generally solve minimization problems numerically. For example, the network can calculate the gradient of the cost function at any given point and then make a shift by some small amount depending on whether the gradient is positive or negative. The magnitude of the adjustment can depend on the magnitude of the gradient. For example, if the gradient is large, it is expected that the minimum is "far away", so a larger adjustment is made. As the gradient gets smaller, smaller adjustments are made to avoid overshooting the minimum. From the perspective of multivector calculus, this is a gradient function of many variables:

Number

[0249]

Number

Number

Number

[0250] Gradient descent involves calculating the gradient function and taking small steps (with step sizes depending on the magnitude of the gradient) in the "downhill" direction of the gradient, and then repeating until a minimum is found within a threshold.

[0251] Finding the minimum is

number

[0252] In many cases, the most difficult part of a gradient descent is

number

number

[0253] However, the perceptron value is not hardcoded, but rather depends on the activation value received from the preceding layer. The parameters of the perceptron itself (weights and biases) can be adjusted, and it may even be desirable to receive different activation values ​​from the preceding layer. For example, if a larger activation value is received from the preceding layer, the weights are multiplied by the larger value and therefore have a greater effect on the perceptron's final activation value. The perceptron figuratively "hopes" that certain activations from the preceding layer are larger or smaller. These hopes can be backpropagated to the neurons in the preceding layer.

[0254] In the next layer, the neuron considers the desires from the next downstream layer when determining its own preferred activation value. Again, in this layer, the activation value is not hardcoded. Each neuron can adjust its own weights and biases and then backpropagate the changes to the activation value it desires to occur. Backpropagation continues layer by layer until the weights and biases of the first hidden layer are set. This layer cannot backpropagate the desired changes to the input layer because the input layer receives the activation value directly from the input image.

[0255] After such a round of approach, the network may undergo another round of training using the same or different training datasets, and the process is repeated until local and / or global minimums are found for the cost function.

[0256] Figure 10 is a flowchart of Method 1000 according to various embodiments. Method 1000 may be used to train a neural network such as the neural network 900 in Figure 9.

[0257] In block 1004, the network is initialized. Initially, the neural network 900 contains several neurons. Each neuron contains a transfer function or kernel. In the case of a neural network, each neuron contains parameters such as a biased value obtained by adding a weight to the weighted sum of the values ​​of each neuron from the previous layer. The final values ​​of a neuron may be normalized to values ​​between 0 and 1 using a function such as sigmoid or ReLU. Since an untrained neural network knows nothing about its problem space, and it is very difficult to manually program a neural network to perform a desired function, the parameters for each neuron may initially be set simply to some random values. For example, the values ​​may be selected using a pseudorandom number generator on the CPU and then assigned to each neuron.

[0258] In block 1008, the neural network is provided with a training set. In some cases, the training set may be divided into smaller groups. For example, if the training set has 100,000 objects, it may be divided into 1,000 groups, each containing 100 objects. These groups can then be used to incrementally train the neural network. In block 1008, the initial training set is provided to the neural network. Alternatively, the complete training set can be used in each iteration.

[0259] In block 1012, the training data is propagated through the neural network. Since the initial values ​​are random and therefore essentially garbage, it is expected that the output will also be garbage values. In other words, if the neural network 900 in Figure 9 is not trained, and the input image 904 is fed into the neural network, it is not expected that the output layer 920 will ignite the perceptron 4 using the first training set. Rather, the perceptron may have values ​​that are present throughout the map, without a clear winner and with very little relevance to number 4.

[0260] In block 1016, the cost function is calculated as described above. For example, in neural network 900, it is desirable that perceptron 4 has a value of 1.00 and each other perceptron has a value of 0.00. The difference between the desired value and the actual output value is calculated and squared. An individual cost function can be calculated for each training input, and the total cost function for the network can be calculated as the average of the individual cost functions.

[0261] In block 1020, the network may then calculate the negative gradient of the cost function to find its local minimum, or in other words, the error. For example, the system may use backpropagation to numerically determine the negative gradient. After calculating the negative gradient, the network may adjust the parameters (weights and biases) by some amount in the "downward" direction of the negative gradient.

[0262] After calculating the negative gradient, in decision block 1024, the system determines whether it has reached a local minimum (for example, whether the gradient has reached zero within the threshold). If it has not reached a local minimum, the neural network is not sufficiently trained, and control returns to block 1008 with a new training set. The training sequence continues in block 1024 until a local minimum is reached.

[0263] Once a local minimum is reached and the correction is backpropagated, the neural network is ready in block 1032.

[0264] Figure 11 is a flowchart of Method 1100. Method 1100 shows how to use a neural network, such as Network 900 in Figure 9, to classify objects.

[0265] In block 1104, the network extracts activation values ​​from the input data. For example, in the example in Figure 9, each pixel in the input image 904 is assigned as an activation value to neuron 908 in the input layer 912.

[0266] In block 1108, the network propagates activation values ​​from the current layer to the next layer in the neural network. For example, after activation values ​​have been extracted from the input image, these values ​​may be propagated to the first hidden layer of the network.

[0267] In block 1112, for each neuron in the current layer, the neuron calculates the sum of the weighted and biased activation values ​​received from each neuron in the previous layer. For example, in the example in Figure 9, neuron 0 in the first hidden layer is connected to each neuron in the input layer 912. The sum of the weighted values ​​is calculated from their activation values ​​and a bias is applied.

[0268] In block 1116, for each neuron in the current layer, the network normalizes the activation value by applying a function such as sigmoid, ReLU, or some other function.

[0269] In the determination block 1120, the network determines whether it has reached the final layer in the network. If it is not the final layer, control returns to block 1108, where the activation value in this layer is propagated to the next layer.

[0270] Returning to the determination block 1120, if the network is in the final layer, the neurons in this layer are perceptrons that provide the final output value for the object. At terminal 1124, the perceptrons are classified and used as output values.

[0271] Figure 12 is a block diagram showing selected elements of the analyzer engine 1204. The analyzer engine 1204 may be configured to provide analysis services via a neural network, etc. Figure 12 shows a platform for providing analysis services. Neural analysis and analysis of other machine learning models, etc., may be used in some embodiments to provide one or more features of this disclosure.

[0272] Although the analyzer engine 1204 is shown here as a single modular object, it should be noted that in some cases, different embodiments of the analyzer engine 1204 may be provided by separate hardware or by separate guests (e.g., VMs or containers) on a hardware system.

[0273] The analyzer engine 1204 includes the operating system 1208. Generally, the operating system 1208 is the Linux® operating system, but other operating systems such as Microsoft Windows®, Mac OS X®, UNIX®, etc., can be used. The analyzer engine 1204 also includes the Python® interpreter 1212, which can be used to run Python® programs. The Python® module known as Numerical Python® (NumPy) is often used for neural network analysis. While this is a popular choice, other non-Python® or non-NumPy systems can also be used. For example, neural networks can be implemented in Matrix Laboratory (MATLAB®), C, C++, Fortran, R, or any other compiled or interpreted computer language.

[0274] The GPU array 1224 may include an array of graphics processing units that can be used to perform the neural network functions of the neural network 1228. While GPU arrays are a popular choice for this type of processing, it should be noted that the neural network can also be implemented in a CPU, or in an ASIC or FPGA specifically designed to implement the neural network.

[0275] Neural Network 1228 includes the actual code for running the neural network, which is typically programmed in Python®, as mentioned above.

[0276] The results interpreter 1232 may include logic separate from the neural network functions that can be used to operate on the output of the neural network to assign objects for specific classifications, perform additional analysis, and / or provide recommended corrective actions.

[0277] The object database 1236 may include a database of known malware objects and their classifications. The neural network 1228 may initially be trained on objects in the object database 1236, and as new objects are identified, the object database 1236 may be updated with the results of additional neural network analysis.

[0278] Once the results are obtained, they may be sent to the appropriate destination via the network interface 1220.

[0279] Figure 13 is a block diagram of a circuit programming ecosystem according to various embodiments.

[0280] The circuit programming ecosystem 1300 includes a computing device 1302 and an accelerator circuit 1304. The computing device 1302 may be, for example, an engineering workstation or other suitable computing device to which the accelerator circuit 1304 is attached. In one example, the accelerator circuit 1304 is a Peripheral Component Interconnect Express (PCIe) card that extends the functionality of the computing device 1302, for example, by providing hardware acceleration for AI problems. In another example, the SoC may include both the computing device 1302 and the accelerator circuit 1304 in a closely coupled configuration (e.g., with a direct hardware connection), as shown in Figure 8 above. In yet another example, the computing device 1302 may be an orchestrator that manages a data center or cloud service. In that case, the accelerator circuit 1304 can be attached to a rack-mount server as a PCIe extension. Alternatively, the accelerator circuit 1304 can be part of a “thread” of a similar device in a rack-scale architecture. In that case, the thread may provide backplane connectivity to the network fabric, which may be, or include, some, non-limiting examples such as Intel® Omni-Path® Architecture (OPA), TrueScale®, Ultra Path Interconnect (UPI) (officially called QPI or KTI), FibreChannel, Ethernet®, FibreChannel over Ethernet® (FCoE), InfiniBand, PCI, PCIe, or optical fiber. Many other configurations are possible between the computing device 1302 and the accelerator circuit 1304.

[0281] The computing device 1302 includes a hardware platform 1308. An example of a hardware platform is provided in the SoC800 in Figure 8. Other hardware platforms can also be provided, and in general, any device with a suitable processor and memory (e.g., any "von Neumann machine") can be used for hardware platform 1308.

[0282] The computing device 1302 includes a communication driver 1312, which enables the computing device 1302 to communicate with the accelerator circuit 1304. The accelerator circuit 1304 may be any suitable circuit provided with a flexible or dynamic register file, as described throughout this specification. For example, the hardware circuit 100 in Figure 1 provides such an accelerator.

[0283] The computing device 1302 also includes programming software 1310. The programming software 1310 may include machine-executable instructions stored on one or more tangible, non-temporary, computer-readable storage media. When executed, these instructions instruct the hardware platform 1308 to perform a specific method, such as the method (or any part thereof) shown in Figure 14 below.

[0284] During use, an engineer or other user operates the programming software 1310 by selecting an appropriate per-layer register configuration for various layers of a known neural network. When selecting the register configuration, the programmer may consider factors such as data sparsity, tensor shape, and other factors that may affect the efficiency of register usage within the layer. In some cases, the programming software 1310 may include an application to assist the user in selecting an appropriate register size.

[0285] Several existing solutions have similar software to assist the user in discovering the optimal data size for a particular tensor within a layer, taking into account factors such as data stationarity, data sparsity, and tensor shape. However, these existing systems are limited to fixed register sizes provided by the circuit. For example, software may determine that 128 bytes is a preferred size for an IF tensor within a layer. However, if the accelerator circuit has a fixed 64-byte register, the software can allocate up to 64 bytes for the IF. The only option to obtain a larger register of 128 bytes was to reconfigure the circuit (e.g., reconfigure the FPGA) with a larger IF register. However, those register configurations were then fixed for the entire neural network. If less space was needed for the IF in a different layer, the excess capacity was wasted.

[0286] In contrast, the accelerator circuits of this specification may provide elastic registers, where the register size can be reconfigured at runtime on a layer-by-layer basis. In that case, the software may allow "borrowing" surplus capacity from other registers in the same register file, subject only to the resolution constraints of the register subbanks and, in some cases, the requirement that one or more subbanks can be "reserved" for each tensor as the minimum register size for that tensor.

[0287] Therefore, when interfaced with the accelerator circuits of this specification, the configuration software is free to allocate larger registers for specific tensors. The software may do this by borrowing from subbanks of other registers within the same register file if a particular layer requires a larger data size for a particular tensor.

[0288] After the user has made a final decision on the layer-level register file selection, the programming software 1310 may operate the communication driver 1312 to transmit the NN input and layer-level register configuration to the accelerator circuit 1304.

[0289] The accelerator circuit 1304 receives NN inputs and layer-level register configurations to the SRAM. These data may be used to program the glue logic 1318, which tracks active layer and inter-layer data propagation. The glue logic 1318 may use the layer-level register configuration to program the configuration register 1320 using the register configuration for the active layer of the NN.

[0290] The configuration register 1320 programs the flexible register 1328 with a desired register configuration for the active layer. For example, appropriate values ​​may be provided to the multiplexer and / or demultiplexer, as shown in Figure 5.

[0291] Once the appropriate data available in SRAM1316 and the desired register configuration is applied to the flexible register 1328, the PE bank 1324 can then perform mathematical operations on the layers, for example, by performing multiple parallel MAC operations.

[0292] Figure 14 is a flowchart of method 1400 for programming hardware circuits according to various embodiments. Method 1400 may be performed entirely or partially by a computing device such as the computing device 1300 in Figure 13, or by any other suitable device.

[0293] In block 1404, the device receives input data for an AI problem that can be solved by a neural network, such as by a DNN accelerator circuit, as described throughout this specification.

[0294] In block 1408, the operator determines the tensor shape, data sparsity, data stationarity, and other relevant information for each layer in the DNN. These factors influence the preferred register file size for each layer.

[0295] In block 1412, the user determines the preferred register configuration for each layer according to the input received in block 1408. In some cases, computer software may assist the user in determining the preferred register configuration, for example, by providing hints or suggestions for a particular layer.

[0296] In block 1416, the system transmits a configuration to an AI accelerator circuit, such as the hardware circuit 100 in Figure 1 or any other suitable circuit. This may include flashing a ROM, transmitting data to flash memory or any other SRAM, or performing any other action to load appropriate data into the accelerator circuit.

[0297] In block 1420, the system starts the accelerator circuit by applying power or by sending a "start" signal to the circuit, etc. The accelerator circuit then performs DNN inference computations in hardware, including using the provided layer-level register configuration.

[0298] In block 1424, the system receives inference results from the DNN via the accelerator circuit. The user may then apply the results as needed.

[0299] The method is completed in block 1490. Transformations in implementation

[0300] The foregoing briefly describes some features of embodiments so that those skilled in the art may better understand various aspects of the present disclosure. The preceding detailed description describes examples of apparatus, methods, and systems relating to a system for runtime configuration of a register file according to one or more embodiments of the present disclosure. For example, features such as structure, function, and / or characteristics are described in relation to one embodiment for convenience; various embodiments may be implemented using any suitable one or more of the described features.

[0301] Where used throughout this specification, the phrase “one embodiment” is intended to refer to one or more embodiments. Furthermore, different uses of the phrase “one embodiment” may refer to different embodiments. The phrases “in another embodiment” or “in a different embodiment” refer to an embodiment different from the embodiments described previously, or the same embodiment having additional features. For example, “In one embodiment, features may exist. In another embodiment, additional features may exist.” The above examples may, firstly, refer to an embodiment having features A, B, and C, while secondly, refer to an embodiment having features A, B, C, and D, an embodiment having features A, B, and D, an embodiment having features D, E, and F, or any other modification.

[0302] In the foregoing description, various aspects of the exemplary implementation may be described using terminology commonly used by those skilled in the art to communicate the substance of their research to others skilled in the art. Those skilled in the art will see that the embodiments disclosed herein may be carried out using only some of the described embodiments. For explanatory purposes, certain numbers, materials, and configurations are described to provide a complete understanding of the exemplary implementation. In some cases, the disclosed embodiments may be carried out without specific details. In other instances, well-known features are omitted or simplified so as not to obscure the illustrated embodiments.

[0303] For the purposes of this disclosure and the attached claims, the article "a" refers to one or more items. The phrase "A or B" is intended to encompass an "inclusive OR," e.g., A, B, or (A and B). "A and / or B" means A, B, or (A and B). For the purposes of this disclosure, the phrase "A, B, and / or C" means A, B, C, (A and B), (A and C), (B and C), or (A, B, and C).

[0304] The embodiments disclosed may readily be used as a basis for designing or modifying other processes and structures to carry out the teachings herein. Any equivalent structure to those disclosed shall not depart from the spirit and scope of this disclosure. Design considerations may result in alternative arrangements, design options, device possibilities, hardware configurations, software implementations, and equipment options.

[0305] As used throughout this Spec., “memory” is expressly intended to include both volatile and non-volatile memory. Therefore, for example, an “engine” as described above may include instructions encoded in volatile or non-volatile memory that, when executed, instruct a processor to perform any of the methods or procedures disclosed herein. This configuration is expressly intended to correspond to a computing device that is “sitting on a shelf” in a non-operating state. For example, in this example, “memory” may include one or more tangible, non-temporary, computer-readable storage media for storing the stored instructions. These instructions, in combination with a hardware platform (including a processor) on which they are stored, may constitute a computing device.

[0306] In other embodiments, the computing device may also correspond to an operating device. For example, in this configuration, “memory” may include volatile or runtime memory (e.g., RAM) on which instructions have already been loaded. When these instructions are fetched and executed by the processor, they may provide methods or procedures as described herein.

[0307] In yet another embodiment, there may be one or more tangible non-temporary computer-readable storage media storing executable instructions that, when executed, cause a hardware platform or other computing system to perform a method or procedure. For example, the instructions may be executable object code, including software instructions that can be executed by a processor. One or more tangible non-temporary computer-readable storage media may, in exemplary and non-limiting examples, include magnetic media (e.g., hard drives), flash memory, ROM, optical media (e.g., CD, DVD, Blu-ray®), non-volatile random access memory (NVRAM), non-volatile memory (NVM) (e.g., Intel 3D Xpoint), or other non-temporary memory.

[0308] This specification also provides specific methods, for example, as shown in flowcharts and / or signal flow diagrams. The order or actions disclosed in these methods disclose exemplary orderings that may be used in some embodiments, but this ordering is not intended to be restrictive unless expressly stated otherwise. In other embodiments, actions may be performed in other logical orders. In general, one action should be considered to necessarily precede another action only if the first action provides the result required to perform the second action. Furthermore, the sequence of actions itself should be understood as a non-limiting example. In suitable embodiments, some actions may be omitted as unnecessary or undesirable. In the same or different embodiments, other actions not shown may be included in the method to provide additional results.

[0309] In certain embodiments, some of the components shown herein may be omitted or integrated. In a general sense, the arrangements shown in the figures may be more logical in their representation, while the physical architecture may include various permutations, combinations, and / or hybrids of these elements.

[0310] Interactions can be described in terms of two, three, four, or more electrical components using the numerous examples provided herein. These descriptions are provided for clarity and illustrative purposes only. Any of the components, modules, and elements shown in the figures can be combined in various configurations, all of which fall within the scope of this specification.

[0311] In certain cases, it may be easier to describe one or more functions by disclosing only selected elements. Such elements are selected to provide specific information to facilitate the description. The inclusion of an element in a figure is not intended to suggest that the element must appear in this disclosure as claimed, and the exclusion of a particular element from a figure is not intended to suggest that the element is excluded from this disclosure as claimed. Similarly, any method or flow shown herein is provided merely as an example. Including or excluding actions in such a method or flow should be understood as the same as including or excluding other elements as described in this paragraph. Where actions are shown in a particular order, that order is merely a non-limiting example. Unless expressly specified, the order of actions may be modified to suit a particular embodiment.

[0312] Other changes, substitutions, modifications, alternatives, and alterations will be apparent to those skilled in the art. All such changes, substitutions, modifications, alternatives, and alterations fall within the scope of this specification.

[0313] To assist the United States Patent and Trademark Office (USPTO) and any reader of any patent or publication derived herein, the applicant does not intend to exercise any of the appended claims or their equivalents as existing on the filing date of the invention under Section 112(f) of the United States Patent Act unless the terms “means” or “steps” are specifically used in a particular claim; and (b) nothing by any reference herein is intended to limit the disclosure in any way not expressly reflected in the appended claims as originally presented or amended. (Other possible items) (Item 1) The step of generating multiple layer-specific register schedules for a deep learning neural network, wherein at least two layer-specific register schedules are different from each other, the layer-specific register schedules are for dividing a register file into multiple tensor-specific registers, the register file includes multiple discrete subbanks, and each tensor-specific register includes one or more of the subbanks; and The step of programming the artificial intelligence (AI) hardware circuit using the aforementioned multi-layer-specific register schedules. A method comprising a programming step comprising a step of programming a configured register to provide the layer-specific register schedule. (Item 2) The method according to item 1, wherein the plurality of tensor-specific registers include registers for input features (IF), output features (OF), and filter weights (FL). (Item 3) The method according to item 1, wherein the layer-specific register schedule is for multiple register files, and the schedule for the multiple register files is the same within the layer. (Item 4) The register file is associated with each processing element of the AI ​​hardware circuit, as described in item 3. (Item 5) The method according to item 1, wherein the step of generating a layer-specific register schedule includes providing smaller registers for tensors having sparse data in the layer compared to tensors having non-sparse data in the layer. (Item 6) The method according to item 1, wherein the step of generating a layer-specific register schedule includes a step of providing surplus capacity for tensors having high stationarity within the layer. (Item 7) The method according to item 1, wherein the step of generating a layer-specific register schedule includes a step of considering the tensor shape within the layer. (Item 8) Multiple processing element (PE) circuits that provide one or more neuronal layers for a neural network; Multiple register files are communicably coupled to and associated with each of the circuits of the PE circuit, wherein each register file has a circuit for storing multiple types of data, and each of them is C TOT Having a total capacity of bytes, the C TOT Each byte is divided into a subbank of B bytes, where C TOT B is an integer, and the subbank has an input and output multiplexer circuit configured to selectively assign the subbank to a selected input or output of the PE, where the input or output represents multiple types of data; and A control circuit configured to change subbank assignments according to the active layer of the neural network during runtime. A device equipped with the following features. (Item 9) The apparatus described in item 8, wherein the PE circuits are substantially identical to each other in hardware. (Item 10) The PE circuit is a multiply-accumulate (MAC) unit, as described in item 8. (Item 11) The apparatus according to item 8, wherein the control circuit has an input multiplexer and an output demultiplexer for each of the subbanks. (Item 12) The apparatus described in item 8, which includes data for at least two species, including data for three species. (Item 13) The apparatus described in item 12 includes the three types of data described above, including input features (IF), output features (OF), and filter weights (FL). (Item 14) The apparatus according to item 13, wherein the register file includes at least one dedicated subbank for each of the three types of data. (Item 15) The apparatus as described in item 14, wherein the dedicated subbank does not have input and output multiplexers. (Item 16) B is between 1 and 128, the apparatus described in item 8. (Item 17) The apparatus according to item 8, wherein the data of the aforementioned type includes an input tensor or an output tensor for the neural network. (Item 18) The apparatus according to item 8, wherein the control circuit further includes a stored layer-level register configuration for the register file. (Item 19) The apparatus described in item 18, wherein the layer-level register configuration takes into account the data sparsity and data stationarity within each layer of the neural network. (Item 20) The apparatus described in item 18, wherein the layer-level register configuration takes into account the tensor dimensions within each layer of the neural network. (Item 21) One or more tangible, non-temporary, computer-readable media storing instructions that constitute a deep neural network (DNN) accelerator circuit, wherein the instructions are: To generate a plurality of layer-specific register schedules for the DNN accelerator circuit, wherein at least two of the layer-specific register schedules are distinct from each other, and the layer-specific register schedules are for dividing a register file into a plurality of tensor-specific registers, the register file comprising a plurality of discrete subbanks, and each of the tensor-specific registers comprising one or more of the subbanks; Transmitting the aforementioned layer-specific register schedules, along with the deep learning problem, to a neural network hardware accelerator; and To instruct the DNN accelerator circuit to start execution. One or more tangible, non-temporary computer-readable media comprising: (Item 22) The plurality of tensor-specific registers include registers for input features (IF), output features (OF), and filter weights (FL), and are contained in one or more tangible, non-transient, computer-readable media as described in item 21. (Item 23) The layer-specific register schedule is for a plurality of register files, and the schedule for the plurality of register files is the same within the layer, one or more tangible non-temporary computer-readable media as described in item 21. (Item 24) The register file is one or more tangible, non-temporary computer-readable media as described in item 23, associated with each processing element of the neural network accelerator circuit. (Item 25) One or more tangible non-temporary computer-readable media as described in item 21, wherein generating a layer-specific register schedule provides smaller registers for a tensor having sparse data in a layer compared to a tensor having non-sparse data in the layer. (Item 26) One or more tangible, non-transient computer-readable media as described in item 21, wherein generating a layer-specific register schedule provides surplus capacity for tensors having high stationarity within the layer. (Item 27) The generation of a layer-specific register schedule comprises considering the tensor shape within the layer, in one or more tangible, non-transient computer-readable media as described in item 21.

Claims

1. The steps include generating multiple layer-specific register schedules for a deep learning neural network, wherein at least two layer-specific register schedules are distinct from one another, the layer-specific register schedules are for dividing a register file into multiple tensor-specific registers, the register file each containing multiple discrete subbanks of B bytes, where B is an integer, and each tensor-specific register containing one or more of the discrete subbanks; and The step of programming the artificial intelligence (AI) hardware circuit using the aforementioned multi-layer-specific register schedules. A method comprising a programming step comprising a step of programming a configured register to provide the layer-specific register schedule.

2. The method according to claim 1, wherein the plurality of tensor-specific registers include registers for input features (IF), output features (OF), and filter weights (FL).

3. The method according to claim 1, wherein the layer-specific register schedule is for a plurality of register files, and the layer-specific register schedule for the plurality of register files is the same within the layer.

4. The method according to claim 3, wherein the register file is associated with each processing element of the AI ​​hardware circuit.

5. The step of generating multiple layer-specific register schedules for a deep learning neural network, wherein at least two layer-specific register schedules are different from each other, the layer-specific register schedules are for dividing a register file into multiple tensor-specific registers, the register file comprises multiple discrete subbanks, and each tensor-specific register comprises one or more of the discrete subbanks; and The step of programming the artificial intelligence (AI) hardware circuit using the aforementioned multi-layer-specific register schedules. The programming step includes a step of programming a configuration register to provide the layer-specific register schedule, A method comprising the step of generating a layer-specific register schedule, wherein the step of providing smaller registers for tensors having sparse data in a layer compared to tensors having non-sparse data in the layer.

6. The step of generating multiple layer-specific register schedules for a deep learning neural network, wherein at least two layer-specific register schedules are different from each other, the layer-specific register schedules are for dividing a register file into multiple tensor-specific registers, the register file comprises multiple discrete subbanks, and each tensor-specific register comprises one or more of the discrete subbanks; and The step of programming the artificial intelligence (AI) hardware circuit using the aforementioned multi-layer-specific register schedules. The programming step includes a step of programming a configuration register to provide the layer-specific register schedule, A method comprising the step of generating a layer-specific register schedule, which includes the step of providing surplus capacity for tensors with high stationarity within the layer.

7. The step of generating multiple layer-specific register schedules for a deep learning neural network, wherein at least two layer-specific register schedules are different from each other, the layer-specific register schedules are for dividing a register file into multiple tensor-specific registers, the register file comprises multiple discrete subbanks, and each tensor-specific register comprises one or more of the discrete subbanks; and The step of programming the artificial intelligence (AI) hardware circuit using the aforementioned multi-layer-specific register schedules. The programming step includes a step of programming a configuration register to provide the layer-specific register schedule, A method for generating a layer-specific register schedule, comprising the step of considering the tensor shape within the layer.

8. Multiple processing element (PE) circuits that provide one or more neuronal layers for a neural network; Multiple register files are communicably coupled to and associated with each of the circuits of the PE circuit, wherein each register file has a circuit for storing multiple types of data, and each of them is C ＴＯＴ Having a total capacity of bytes, the C ＴＯＴ Each byte is divided into a subbank of B bytes, and here, C ＴＯＴ and B are integers, and the subbank has an input and output multiplexer circuit configured to selectively assign the subbank to a selected input or output of the PE, where the input or output represents multiple types of data; and A control circuit configured to change the assignment of the subbanks according to the active layer of the neural network during runtime. A device equipped with the following features.

9. The apparatus according to claim 8, wherein the PE circuits are identical to each other in hardware.

10. The apparatus according to claim 8, wherein the PE circuit is a multiply-accumulate (MAC) unit.

11. The apparatus according to claim 8, wherein the control circuit has an input multiplexer and an output demultiplexer for each of the subbanks.

12. The apparatus according to claim 8, wherein the data of the plurality of species includes data of three species.

13. The apparatus according to claim 12, wherein the three types of data include input features (IF), output features (OF), and filter weights (FL).

14. The apparatus according to claim 13, wherein the register file includes at least one dedicated subbank for each of the three types of data, and the dedicated subbank does not have input and output multiplexers.

15. The apparatus according to claim 8, wherein B is between 1 and 128.

16. The apparatus according to claim 8, wherein the data of the aforementioned type includes an input tensor or an output tensor for the neural network.

17. The apparatus according to any one of claims 8 to 16, wherein the control circuit further includes a stored layer-level register configuration for the register file.

18. The apparatus according to claim 17, wherein the layer-level register configuration takes into account the data sparsity and data stationarity within each layer of the neural network, and the layer-level register configuration takes into account the tensor dimension within each layer of the neural network.

19. A computer program comprising instructions for a deep neural network (DNN) accelerator circuit, wherein the instructions are: To generate a plurality of layer-specific register schedules for the DNN accelerator circuit, wherein at least two of the layer-specific register schedules are distinct from each other, and the layer-specific register schedules are for dividing a register file into a plurality of tensor-specific registers, the register file comprising a plurality of discrete subbanks, each of which is B bytes, where B is an integer, and each of the tensor-specific registers comprising one or more of the discrete subbanks; Transmitting the aforementioned layer-specific register schedules, along with the deep learning problem, to a neural network hardware accelerator; and To instruct the DNN accelerator circuit to start execution. A computer program that includes the following features.

20. The computer program according to claim 19, wherein the plurality of tensor-specific registers include registers for input features (IF), output features (OF), and filter weights (FL).

21. The computer program according to claim 19, wherein the layer-specific register schedule is for a plurality of register files, and the layer-specific register schedule for the plurality of register files is the same within the layer.

22. The computer program according to claim 21, wherein the register file is associated with each processing element of the DNN accelerator circuit.

23. A computer program comprising instructions for a deep neural network (DNN) accelerator circuit, wherein the instructions are: To generate a plurality of layer-specific register schedules for the DNN accelerator circuit, wherein at least two of the layer-specific register schedules are distinct from each other, and the layer-specific register schedules are for dividing a register file into a plurality of tensor-specific registers, the register file comprising a plurality of discrete subbanks, and each of the tensor-specific registers comprising one or more of the discrete subbanks; Transmitting the aforementioned layer-specific register schedules, along with the deep learning problem, to a neural network hardware accelerator; and To instruct the DNN accelerator circuit to start execution. Equipped with, A computer program that generates layer-specific register schedules, thereby providing smaller registers for tensors having sparse data in a layer compared to tensors having non-sparse data in the layer.

24. A computer program comprising instructions for a deep neural network (DNN) accelerator circuit, wherein the instructions are: To generate a plurality of layer-specific register schedules for the DNN accelerator circuit, wherein at least two of the layer-specific register schedules are distinct from each other, and the layer-specific register schedules are for dividing a register file into a plurality of tensor-specific registers, the register file comprising a plurality of discrete subbanks, and each of the tensor-specific registers comprising one or more of the discrete subbanks; Transmitting the aforementioned layer-specific register schedules, along with the deep learning problem, to a neural network hardware accelerator; and To instruct the DNN accelerator circuit to start execution. Equipped with, A computer program that generates layer-specific register schedules, which provides surplus capacity for tensors with high stationarity within a layer.

25. A computer program comprising instructions for a deep neural network (DNN) accelerator circuit, wherein the instructions are: To generate a plurality of layer-specific register schedules for the DNN accelerator circuit, wherein at least two of the layer-specific register schedules are distinct from each other, and the layer-specific register schedules are for dividing a register file into a plurality of tensor-specific registers, the register file comprising a plurality of discrete subbanks, and each of the tensor-specific registers comprising one or more of the discrete subbanks; Transmitting the aforementioned layer-specific register schedules, along with the deep learning problem, to a neural network hardware accelerator; and To instruct the DNN accelerator circuit to start execution. Equipped with, A computer program that generates layer-specific register schedules involves considering the tensor shapes within the layers.

26. A tangible, non-temporary computer-readable medium for storing a computer program according to any one of claims 19 to 25.

Images