Byte stream processing pipeline for hardware integrated circuits

Abstract

A method and system including a computer-readable medium describes a byte-stream processing pipeline using the instruction set architecture of a hardware integrated circuit. The byte-stream pipeline is used to access the memory banks of a computation tile in the circuit based on instructions executed on the computation tile. Opcodes are identified within the instructions. Opcodes indicate the operation to access the memory bank of memory on the tile. From the opcodes, a byte-addressing sequence is generated to access contiguous bytes of different lengths stored in the bank. Based on the byte-addressing sequence, multiple byte-addressable memory accesses are performed in memory to retrieve multiple input vectors from the memory bank.

Description

Technical Field

[0004] , , , ,

[0001] This specification generally relates to memory operations of hardware integrated circuits.

Background Art

[0002] A neural network is a machine learning model that uses one or more layers of nodes to generate an output, such as a classification, for received input. Some neural networks include one or more hidden layers in addition to an output layer. Some neural networks can be convolutional neural networks (CNNs) configured for image processing or recurrent neural networks (RNNs) configured for speech and language processing. Various types of neural network architectures can be used to perform various tasks related to classification or pattern recognition, prediction involving data modeling, and information clustering.

[0003] A neural network layer can have a corresponding set of parameters or weights. The weights are used to process an input (e.g., a batch of inputs) through the neural network layer and generate a corresponding output of the layer for computing neural network inference. The batch of inputs and the set of kernels can each be represented as tensors of inputs and weights, i.e., multi-dimensional arrays. A hardware accelerator is an application-specific integrated circuit for implementing a neural network. This circuit includes memory having locations corresponding to elements of the tensors that can be traversed or accessed using the control logic of the circuit.

Summary of the Invention

[0004] This document describes enhanced instruction set configurations that can be executed by a processor or hardware accelerator to provide efficient byte-level data streaming processes. The disclosed technologies and instruction sets also enable corresponding synchronization of those streaming processes for a variety of computationally intensive operations, including various machine learning (ML) workloads for inference or predictive operations.

[0005] One aspect of the subject matter described herein can be embodied in a method performed by a computer for accessing memory banks of a hardware accelerator. The method includes receiving an instruction for a compute tile of the hardware accelerator. The instruction is executable on the compute tile and thereby causes the execution of an operation which includes i) in the instruction identifying an opcode for an operation involving accessing one or more banks of a first memory of the compute tile; ii) based on the opcode generating a byte-addressing sequence used to access contiguous bytes of different lengths stored in one or more banks; iii) based on the byte-addressing sequence, performing a plurality of byte-addressable memory accesses in the first memory to obtain a plurality of input vectors from one or more banks of the first memory; and iv) based on the instruction, processing each of the plurality of input vectors in the hardware accelerator.

[0006] These and other embodiments may each optionally include one or more of the following features. For example, in some embodiments, a byte addressing sequence can be operated to access contiguous bytes of varying lengths starting from any given byte address across one or more banks of a first memory. The method includes generating a byte access request that provides byte-granularity access to retrieve at least one byte of data stored in a given row of one of the banks of the first memory, based on an opcode.

[0007] In some embodiments, generating a byte addressing sequence involves generating a byte addressing sequence corresponding to multiple byte access requests, each byte access request used to access a distinct portion of byte-level data stored at each address location across one or more banks of a first memory. In some cases, each byte access request can be used to access each distinct portion of byte-level data stored at each address location. The opcodes may be for tensor operations performed on the computation tile to traverse each element of the input tensor.

[0008] In some embodiments, the method further includes i) performing a plurality of byte access requests using a byte addressing sequence; ii) performing a plurality of byte-granular accesses in a first memory based on the plurality of byte access requests performed; and iii) traversing each element of the input tensor by performing the plurality of byte-granular accesses in the first memory. In some embodiments, each distinct portion of byte-level data stored at each address location across one or more banks of the first memory corresponds to each element of the input tensor, and each address location storing a distinct portion of byte-level data is mapped to the corresponding element of the input tensor.

[0009] In some embodiments, a portion of the byte-level data corresponds to one of several input vectors, and the multiple byte-addressable memory accesses used to obtain the multiple input vectors correspond to at least one of i) multiple byte-granularity accesses, or ii) multiple byte access requests. In some embodiments, processing the multiple input vectors includes processing the multiple input vectors through a neural network layer of a neural network implemented in a hardware accelerator, and generating an output of the neural network layer in response to processing the multiple input vectors through the neural network layer.

[0010] This aspect and other embodiments of other aspects include corresponding systems, devices, and computer programs configured to perform actions in a manner encoded on a computer storage device. One or more computer systems can be configured in this way by software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform actions when in operation. One or more computer programs can be configured in this way by having instructions that cause the device to perform actions when executed by a data processing device.

[0011] The subject matter described herein can be implemented in particular embodiments to achieve one or more of the following advantages:

[0012] Unlike previous methods for data access, which were limited to row-addressing schemes for accessing physical memory, the disclosed technology provides an instruction set architecture (ISA) that enables enhanced byte-level addressing capabilities for memory, such as on-chip static random access memory (SRAM), in exemplary dedicated integrated circuits (e.g., hardware accelerators including tensor processing units (TPUs) or vector processing units (VPUs), graphics processing units (GPUs), or neural network (NN) processors). The dedicated integrated circuit may also be a hardware accelerator or neural network processor that uses a particular type of memory structure to store inputs for ML workloads.

[0013] Existing ISAs have data padding and access alignment requirements, which can make programming and debugging difficult. In contrast, the byte addressing provided by the improved instruction set in this document allows local access to contiguous bytes of varying lengths from any given byte address in the input memory of the computation tile. The techniques and systems disclosed herein can provide novel combinations of byte access patterns that support ML computations. Such data access patterns enable various types of fine-grained stride operations for ML image processing operations, as well as spatial and temporal data reuse to support ML inference and training workloads for various data science applications.

[0014] The improved ISA includes one or more “ByteAddressingMode” instructions having an integrated field that enables automatic zero-padding when accessing data from the input memory of a compute tile. For example, based on the disclosed instruction set, a compute tile processor can issue a byte access request that provides byte-granular access to retrieve at least one byte of data stored in a given row of a physical memory bank in one or more banks of a first memory. Thus, a single byte can be retrieved without requiring irrelevant data padding, such as padding the request with other bytes, and without requiring access alignment across a particular set of rows or banks.

[0015] Details of one or more embodiments of the subject matter described herein are given in the accompanying drawings and the following description. Other potential features, aspects and advantages of the subject matter will become apparent from the description, drawings and claims. [Brief explanation of the drawing]

[0016] [Figure 1] This is a block diagram of an exemplary computing system for implementing a neural network machine learning model. [Figure 2] This illustrates an exemplary processing pipeline for routing inputs and outputs between the memory and compute cells of a hardware integrated circuit. [Figure 3] The exemplary field and message structure of an exemplary instruction set is shown. [Figure 4] The following is code for an example byte addressing operation in the instruction set. [Figure 5] An example of a byte addressing operation is shown. [Figure 6] This is an exemplary process of byte stream processing using exemplary byte addressing operations in the instruction set. [Figure 7] Examples of input tensors, parameter tensors, and output tensors are shown. [Modes for carrying out the invention]

[0017] Similar reference symbols and designations in various drawings refer to the same elements.

[0018] Figure 1 is a block diagram of an exemplary computing system 100 for implementing a neural network model on a hardware integrated circuit, such as a machine learning hardware accelerator. The computing system 100 includes one or more computing tiles 101, a host 120, and a higher-level controller 125 ("controller 125"). As will be described in more detail below, the host 120 and the controller 125 work together to provide datasets and instructions to one or more computing tiles 101 of the system 100.

[0019] In some embodiments, the host 120 and the controller 125 are the same device. The host 120 and the controller 125 can perform separate functions, or they can be integrated into a single device package. For example, the host 120 and the controller 125 can form a central processing unit (CPU) that interacts with or collaborates with a hardware accelerator having multiple compute tiles 101. In some embodiments, the host 120, the controller 125, and the multiple compute tiles 101 are contained as different sections on a single integrated circuit die, or formed on that die. For example, the host 120, the controller 125, and the multiple compute tiles 101 can form a dedicated system-on-a-chip (SoC) optimized for running neural network models to process machine learning workloads.

[0020] Each computation tile 101 generally includes a controller 103, which provides one or more control signals 105 to cause the input (or activation) of the input vector 102 to be stored in a memory location in a first memory 108 ("memory 108") or to be accessed from that memory location. Similarly, the controller 103 may also provide one or more control signals 105 to cause the weights (or parameters) of the matrix structure of the weights 104 to be stored in a memory location in a second memory 110 ("memory 110") or to be accessed from that memory location. In some examples, memory 108 is identified as "narrow" and memory 110 is identified as "wide". In some embodiments, the input vector 102 is obtained from an input tensor, while the matrix structure of the weights is obtained from a parameter tensor. The input tensor and the parameter tensor, respectively, may be multidimensional data structures such as multidimensional matrices or tensors. This will be described in more detail below with reference to Figure 7.

[0021] Each memory location of memories 108 and 110 can be identified by a corresponding memory address, such as a logical address having a corresponding mapping to a physical row of a physical memory bank of the memory. The compute tile 101 can derive a set of consecutive addresses (e.g., virtual / logical addresses) from a group of requests. For example, the set of consecutive addresses can be derived by referring to the logical memory corresponding to the physical memory 108 of the compute tile 101.

[0022] The logical memory can have a plurality of logical ports, and each port can be connected to or associated with different requesters that request access to the physical resources of the memory 108. To process a given access request, for each port, the tile 101 (or its controller 103) determines the bank to which the request is to be routed based on the address within the request.

[0023] For each bank, the compute tile 101 can include an arbiter that coordinates access to that bank from multiple ports according to a bank generation function that is uniquely configured, for example, to reduce some (or all) of the requests routed to the same physical memory bank. The logical memory, the ports of the logical memory, and the arbiter can be implemented in software, hardware, or both. In some embodiments, the logical memory and its ports, as well as the arbiter, are controlled based on control signals generated by the controller 103.

[0024] Each of memories 108 and 110 can be implemented as a series of physical banks, units, or any other relevant storage medium or device. Each of memories 108 and 110 can include one or more registers, buffers, or both. In some embodiments, memory 108 is an input / activation memory, while memory 110 is a parameter memory. In some other embodiments, the input or activation is stored in memory 108, memory 110, or both, and the weights are stored in memory 110, memory 108, or both. For example, the input and weights can be transferred between memory 108 and memory 110 to facilitate certain neural network calculations. In some embodiments, each of memories 108 and 110 is referred to as tile memory.

[0025] Each computing tile 101 also includes an input activation bus 106, an output activation bus 107, and a computing unit 112, and the computing unit has one or more hardware multiply-accumulate circuits (MACs) in each cell 114a / b / c. The controller 103 can generate a control signal 105 to obtain the operands stored in the memory of the computing tile 101. For example, the controller 103 can generate a control signal 105 to obtain i) an exemplary input vector 102 stored in the memory 1 _ 08 and ii) the weights 104 stored in the memory 110. Each input obtained from the memory 108 is provided to the input activation bus 106 for routing (e.g., directly routing) to the computing cells 114a / b / c within the computing unit 112. Similarly, each weight obtained from the memory 110 is routed to the cells 114a / b / c of the computing unit 112.

[0026] As described below, each cell 114a / b / c performs a computation to obtain a partial sum or cumulative value to generate the output of a given neural network layer. An activation function can be applied to the set of outputs to generate a set of output activations for the neural network layer. In some embodiments, the outputs or output activations are routed through an output activation bus 107 for storage and / or transfer. For example, a set of output activations can be transferred from the first computation tile 101 to a second different computation tile 101 to be processed as input activations to different layers of the neural network at the second computation tile 101.

[0027] Generally, each computation tile 101 and system 100 may include additional hardware structures to perform computations associated with multidimensional data structures such as tensors, matrices, and / or data arrays. In some embodiments, input vectors (or tensors) 102 and weights 104 of parameter tensors can be preloaded into the memories 108, 110 of the computation tile 101. The inputs and weights are received as a set of data values ​​that reach a particular computation tile 101 from a host 120 (e.g., an external host) via a host interface, or from a higher-level control such as a controller 125.

[0028] Each of the compute tile 101 and controller 103 may include one or more processors, processing devices, and various types of memory. In some embodiments, the processors of the compute tile 101 and controller 103 include one or more devices such as a microprocessor or central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuit (ASIC), or a combination of different processors. Each of the compute tile 101 and controller 103 may also include other computing and storage resources such as buffers, registers, and control circuits. These resources cooperate to provide additional processing options for performing one or more of the decisions and calculations described herein.

[0029] In some embodiments, the processing unit(s) of the controller 103 execute programmed instructions stored in memory to cause the controller 103 and the compute tile 101 to perform one or more of the functions described herein. The memory of the controller 103 may include one or more non-temporary machine-readable storage media. Non-temporary machine-readable storage media may include solid-state memory, random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (e.g., EPROM, EEPROM, or flash memory), or any other tangible media capable of storing information or instructions.

[0030] System 100 receives instructions that define specific computational operations performed by computational tiles 101. System 100 also receives data such as inputs, activations, and weights (or parameters) that are operands for the computational operations. System 100 may include one or more buses 109 used to route instructions and data. For example, System 100 may include a first bus 109-1 that provides instructions, including the relevant commands, opcodes, and operation parameters (not weights), to each of the one or more computational tiles 101 of System 100. System 100 may also include a second bus 109-2 that provides data or operands to each of the one or more computational tiles 101 of System 100.

[0031] Bus 109 can be configured to provide one or more interconnected data communication paths between two or more compute tiles of system 100. For example, a first bus 109-1 may be a ring bus that traverses each compute tile to deliver a dataset and a single instruction (or multiple instructions) to one or more compute tiles to execute an ML workload, while a second bus 109-2 may be a mesh bus that interconnects two or more tiles to provide data or operand sets between two or more compute tiles 101.

[0032] In some embodiments, buses 109-1 and 109-2 are separate data buses. In some other embodiments, buses 109-1 and 109-2 are the same data bus. In some cases, to handle an exemplary workload, the first compute tile 101 coordinates and executes access requests (e.g., read / write requests) to memory 108, where the requests are based on external data communications originating from outside the first compute tile 101, such as from a second different compute tile 101. Such external communications may be received by the first tile 101 via bus 109-2 (e.g., a mesh bus).

[0033] Each computation tile 101 is an individual computing unit that collaborates with other tiles 101 in the system 100 to accelerate computation across one or more layers of a multilayer neural network or across one or more sections of other machine learning structures. Each computation tile 101 can function as an individual computing unit. For example, each computation tile 101 is an autonomous computing component configured to perform a subset of tensor or ML computations independently of other computation tiles. In some embodiments, computation tiles 101 can also share the execution of tensor computations related to a given instruction.

[0034] In some embodiments, the host can generate a set of parameters (i.e., weights) and corresponding inputs for processing in a neural network layer. For example, the host can generate a set of compressed parameters (CSPs) and corresponding mapping vectors (e.g., non-zero maps (NZMs)) that map neural network inputs to non-zero parameters in the set of CSPs for a given operation. The host 120 can send parameters to the computation tile 101 via the host interface for further processing at the computation tile 101. The controller 103 can execute programmed instructions to analyze the received data stream associated with the weights and inputs, including the compressed parameters and corresponding mapping vectors.

[0035] The controller 103 ensures that the inputs and weights of the data stream are stored in the compute tile 101. For example, the controller 103 can store inputs including any associated mapping vectors and weights / parameters including any associated compressed sparse parameters in the local tile memory of the compute tile 101. This will be explained in more detail below. The controller 103 can also analyze the input data stream to detect operation codes ("opcodes"). The system 100 can support various types of opcodes, such as vector-matrix multiplication, element-wise vector operations, and opcode types that indicate whether a given operation uses compressed sparse parameters / weights versus uncompressed parameters / weights.

[0036] Based on one or more opcodes, the controller 103 may activate or execute a bank generation function to coordinate requests for accessing the tile memory of the compute tile 101. For example, the controller 103 may leverage the bank generation function to coordinate two or more requests so that each of the two or more requests is processed for a different physical bank of the tile memory. The controller 103 may employ a predetermined bank selection scheme programmed or encoded in the controller 103 before the inference decision is executed on the compute tile 101.

[0037] In some embodiments, a given computational operation involves multiple requesters, each requesting access to a resource in memory 108. For example, a computational workload performed on computational tile 101 may trigger memory access requests through tensor traversal operations that request read and write access to various address locations in memory 108. System 100 can handle these requests to access one or more bytes of data stored at the address locations in memory 108, 110 using the byte-level enhanced addressing capabilities of the disclosed ISA. As described below, these address locations may correspond to elements of an input (or parameter / weight) tensor being processed as part of the workload.

[0038] In addition to tensor read / write operations, processing the workload may also include i) moving data (e.g., parameters) from memory 108 (e.g., first on-chip SRAM) to memory 110 (e.g., second on-chip SRAM), and ii) processing read or write access requests to move data from memory 110 to memory 108. Tensor operations can be indicated by opcodes in instructions (e.g., single instructions) received at computation tile 101. For example, the "ByteAddressingMode" instruction may include one or more opcodes for a tensor operation to be performed at computation tile 101 to traverse each element of the input tensor. Generally, tensor operations can be performed to carry out specific ML computation operations that involve i) consuming (or reading) tensors from memories 108, 110, and ii) generating (or writing) tensors to memories 108, 110.

[0039] As used in this document, the first on-chip SRAM may refer to one or more memory units that each manipulate and store data having a size (or width) of, for example, 16 bits, 32 bits, or 64 bits or less, and the second on-chip SRAM may refer to one or more memory units that each manipulate and store data having a size (or width) of 64 bits, or larger than the data size manipulated by the first SRAM unit. For example, the size or width of on-chip SRAM 108 may be between 8 bits and 64 bits, while the size or width of on-chip SRAM 110 may be between 64 bits and 256 bits. Naturally, other sizes and data widths beyond these exemplary ranges are also contemplated for memory 108 and 110, respectively.

[0040] Figure 2 shows an exemplary processing pipeline for routing inputs and outputs between the memory and compute cells of a hardware integrated circuit. Generally, the pipeline 200 uses the input bus 106 to route inputs retrieved from memory locations in memory 108 to one or more compute cells 114. In some embodiments, the inputs routed to the compute cells 114 via the input bus 106 are inputs retrieved from memory 108 using various combinations of byte access requests based on the byte-level addressing capabilities of the enhanced ISA disclosed herein.

[0041] The pipeline 200 routes outputs to memory locations in memory 108 using the output bus 107. These outputs may be numerical or quantitative outputs generated from computational operations (e.g., multiplication or other arithmetic) performed in one or more computational cells 114. Each tile 101 may include a nonlinear unit (NLU) 210 that applies a nonlinear activation function to the output, which is the result of arithmetic performed in the computational unit 112. For example, computational tile 101 causes the NLU 210 to apply an activation function (e.g., sigmoid, tanh) to generate output activations for computations in a given neural network layer. The pipeline 200 routes outputs (or output activations) to memory locations in memory 108 using the output bus 107.

[0042] The pipeline 200 leverages a hardware architecture in which the input bus 106 is coupled (e.g., directly coupled) to each of multiple groupings of hardware computing cells of a dedicated integrated circuit. Although memory 110 is not shown in the example in Figure 2, the system 100 can provide a first operand from a location in memory 108 and a second operand from a location in memory 110, as described above with reference to Figure 1. The operand 204 is used in a computation performed in cell 114, which generates an output corresponding to the result of the computation. In some embodiments, the computation is for machine learning operations, such as processing input through the neural network layer of an artificial neural network.

[0043] In this embodiment, the computation tile 101 can provide a subset of cells 114 with first operands (e.g., a0, a1, a2, etc.) corresponding to inputs or activations of the input feature map. For example, each input of the input vector 102 is provided to each MAC in the subset via the input bus 106 of the computation tile 101, following a byte-level data access operation performed on the memory bank of memory 108. The system 100 can provide different sets of inputs based on byte-level access operations performed locally at each computation tile.

[0044] System 100 can process, execute, or otherwise handle different combinations of byte access requests to obtain or write a set of operands for a given computational operation. For example, across multiple computational tiles 101, System 100 can perform a broadcast operation to route a group of inputs (or operands) to a cell 114 of a given tile. System 100 can compute a product over a given neural network layer using each grouping of inputs and their corresponding weights at each computational tile 101. At a given computational tile 101, the product is computed by multiplying each input (e.g., a1) and its corresponding weight (e.g., w1) by each MAC in the subset using a MAC multiplication circuit.

[0045] System 100 can generate the layer output based on the accumulation of multiple products calculated in each MAC of cells 114a / b / c within a subset of cells 114a / b / c of ​​the computing unit 112. A multiplication operation performed within a computing tile 101 may require i) a first operand (e.g., input or activation) stored in a memory location in memory 108 corresponding to each element of the input tensor, and ii) a second operand (e.g., weight) stored in a memory location in memory 110 corresponding to each element of the parameter tensor.

[0046] In the example in Figure 2, the shift register 202 can provide a shift function, thereby broadcasting the input of operand 204 onto the input bus 106 and routing it to one or more MACs 114 of cell 114. In some embodiments, the shift register 202 enables one or more input broadcast modes on the compute tile 101. For example, the shift register 202 can be used to broadcast inputs sequentially from memory 108, e.g., one by one (first broadcast mode), simultaneously from memory 108, e.g., in parallel (second broadcast mode), or using any combination of these broadcast modes. The shift register 202 may be an integrated function of memory 108 and may be implemented in hardware, software, or both.

[0047] In some embodiments, the weight (w3) of operand 204 may have a weight value of zero. When controller 103 determines that the weight (w3) has a zero value, it may skip the multiplication between the input (a2) and the weight (w3) to conserve processing resources, and as a result, those operands are neither routed to nor consumed by cells 114a / b / c. The decision to skip that particular multiplication operation may be based on a mapping vector that maps the discrete input (an) of the input vector to the individual weights (wn) of the parameter tensor, as described above.

[0048] Figure 3 shows exemplary field and message structures for an exemplary instruction set. A first set of field and message structures is associated with a first instruction set 302, and a second distinct set of field and message structures is associated with a second distinct instruction set 304. Instruction set 302 represents an earlier method for data access that is limited to a row addressing scheme for accessing physical memory. For example, instruction set 302 may represent a flawed existing instruction set because it includes operations that require data padding and access alignment requirements that make it difficult to program and debug. This is illustrated below with reference to the example in Figure 4.

[0049] In the example in Figure 3, instruction set 304 represents an ISA that enables enhanced byte-level addressing capabilities for the memory of an exemplary dedicated processor (e.g., on-chip SRAM). For example, the processor may be a TPU, hardware accelerator, or neural network processor that uses a specific type of memory structure to store inputs for computational operations such as ML workloads. As described above, the processor may include multiple computation tiles 101, each computation tile 101 may be configured to perform some (or all) of the computational operations, such as processing batches of inputs through layers of an artificial neural network.

[0050] The inputs within a batch can correspond to images, and the pixels of that image can be represented as input tensors (e.g., multidimensional input tensors). Each pixel in the input tensor may correspond to a neural network input provided as input to a layer of a neural network. The neural network input may also be an activation value corresponding to a neural network output generated as the output of the neural network layer. Each input (or activation) value of the tensor corresponds to an element along a specific dimension (e.g., X, Y dimensions) of the multidimensional tensor. In connection therewith, each element of the tensor may correspond to a memory location in memory 108 (or memory 110), and each memory location may have a corresponding address.

[0051] In some embodiments, each computation tile 101 includes each tensor traversal unit (TTU) that generates addresses for traversing elements along a given dimension of a tensor. An exemplary TTU is described below with reference to Figure 5. Each input can be represented by a group of bits (e.g., a byte) stored at a given memory address location in memory 108. Using the disclosed instruction set 304, the system 100 can cause one or more of the TTUs to generate addresses at byte-level granularity. This is described in more detail below.

[0052] Generally, ISA302, 304 may include data fields and corresponding messages associated with those data fields. For example, data fields may specify attributes or details of an operation, and messages may indicate the type of operation. In the example in Figure 3, ISA302 requires multiple fields 306 and messages 308 for a given operation. For example, memory access using ISA302 opcodes often requires multiple data fields, such as row information fields or hardware component fields, which often leads to increased processing overhead in the compute tile.

[0053] In contrast, ISA304 represents an improved instruction set that provides a streamlined approach to specifying attribute and type information for various memory access operations. For example, an integrated aspect of ISA304 provides a simplified, modular ISA that provides integrated, semantically consistent fields to enable diverse memory access patterns. Rather than relying on multiple information fields, ISA304 incorporates an integrated information field 310 and an integrated message field 312, thereby improving overall performance by avoiding the need to store and process values ​​of irrelevant fields and reducing the data communication required to transmit those values.

[0054] In some embodiments, the opcode generated by ISA304 may have an information field 310 that includes a byte count parameter indicating the number of bytes accessed from the memory location of the compute tile 101. In some cases, the information field 310 is included in the opcode to indicate that the byte address mode of the compute tile has been activated. For example, an opcode in an instruction sent to a compute tile may convey a “ByteAddressMode” message that provides an integrated field for byte-level access to data in memories 108, 110. This will be described in more detail below with reference to Figure 4.

[0055] In the case of ISA304, the integrated information field 310 and message field 312 work together to provide byte-granular addressing that allows access to contiguous bytes of varying lengths from any given byte address in the memory bank of the compute tile 101. In some embodiments, the compute tile 101 may contain one or more TTUs, and ISA304 is configured such that each TTU may have a corresponding ByteAddressMode field or instruction. The message field 312 represents an efficient approach to indicating the pipeline type (or type of operation) in which byte-level access may occur. For example, the message field 312 may indicate a tensor traversal for a tensor read / write operation.

[0056] In some embodiments, the message field 312 may indicate byte-level access for operations such as read or write access requests for i) moving data (e.g., parameters) from the first on-chip SRAM 108 to the second on-chip SRAM 110, and ii) moving data from memory 110 to memory 108. One or more of these move operations can be used to support parameter caching. In some cases, to handle an exemplary workload, the first compute tile 101 coordinates and executes access requests (e.g., read / write requests) for memory 108, where the requests are based on external data communication originating from outside the first compute tile 101.

[0057] Figure 4 shows exemplary code for a tensor operation 400, which involves performing an input memory read operation (402) using the byte addressing features of the improved instruction set disclosed herein. In some embodiments, the code for the operation 400 corresponds to instructions for memory addressing techniques performed to access data from a portion of on-chip SRAM that primarily stores inputs and activations for later processing through a neural network. Instructions (and / or code) for the operation 400 may be derived from ISA 304 for an exemplary TPU including one or more compute tiles 101.

[0058] As described above, existing ISAs (e.g., ISA302) have data padding and access alignment requirements, which can make programming and debugging difficult. The alignment aspect relates to the requirement that, for a given access request, the number of bytes of data accessed must be a multiple of a certain integer value, where the integer value is greater than 1 (e.g., 4). The data padding aspect corresponds to the alignment aspect and relates to the requirement to pad one or more bytes returned for a given request with a certain number of bytes. The number of bytes padded to the returned bytes is based on integer multiples.

[0059] For example, existing / previous instruction sets may be tightly coupled to the circuit's microarchitecture, resulting in the circuit's memory units needing to return at least 4 bytes even if a particular request only requires 1, 2, or 3 bytes of data. Therefore, if a given request wants to access 1 byte, but the previous instruction set imposes a 4-byte requirement (i.e., integer value = 4), the padding and alignment constraints of this previous ISA would require 3 bytes of padding. This operation would require an exemplary computation tile 101 to return 4 bytes of data, but then the 3 padding bytes would be discarded. In some embodiments, such inefficient padding and alignment requirements relate to a row addressing scheme that imposes unnecessary constraints on read access to memories 108, 110.

[0060] In contrast to existing / previous instruction sets, the technologies and systems disclosed herein provide novel combinations of byte access patterns that support ML computations. More specifically, the disclosed technologies support more flexible memory access patterns and do not limit the opportunities for diverse access patterns in new areas such as image processing. In some embodiments, improved instruction sets and byte access techniques (e.g., ISA304) are used to optimize stencil operations with data reuse patterns. Such operations are strong candidates for leveraging data locality in specific image processing applications.

[0061] For example, referring again to tensor operations 400, the data access patterns enable various types of fine-grained stride (404) operations for workloads such as ML image processing operations, as well as spatial and temporal data reuse (404) to support ML inference and training workloads for various data science applications. In the example in Figure 4, the nature of the stride operation can be controlled via the “step” parameter 406. Generally, strides are processed in relation to memory access to obtain data such as image pixel values ​​for image processing ML workloads.

[0062] Images, or pixel values ​​associated with images, can be stored across physical rows and physical banks of memory 108,110, referencing the hardware configuration of the physical rows and physical banks of memory. In this example, stride is a component or parameter of the neural network's filter (or kernel). Stride is used to modify the amount the filter moves across the image or video. For example, if the stride is set to 1, the computation tile 101 moves the filter(s) one pixel (or input) at a time across the region. Similarly, if the stride is 2, the computation tile 101 moves the filter(s) two pixels at a time across the region. Thus, filters may be shifted based on the stride value of the layer, and in some embodiments, the system 100 can repeat this process across multiple computation tiles 101 of different layers until the inputs of different regions of the image have the corresponding dot product.

[0063] Moving a filter across an input region of an image based on a stride (or “step”) value may involve searching, retrieving, or otherwise accessing the input from various locations in memory 108, 110 according to that stride value. In the example in Figure 4, the stride can be expressed as the “step” size (e.g., step = 2) used when sliding the convolution filter across the input image. By enabling localized tile access to contiguous bytes of varying lengths, the improved byte-addressing ISAs herein (e.g., ISA 304) support and enable the application of various strides (or step values) from any given byte address in the input memory of the computation tile.

[0064] As described above, the improved ISA includes one or more ByteAddressingMode instructions with integrated fields for memory access granularity that enable automatic zero-padding when accessing data from the input memory of a compute tile. For example, the instruction byte_address_mode(408) may include one or more “access_bytes” fields to indicate the number of bytes written to the address space of memory 108, 110 in a given iteration. In some embodiments, bytes may be written to a contiguous address space, e.g., linear access, or a discontinuous address space. Automatic zero-padding can be performed on the fly via hardware logic, software logic, or both.

[0065] With respect to the access_bytes field, an instruction may include one or more fields such as access_bytes_loop_map, last_access_bytes, and default_access_bytes. The valid byte size of a given data packet can be indicated by the valid_byte_size field or parameter of a particular byte_address_mode instruction or message. In some embodiments, the value of the access_bytes field in an instruction for byte-level access can be greater than the maximum valid byte size per packet when consecutive packets deliver data stored in a contiguous address space. Thus, access_bytes can be greater than the amount of data carried by each ring / mesh packet.

[0066] In these examples, a hardware compute tile can consume these ring packets and write them sequentially to a specified memory address. Packets can be consumed simultaneously, with the hardware unit determining which portion or part(s) of a particular packet is valid for consumption. For example, there may be 20 incoming ring packets. Each packet carries 32B of data, and only 25B of this 32B is valid. In this example, the access_bytes field / instruction may be 500B / byte for the ring bus consumer pipeline. Therefore, the ring consumer consumes all 20 ring packets, extracting 25B from each, and these bytes, totaling 500, are written sequentially to the start address specified by the TTU.

[0067] System 100 uses access_bytes_loop_map to determine whether to access default_access_bytes or last_access_bytes. For each TTU iteration, the system reads these access bytes (e.g., default or last) depending on the specific access_bytes_loop_map. For each TTU clock cycle, a new start address is generated, and one or more access_bytes may be obtained. In some embodiments, valid_byte_size is specific to data packets on a ring bus or mesh bus. Each packet may contain, for example, 32B of data, and a count value may be provided along with the valid_byte_size field / instruction to specify the portion of the 32B data packet that is valid for consumption by a particular tile(s).

[0068] In byte-addressable mode, one or more tiles of system 100 can dynamically create write transactions for byte-level access based on the valid_byte_size and unprocessed access_bytes of a given packet. In some embodiments, control logic in a given tile 101 is configured to determine whether a single packet needs to be split into multiple write transactions, and in response to determining that the packet needs to be split, to generate a control signal to support or perform the split operation and generate the write transactions.

[0069] For example, system 100 can utilize this partitioning feature to enable on-the-fly packet scattering support, thereby allowing for more efficient use of the available bandwidth on bus 109. In some embodiments, a write transaction may include multiple byte-level requests. In some cases, the byte-level requests correspond to a byte-addressing sequence. Each address in this sequence identifies a memory location for storing (and later accessing) bytes of data, such as inputs to be processed for a machine learning workload.

[0070] As described above, bus 109-1 may be a ring bus that traverses two or more computation tiles to communicate datasets such as inputs and weights to the computation tiles in order to execute the ML workload. These datasets may be alternatively called ring packets. For example, the dataset may be an inbound ring packet arriving at computation tile 101 via the ring bus 109-1 of system 100. The input of the ring packet received at computation tile 101 is stored in memory 108 and then accessed and routed to computation unit 112 for the computations required to execute the ML workload.

[0071] The improved ISA and its corresponding byte_address_mode instruction enable the management of input data within ring packets at a byte-level granularity. In contrast to row addressing schemes where data is written over several cycles due to padding and alignment requirements, the byte-level scheme enabled by the improved ISA provides a mechanism for storing data at a coarser granularity, such as ring packets or partial sum caching, using fewer clock cycles and overall bandwidth.

[0072] For example, during an infeed operation, a tile may receive a packet containing 32 bytes (32B) of data from bus 109-1 (e.g., a ring bus). The structure of memory 108 may be limited to 4B per row, and therefore requires an alignment across 8 rows to receive 32 bytes of data. However, in some cases, the exemplary address of memory 108 to which the data is written (e.g., 0x0...4) does not align with 8 rows. In earlier instruction sets, such as ISA302 or related standard approaches, writing data to this unaligned address would require the system to first write the 32 bytes to an 8-row aligned address using one or more cycles. The system then uses an exemplary SRAM1-to-SRAM1 operation to return the data to the target address (i.e., 0x0...4), and the data is moved within (or locally) memory 108. Referring to earlier instruction sets, the bandwidth of the SRAM1-to-SRAM1 pipeline on compute tile 101 could be 4 bytes (4B) per cycle. Therefore, with the previous instruction set, the end-to-end cycle time for an infeed operation (writing from the ring to SRAM1) required at least 9 cycles, which is unnecessarily long.

[0073] The improved instruction set of ISA304 enables memory writes to SRAM1 (or SRAM2) to any byte address (e.g., in memories 108, 110) without the irrelevant overhead of additional cycles and bandwidth consumption. System 100 can perform certain write operations in fewer clock cycles and without the row alignment constraints or additional operations required by previous instruction sets. System 100 can perform a series of write operations in a single cycle using the configurations permitted by ISA304, regardless of how the memory banks of memories 108, 110 are organized in tiles. For example, when the required bundle size is 8 (e.g., 32 bytes), the address set generated by TTU in tiles does not need to be aligned in 8 rows (e.g., row addresses 0, 8, 16...) to complete the write of the bundle in a single cycle.

[0074] Furthermore, instead of discarding packets, the improved ISA's byte addressing mode supports efficient byte-granularity filtering of data packets received at tile 101, such as in the ring consumer mode of bus 109. System 100 can operate to isolate packet boundaries from filtering and focus on the data itself without padding. One principle behind byte-granularity filtering is to configure the new ISA so that TTU can be decoupled from traditional packet boundary-based data processing schemes.

[0075] The improved ISA byte addressing mode and filtering capabilities can utilize specific fields that allow bus 109 to supply valid byte data in a linearized order. This allows tile 101 to selectively filter and take in or exclude byte streams as the tile consumes packets of inbound data. To support filtering and other byte-granularity operations, exemplary byte addressing mode instructions in ISA 304 may include fields such as first_discard_byte_count, discard_byte_count(filter_out_bytes), access_bytes, and last_discard_byte_count. In some embodiments, each tile may include a register and associated control logic that tracks the unprocessed access_bytes, which are initialized by access_bytes when an iteration is initiated. When a write transaction is generated, the register value is decremented by the number of bytes valid in that transaction.

[0076] The byte-granularity filtering function can be used in conjunction with the TTU of the computation tile 101 to support arbitrary slicing of one or more tensors, such as slicing elements along a specific dimension of the tensor. For example, system 100 can leverage the filtering function to move slices of a tensor to different tiles, and the slicing may be based on any selected dimension. Furthermore, this can be extended to any rearrangement changes, where data associated with a tensor is stored in memory 108 (or 110) based on a first layout, and then undergoes a rearrangement, changing how (or in what order) the data is stored in the tile memory.

[0077] Figure 5 shows an exemplary byte addressing operation. This exemplary operation may be associated with a read operation performed using the TTU of compute tile 101.

[0078] In the example in Figure 5, the partition of memory 108 contains n physical memory banks 502, where n is an integer greater than or equal to 1. For example, the memory of the computation tile 101 may contain 32 physical memory banks 502, which may be identified as banks 0 to 31. Each bank may contain several numbers of rows, for example, 4, 16, 24, etc. Each row has a width that can be defined in bytes. For example, each row may be 16 bytes (16B) wide so that the computation tile 101 accesses data from memory 108 in 16B chunks (e.g., 128 bits). In some embodiments, memory 108 may contain more or fewer physical memory banks, and each row of a bank may have a width greater than or less than 16B (e.g., 1B).

[0079] As described above, the earlier instruction sets are limited by padding and alignment requirements that impose storage and bandwidth overhead, ultimately resulting in negative impacts on performance and power. Specifically, these earlier instruction sets can be linked to row addressing schemes that unnecessarily restrict memory access operations. In some embodiments, for certain sizes and structures of memory, the row addressing scheme can be limited to a 4-byte granularity when reading data from memory.

[0080] In the embodiment of Figure 5, an exemplary row of memory 108 is shown as being 4 bytes (4B) wide, but the improved ISA herein allows the compute tile 101 to access data from tile memory in chunks of 1B, 2B, 3B, 4B, or more than 4B. The control logic of the compute tile 101 (e.g., a controller) can identify opcodes and data fields in instructions that perform memory operations in one or more banks 502 of memory 108. The opcodes and data fields can be processed to generate an exemplary byte addressing sequence 504 used to access contiguous bytes of different lengths stored in one or more banks 502.

[0081] The instruction may include a "byte_count" field 506 that specifies the number of consecutive bytes to fetch, referencing the starting address of a memory location in memory 108, 110. As shown in Figure 5, the address can be represented by "0" or "2". The computation tile 101 references a specific starting address in a byte addressing sequence and can then perform multiple byte-addressable memory accesses in memory 108, 110 based on that starting address. Byte accesses can be performed as part of a tensor read operation to retrieve an input vector from one or more banks of memory 108.

[0082] In the example in Figure 5, read operation (508) is performed to retrieve a distinct / different portion of byte-level data stored at each address location in bank 502 of memory 108. This particular read operation (508) is performed to retrieve two bytes (byte count = 2). This operation starts at address "2" in row 0 of memory bank 502 and includes retrieving a first byte-level data stored at address "2" and a second distinct byte-level data stored at address "3". In some embodiments, the distinct portion or length of the byte-level data is defined based on a byte count integer. In these embodiments, a byte addressing sequence is used to access multiple consecutive bytes, and the number of bytes (e.g., length) is variable (as defined, for example, by the instruction).

[0083] Other read operations (510) are performed to retrieve distinct portions of byte-level data stored at each address location in bank 502. This particular read operation (510) is performed to retrieve 4 bytes (byte count = 4). This operation starts at address "2" in row 0 of memory bank 502 and includes retrieving each byte-level data stored at addresses 2 and 3 in row 0, and addresses 4 and 5 in row 1.

[0084] Other read operations (512) are performed to retrieve distinct portions of byte-level data stored at each address location in bank 502. This particular read operation (512) is performed to retrieve 10 bytes (byte count = 10). The operation begins at address "0" in row 0 of memory bank 502 and ends at address "9" in row 2 of memory bank 502. Thus, this particular read operation (512) involves retrieving each byte-level data stored at addresses 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9, with the address sequence spanning row 0, row 1, and row 2.

[0085] Read operations 508, 510, and 512 can be performed on the same memory bank 502 or on different memory banks 502. In particular, read or write operations can begin at any valid byte address and do not need to begin at a row-aligned address 514 such as address 4, 8, 12, or 16. As described above, the integrated fields of the improved ISA enable automatic zero-padding when accessing data from memory 108, 110 of the compute tile. For example, in the case of read operations 508 and 512, system 100 can automatically zero out one or more bytes based on the byte count being accessed for a given read operation.

[0086] Each compute tile 101 includes a TTU520 used to perform access operations in tile memory. In the exemplary read operation shown in Figure 5, based on the improved ISA opcodes and data fields, the TTU520 is operable to generate an address that can represent one or more valid byte addresses. Specifically, the improved ISA and the streamlined configuration of the associated byte address mode allow the TTU520 to generate addresses at the byte level granularity. The TTU520 is also decoupled from conventional packet boundary data processing.

[0087] The ISA304 can be configured to provide explicit hints about the size of contiguous data and its synchronization timing. For example, the "access_bytes" field can be used to define the minimum synchronization granularity, its corresponding timing, and the amount of contiguous data. The ISA304 can enable software-independent optimizations, such as leveraging TTU byte-level addressing without the overhead of arbitrary TTU reprogramming for new instruction sets. In some embodiments, these characteristics of the improved ISA can be used to simplify architectural design in aspects such as synchronization, consistency, and tag matching.

[0088] In addition to the ISA, the compute tile 101 may include certain hardware features and data pipeline configurations that enable the tiles to operate more efficiently on data at their granularity. In some embodiments, the compute tile 101 may include an integrated buffer 522 used to increase the write granularity of the SRAM 1 (e.g., narrow memory) when writing the output of the NLU 210 to memory 108. In some embodiments, the system 100 can implement a new byte-addressable ISA in conjunction with the integrated buffer 522 to perform on-the-fly collection.

[0089] By using the byte-addressable ISA and the unified buffer 522 on the compute tile 101, data from multiple packets can be unified, and as a result, the unified data can be written to an exemplary contiguous address space all at once (e.g., simultaneously). For example, based on the unified buffer 522, instead of writing data to memory 108 using multiple separate 4B transactions, a set of data can be written as a single 16B write transaction (524), thereby saving clock cycles for a particular write operation. The value of 16B can be indicated via the access_bytes field of the instruction issued via the byte-addressable ISA.

[0090] Compared to previous approaches, this integration and write operation can increase the memory bandwidth of SRAM1 and reduce the likelihood of experiencing bank contention in memories 108 and 110. In some cases, the multiple packets of data being integrated may be data packets generated locally on the compute tile 101, or external incoming data packets received on the compute tile via the bus 109. In some embodiments, this integration operation can be used when writing output activations to memory 108 to enhance the granularity and bandwidth of the NLU output. For example, this can enable byte-level NLU output granularity.

[0091] Figure 6 shows an exemplary process 600 of byte stream processing using exemplary byte addressing operations of an instruction set. Process 600 can be implemented or executed using the system 100 described above. Therefore, the description of process 600 may refer to the computing resources of system 100 described above. In some examples, the steps or actions of process 600 are made possible by programmed firmware instructions, software instructions, or both. Each type of instruction may be stored in a non-temporary machine-readable memory device and can be executed by one or more processors of the devices and resources described in this document, such as a hardware accelerator or a compute tile of a neural network processor.

[0092] In some embodiments, the steps of process 600 are performed by hardware circuitry to generate layer outputs of a neural network layer. The outputs may be part of a computation for a machine learning task or inference workload to generate image processing or image recognition outputs. As described above, the integrated circuit may be a dedicated neural network processor or hardware machine learning accelerator configured to accelerate computations for generating various types of data processing outputs.

[0093] Referring again to process 600, system 100 receives instructions for the compute tile of the hardware circuit (602). System 100 can identify opcodes in the instructions for operations involving accessing one or more banks of the first memory of the compute tile (604). For example, the instructions may be byte-addressed mode instructions that include opcodes indicating a tensorop read operation or a tensorop write operation. In some embodiments, the instruction itself, or one or more opcodes of the instruction, includes a byte count field indicating the number of bytes accessed from the memory location of compute tile 101.

[0094] System 100 can determine and / or generate a byte addressing sequence based on any associated data fields of the opcode and instruction (606). The byte addressing sequence is used to access contiguous bytes of distinct lengths stored in one or more memory banks 502 of memories 108, 110. For example, the byte addressing sequence can be operated to access contiguous bytes of different lengths starting from any byte address in any row of multiple rows of memory banks and in any bank of multiple memory banks of memories 108, 110. In some embodiments, the byte addressing sequence can be generated by the TTU of a computation tile to retrieve or store the input of an input tensor or parameter tensor. For example, each address in the byte addressing sequence identifies a memory location for reading (or writing) bytes of data. One or more bytes of data can represent an input or input vector for processing in a layer of a neural network.

[0095] Based on a byte-addressing sequence, the system 100 can perform multiple byte-addressable memory accesses to retrieve an input vector from a memory bank (608). For example, a compute tile 101 processes the opcode and parameter values ​​of the byte count field (e.g., access_bytes) in a byte-addressed mode instruction. In response to this processing, the compute tile 101 can generate one or more byte access requests based on the instruction's fields or opcode. Each byte access request provides byte-granularity access to at least one byte of data at any given address in any given row of the tile's memory bank.

[0096] A byte access request may be for a write transaction, a read transaction, or a combination thereof. For example, a byte access request may be a write transaction containing or corresponding to a byte address sequence. In some embodiments, a byte access request may be a read transaction incorporating auto-zero padding. An exemplary byte address sequence for a read transaction may contain each address to a memory location in tile memory storing distinct portions of byte-level data. Each address location may correspond to an element of an input tensor being processed as part of a machine learning workload. For example, each address location may be mapped to the corresponding element of the input tensor, along with the byte-level data stored at that location.

[0097] As described above, byte-level data may be an input or an input vector, and system 100 can process the input vector in hardware circuitry based on instructions (610). For example, a computation tile 101 may generate an output of a neural network layer in response to processing an input vector through the neural network layer. This processing may include arithmetic operations performed by a computation unit 112. Arithmetic operations may produce a set of outputs that are written to (stored) memory 108. The outputs may be activation values ​​written to memory 108 using byte-addressable memory access. These byte-level accesses can be performed based on opcodes of byte-addressable mode instructions. For example, the opcode may indicate a tensorop write transaction involving the TTU 520 of computation tile 101 and an integrated buffer 522 coupled to memory 108 at the tile.

[0098] Figure 7 shows an example of a tensor or multidimensional matrix 700, which includes an input tensor 704, a variation of the parameter tensor 706, and an output tensor 708. In the example in Figure 7, each tensor 700 contains an element, each of which can correspond to a data value (or operand) for a computation performed in a given layer of a neural network. As described above, the data or operand may be inputs, activations, weights (or parameters) for a computational operation, which includes one or more neural network layers or other types of ML constructs implemented in the computation tile 101.

[0099] For example, each input of the input tensor 704 can correspond to each element along a given dimension of the input tensor 704, each weight of the parameter tensor 706 can correspond to each element along a given dimension of the parameter tensor 706, and each output value or activation in the set of outputs can correspond to each element along a given dimension of the output tensor 708. In connection with this, each element can correspond to each memory location or address in the memory of the computation tile 101 assigned to operate on one or more dimensions of the given tensors 704, 706, and 708.

[0100] A computation performed on a given neural network layer may involve multiplying an input / activation tensor 704 and a parameter / weight tensor 706 in one or more processor clock cycles to generate a layer output, which may include output activations. Multiplying the activation tensor 704 by the weight tensor 706 involves multiplying the activations from the elements of tensor 704 by the weights from the elements of tensor 706 to produce one or more partial sums. The exemplary tensor 706 in Figure 7 may be an unmodified parameter tensor, a modified parameter tensor, or a combination thereof. In some embodiments, each parameter tensor 706 corresponds to a modified parameter tensor containing non-zero CSP values ​​derived based on the sparsity exploitation technique described above.

[0101] The processor cores of system 100 can operate on i) scalars corresponding to discrete elements in a multidimensional tensor 704, 706, ii) a vector of values ​​containing multiple discrete elements 707 along the same or different dimensions of a multidimensional tensor 704, 706 (e.g., input vector 102), or iii) a combination of these. In a multidimensional tensor, each discrete element 707, or multiple discrete elements 707, can be represented using X,Y coordinates (2D) or X,Y,Z coordinates (3D), depending on the number of dimensions of the tensor.

[0102] System 100 can compute multiple partial sums corresponding to products generated by multiplying batches of inputs by corresponding weight values. As described above, System 100 can perform the accumulation of products (e.g., partial sums) over many clock cycles. For example, the accumulation of products can be performed in the random access memory, shared memory, or scratchpad memory (e.g., memory 108) of one or more computation tiles 101 based on the techniques described in this document. In some embodiments, the accumulation of products into memory of computation tiles is performed using byte-addressable mode instructions of ISA304.

[0103] In some embodiments, the input weight multiplication may be written as the sum of the products of each weight element multiplied by the discrete input of the input vector 102, such as a row or slice of the input tensor 704. This row or slice can represent a given dimension, such as a first dimension 710 of the input tensor 704, or a different second dimension 715 of the input tensor 704. The improved byte-granularity filtering capabilities of ISA304 include arbitrary slicing of any one or more tensors, such as slicing elements along a particular dimension of the tensor, and can be used to support the multiplication operation.

[0104] In some embodiments, the output of a convolutional neural network layer can be computed using an exemplary set of calculations. Computing a CNN layer may involve performing a 2D spatial convolution between a 3D input tensor 704 and at least one 3D filter (weight tensor 706). For example, a 2D spatial plane 720 or 725 can be generated by convolving one 3D filter 706 onto the 3D input tensor 704. This computation may involve computing the sum of dot products over a specific dimension of the input volume containing the input vector 102.

[0105] For example, spatial plane 720 may contain the output value of the sum of products calculated from inputs along dimension 710, and spatial plane 725 may contain the output value of the sum of products calculated from inputs along dimension 715. The calculation to generate the sum of products of the output values ​​in spatial planes 720 and 725 can be performed i) in calculation cells 114a / b / c, ii) directly in memory 110 using arithmetic operators coupled to a shared bank of memory 110, or iii) or both. In some embodiments, the reduction operation may be performed either directly at the memory cells (or locations) of memory 110 using various techniques for the reduction of cumulative values.

[0106] The subject matter and functional operation embodiments described herein can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, or computer hardware, including structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs encoded in a tangible non-temporary program carrier, i.e., one or more modules of computer program instructions, to be executed by a data processing device or to control the operation of a data processing device.

[0107] Alternatively or additionally, program instructions may be encoded into artificially generated propagating signals, such as mechanically generated electrical signals, optical signals, or electromagnetic signals, which are generated to encode information for transmission to a receiving device suitable for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random-access memory device, a serial-access memory device, or one or more of these.

[0108] The term “computing system” encompasses all kinds of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may include dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, a device may also include code that constitutes the execution environment for the computer program in question, such as processor firmware, protocol stacks, database management systems, operating systems, or one or more combinations thereof.

[0109] Computer programs (sometimes called, or written as, programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, such as as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0110] Computer programs may, but do not necessarily, correspond to files in a file system. A program may be stored in a single file dedicated to the program itself, or in a group of files, such as files containing one or more scripts stored in other programs or data, for example, in a markup language document. Computer programs can be deployed to run on a single computer, located in one place, or distributed across multiple locations and interconnected by a communication network.

[0111] The processes and logic flows described herein can be performed by one or more programmable computers executing one or more computer programs to act on input data and produce outputs, thereby performing their functions. The processes and logic flows can also be performed by dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), or GPGPUs (General-Purpose Graphics Processing Units), and these devices can also be implemented as dedicated logic circuits.

[0112] A computer suitable for running computer programs includes, for example, a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit, and may be based on them. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. Some elements of a computer are the central processing unit for executing and running instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operably connected to them to receive data from them, transmit data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be incorporated into other devices, such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices (such as Universal Serial Bus (USB) flash drives) (these are just a few examples).

[0113] Computer-readable media suitable for storing computer program instructions and data include, for example, all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be complemented by or incorporated into dedicated logic circuits.

[0114] To provide user interaction, embodiments of the subject matter described herein can be implemented in a computer having a display device for displaying information to the user, such as an LCD (liquid crystal screen) monitor, and a keyboard and pointing device, such as a mouse or trackball, that allows the user to provide input to the computer. Other types of devices can also be used to provide user interaction, for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and the input from the user may be received in any form, including acoustic, voice, or haptic input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0115] Embodiments of the subject matter described herein can be implemented as a computing system including, for example, a backend component as a data server, or a computing system including a middleware component, for example, an application server, or a computing system including a frontend component, for example, a client computer having a graphical user interface or a web browser to enable a user to interact with the embodiment of the subject matter described herein, or in any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), for example, the Internet.

[0116] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The client-server relationship arises from computer programs that run on each computer and have a client-server relationship with each other.

[0117] While this specification includes details of many specific embodiments, these should not be construed as limitations on the scope of any invention or claim, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments, individually or in any preferred secondary combination. Furthermore, features may be described above as functioning in a particular combination, and even if initially claimed as such, one or more features from the claimed combination may be removed from the combination, and the claimed combination may cover secondary combinations or variations of secondary combinations.

[0118] Similarly, while operations are shown in a specific order in the drawings, this should not be understood as requiring that such operations be performed in a specific illustrated or sequential order, or that all illustrated operations be performed, in order to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described can generally be integrated into a single software product or packaged into multiple software products.

[0119] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order, and this may still yield desirable results. As an example, the process shown in the accompanying drawings does not necessarily require to be performed in the specific order or sequence shown to obtain the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

[0120] Further examples are provided in the following sections.

[0121] Example 1: A method performed by a computer to access a memory bank of a hardware accelerator, the method further comprising receiving an instruction for a compute tile of the hardware accelerator, the instruction being executable on the compute tile to cause the execution of an operation, the operation comprising: identifying an opcode for an operation in the instruction that involves accessing one or more banks of a first memory of the compute tile; generating a byte-addressable sequence used to access contiguous bytes of different lengths stored in the one or more banks based on the opcode; performing a plurality of byte-addressable memory accesses in the first memory to obtain a plurality of input vectors from the one or more banks of the first memory based on the byte-addressable sequence; and processing the plurality of input vectors in the hardware accelerator based on the instruction.

[0122] Example 2: The method according to Example 1, wherein the byte addressing sequence is operable to access contiguous bytes of different lengths starting from any given byte address across one or more banks of the first memory.

[0123] Example 3: The method according to Example 1 or 2, further comprising generating a byte access request that provides byte-granularity access to retrieve at least one byte of data stored in a given row of one of the one or more banks of the first memory, based on the opcode.

[0124] Example 4: The method of any of the prior embodiments, wherein generating the byte addressing sequence comprises generating a byte addressing sequence corresponding to a plurality of byte access requests, each byte access request being used to access a distinct portion of byte-level data stored at each address location across one or more banks of the first memory.

[0125] Example 5: The method according to any of the prior embodiments, wherein the opcode is for a tensor operation performed in the computation tile to traverse each element of the input tensor.

[0126] Example 6: The method of Example 5, dependent on Example 4, further comprising: executing the plurality of byte access requests using the byte addressing sequence; executing a plurality of byte-granularity accesses in the first memory based on the plurality of byte access requests that are executed; and traversing each element of the input tensor by executing the plurality of byte-granularity accesses in the first memory.

[0127] Example 7: The method according to Example 6, or Example 5, which is dependent on Example 4, wherein each distinct portion of byte-level data stored at each address location across one or more banks of the first memory corresponds to each element of the input tensor, and each address location storing a distinct portion of byte-level data is mapped to the corresponding element of the input tensor.

[0128] Example 8: The method of Example 6, or Example 7 dependent on Example 6, wherein a portion of the byte-level data corresponds to one of the plurality of input vectors, and the plurality of byte-addressable memory accesses used to obtain the plurality of input vectors corresponds to i) the plurality of byte-granularity accesses, or ii) at least one of the plurality of byte access requests.

[0129] Example 9: The method of any of the prior embodiments, wherein processing the plurality of input vectors includes processing the plurality of input vectors through a neural network layer of a neural network implemented in the hardware accelerator, and generating an output of the neural network layer in response to processing the plurality of input vectors through the neural network layer.

[0130] Example 10: A system comprising a hardware accelerator, a processing device, and a non-temporary machine-readable storage medium for storing instructions used to access the memory banks of the hardware accelerator, wherein the instructions are executable by the processing device to cause the execution of an operation which includes receiving an instruction for a compute tile of the hardware accelerator, the instructions for the compute tile are executable on the compute tile to cause the execution of an operation which includes, in the instruction, identifying an opcode for an operation which involves accessing one or more banks of a first memory of the compute tile, generating a byte-addressing sequence used to access contiguous bytes of different lengths stored in the one or more banks based on the opcode, performing a plurality of byte-addressable memory accesses in the first memory to obtain a plurality of input vectors from the one or more banks of the first memory based on the byte-addressing sequence, and processing the plurality of input vectors in the hardware accelerator based on the instruction.

[0131] Example 11: The system according to Example 10, wherein the byte addressing sequence is operable to access contiguous bytes of different lengths starting from any given byte address across one or more banks of the first memory.

[0132] Example 12: The system according to Example 11, wherein the operation further comprises generating a byte access request that provides byte-granularity access to retrieve at least one byte of data stored in a given row of one of the one or more banks of the first memory, based on the opcode.

[0133] Example 13: The system according to Example 11 or 12, wherein generating the byte addressing sequence includes generating a byte addressing sequence corresponding to a plurality of byte access requests, each byte access request being used to access a distinct portion of byte-level data stored at each address location across one or more banks of the first memory.

[0134] Example 14: The system according to any one of Examples 10 to 13, wherein the opcode is for a tensor operation performed in the computation tile to traverse each element of the input tensor.

[0135] Example 15: The system according to Example 14, dependent on Example 13, wherein the operation further includes performing access requests for the plurality of bytes using the byte addressing sequence, performing a plurality of byte-granularity accesses in the first memory based on the plurality of byte access requests performed, and traversing each element of the input tensor by performing the plurality of byte-granularity accesses in the first memory.

[0136] Example 16: The system according to Example 15, or Example 14 dependent on Example 13, wherein each distinct portion of byte-level data stored at each address location across one or more banks of the first memory corresponds to each element of the input tensor, and each address location storing a distinct portion of byte-level data is mapped to the corresponding element of the input tensor.

[0137] Example 17: The system according to Example 15 or Example 16, wherein a portion of the byte-level data corresponds to one of the multiple input vectors, and the multiple byte-addressable memory accesses used to obtain the multiple input vectors correspond to i) the multiple byte-granularity accesses, and ii) at least one of the multiple byte access requests.

[0138] Example 18: The system according to any one of Examples 10 to 17, wherein processing the plurality of input vectors includes processing the plurality of input vectors through a neural network layer of a neural network implemented in the hardware accelerator, and generating an output of the neural network layer in response to processing the plurality of input vectors through the neural network layer.

[0139] Example 19: A non-temporary, machine-readable storage medium for storing instructions used to access memory banks of a hardware accelerator, wherein the instructions are executable by a processing device to cause the execution of an operation which includes receiving an instruction for a compute tile of the hardware accelerator, the instructions for the compute tile are executable on the compute tile to cause the execution of an operation which includes, in the instruction, identifying an opcode for an operation which involves accessing one or more banks of a first memory of the compute tile, generating a byte-addressing sequence used to access contiguous bytes of different lengths stored in the one or more banks based on the opcode, performing a plurality of byte-addressable memory accesses in the first memory to obtain a plurality of input vectors from the one or more banks of the first memory based on the byte-addressing sequence, and processing the plurality of input vectors on the hardware accelerator based on the instruction.

[0140] Example 20: A non-temporary machine-readable storage medium according to Example 19, wherein the byte addressing sequence is operable to access contiguous bytes of varying lengths starting from any given byte address across the one or more banks of the first memory, and the operation further comprises generating a byte access request that provides byte-granularity access to retrieve at least one byte of data stored in a given row of one of the banks in the one or more banks of the first memory, based on the opcode.

Claims

1. A method performed by a computer to access the memory bank of a hardware accelerator, the method being: The process includes receiving instructions for the compute tile of the hardware accelerator, the instructions being executable on the compute tile to cause the execution of an operation, and the operation being, The instruction identifies an opcode for an operation that involves accessing one or more banks of the first memory of the compute tile, Based on the opcode, generate a byte addressing sequence used to access consecutive bytes of different lengths stored in one or more banks; Based on the byte addressing sequence, the first memory performs multiple byte-addressable memory accesses to obtain multiple input vectors from one or more banks of the first memory, A method comprising processing the plurality of input vectors with the hardware accelerator based on the instruction.

2. The method according to claim 1, wherein the byte addressing sequence is operable to access the contiguous bytes of different lengths starting from any given byte address across one or more banks of the first memory.

3. The method according to claim 1 or 2, further comprising generating a byte access request that provides byte-granular access to retrieve at least one byte of data stored in a given row of one of the one or more banks of the first memory, based on the opcode.

4. Generating the aforementioned byte addressing sequence means The method according to any prior claim, comprising generating a byte addressing sequence corresponding to a plurality of byte access requests, each byte access request being used to access one or more distinct portions of byte-level data stored at each address location across one or more banks of the first memory.

5. The method according to any prior claim, wherein the opcode is for a tensor operation performed in the computation tile to traverse each element of the input tensor.

6. Executing the plurality of byte access requests using the byte addressing sequence, Based on the multiple byte access requests that are executed, multiple byte-granularity accesses are performed in the first memory, The method according to claim 5, which is dependent on claim 4, further comprising traversing each element of the input tensor by performing the plurality of byte-granularity accesses in the first memory.

7. Each distinct portion of the byte-level data stored at each address location across the one or more banks of the first memory corresponds to each element of the input tensor, The method according to claim 6, or claim 5 as dependent on claim 4, wherein each address location that stores a distinct portion of byte-level data is mapped to a corresponding element of the input tensor.

8. A portion of the byte-level data corresponds to one of the input vectors mentioned above. The method according to claim 6, or claim 7 dependent on claim 6, wherein the plurality of byte-addressable memory accesses used to obtain a plurality of input vectors correspond to i) the plurality of byte-granularity accesses, and ii) the plurality of byte access requests.

9. Processing the aforementioned multiple input vectors is, The process involves processing the multiple input vectors through the neural network layer of the neural network implemented in the hardware accelerator, The method according to any prior claim, comprising generating an output of the neural network layer in response to processing the plurality of input vectors through the neural network layer.

10. A system comprising a hardware accelerator, a processing device, and a non-temporary machine-readable storage medium for storing instructions used to access the memory bank of the hardware accelerator, wherein the instructions are executable by the processing device to cause the execution of an operation, and the operation is The process includes receiving instructions for the compute tile of the hardware accelerator, the instructions being executable on the compute tile to cause the execution of an operation, and the operation being, The instruction identifies an opcode for an operation that involves accessing one or more banks of the first memory of the compute tile, Based on the opcode, generate a byte addressing sequence used to access consecutive bytes of different lengths stored in one or more banks; Based on the byte addressing sequence, the first memory performs multiple byte-addressable memory accesses to obtain multiple input vectors from one or more banks of the first memory, A system comprising processing the plurality of input vectors with the hardware accelerator based on the aforementioned instructions.

11. The system according to claim 10, wherein the byte addressing sequence is operable to access the contiguous bytes of different lengths starting from any given byte address across one or more banks of the first memory.

12. The aforementioned operation is, The system according to claim 11, further comprising generating a byte access request that provides byte-granularity access to retrieve at least one byte of data stored in a given row of one of the one or more banks of the first memory, based on the opcode.

13. Generating the aforementioned byte addressing sequence means The system according to claim 11 or 12, comprising generating a byte addressing sequence corresponding to a plurality of byte access requests, each byte access request being used to access a distinct portion of byte-level data stored at each address location across one or more banks of the first memory.

14. The system according to any one of claims 10 to 13, wherein the opcode is for a tensor operation performed in the computation tile to traverse each element of the input tensor.

15. The aforementioned operation is, Executing the plurality of byte access requests using the byte addressing sequence, Based on the multiple byte access requests that are executed, multiple byte-granularity accesses are performed in the first memory, The system according to claim 14, dependent on claim 13, further comprising traversing each element of the input tensor by performing the plurality of byte-granularity accesses in the first memory.

16. Each distinct portion of the byte-level data stored at each address location across the one or more banks of the first memory corresponds to each element of the input tensor, The system according to claim 15, or claim 14 as dependent on claim 13, wherein each address location storing a distinct portion of byte-level data is mapped to a corresponding element of the input tensor.

17. A portion of the byte-level data corresponds to one of the input vectors mentioned above. The system according to claim 15, or claim 16 dependent on claim 15, wherein the plurality of byte-addressable memory accesses used to obtain a plurality of input vectors correspond to at least one of the plurality of byte-granularity accesses, or the plurality of byte access requests.

18. Processing the aforementioned multiple input vectors is, The process involves processing the multiple input vectors through the neural network layer of the neural network implemented in the hardware accelerator, The system according to any one of claims 10 to 17, comprising generating an output of the neural network layer in response to processing the plurality of input vectors through the neural network layer.

19. A non-temporary, machine-readable storage medium for storing instructions used to access the memory bank of a hardware accelerator, wherein the instructions are executable by a processing device to cause the execution of an operation, and the operation is The process includes receiving instructions for the compute tile of the hardware accelerator, the instructions being executable on the compute tile to cause the execution of an operation, and the operation being, The instruction identifies an opcode for an operation that involves accessing one or more banks of the first memory of the compute tile, Based on the opcode, generate a byte addressing sequence used to access consecutive bytes of different lengths stored in one or more banks; Based on the byte addressing sequence, the first memory performs multiple byte-addressable memory accesses to obtain multiple input vectors from one or more banks of the first memory, A non-temporary machine-readable storage medium, which includes processing the plurality of input vectors with the hardware accelerator based on the aforementioned instructions.

20. The byte addressing sequence is operable to access the contiguous bytes of different lengths starting from any given byte address across one or more banks of the first memory, and the operation is, The non-temporary machine-readable storage medium according to claim 19, further comprising generating a byte access request that provides byte-granularity access to retrieve at least one byte of data stored in a given row of one or more banks of the first memory, based on the opcode.

Images