Reducing memory bank contention in hardware accelerators

Abstract

A method and system, including a computer-readable medium, are described for reducing or preventing memory bank contention in a hardware accelerator to enable simultaneous access of memory banks in the hardware accelerator. A computational tile of the hardware accelerator receives requests to be used to access a tiled memory of the accelerator. For each request, a logical address represented by a sequence of bits is identified in the request, and a first subset of bits is obtained from the sequence. An identifier is generated based on a bank generation function using the first subset of bits. The identifier identifies a particular bank among the physical memory banks of the tiled memory. Each request is processed using the individual bank identifier generated for that request. In response to processing the requests, multiple separate memory banks are accessed simultaneously during the same clock cycle.

Description

[Background technology]

[0001] This specification relates generally to memory operations in hardware integrated circuits. A neural network is a machine learning model that utilizes one or more layers of nodes to generate an output, e.g., a classification, for a received input. Some neural networks include one or more hidden layers in addition to an output layer. Some neural networks may be convolutional neural networks (CNNs) configured for image processing or recurrent neural networks (RNNs) configured for speech and language processing. Different types of neural network architectures can be used to perform a variety of tasks related to classification or pattern recognition, prediction involving data modeling, and information clustering.

[0002] A neural network layer can have a corresponding set of parameters or weights. The weights are used to process inputs (e.g., batches of inputs) through the neural network layer and generate corresponding outputs of the layer for computing neural network inferences. The batches of inputs and sets of kernels can be represented as tensors, i.e., multidimensional arrays, of inputs and weights. A hardware accelerator is an integrated circuit dedicated to implementing neural networks. The circuit includes a memory with locations corresponding to elements of the tensor that can be traversed or accessed using the circuit's control logic. Summary of the Invention

[0003] This document describes techniques for reducing (or preventing) memory bank contention in the tiled memory of a hardware accelerator, thereby enabling simultaneous access of these physical memory banks.

[0004] A computational tile of a hardware accelerator receives requests to be used to access a tile memory of the accelerator, and for each of the requests, i) a logical address represented by a sequence of bits is identified in the request, ii) a first subset of the bits is obtained from the sequence, and iii) an identifier is generated based on a bank generation function that uses the first subset of the bits.

[0005] The identifier identifies a particular bank among the physical memory banks of the tiled memory. Each request is processed using an individual bank identifier ("bank ID") generated for that request. In response to processing the requests, multiple distinct memory banks are accessed simultaneously during the same clock cycle (e.g., a single clock cycle).

[0006] One aspect of the subject matter described herein can be embodied in a computer-implemented method for simultaneously accessing memory banks of a hardware accelerator. The method includes receiving a plurality of requests, each request being used to access a tiled memory of the hardware accelerator. For each of the plurality of requests, the method includes identifying, within the request, a respective logical address represented by a sequence of bits; obtaining a first subset of bits from the sequence of bits; and generating, based on a bank generation function using the first subset of bits, a respective bank identifier that identifies a particular bank from among a plurality of physical memory banks of the tiled memory. The method further includes: i) processing each of the plurality of requests using the respective bank identifier generated for the request; and ii) simultaneously accessing multiple separate physical memory banks of the tiled memory during a clock cycle in response to processing each of the plurality of requests.

[0007] These and other implementations may each optionally include one or more of the following features. For example, in some implementations, simultaneously accessing multiple separate physical memory banks includes accessing multiple separate physical memory banks during a single clock cycle for a particular stride value. The particular stride value may be a memory access stride equal to the difference between particular rows of the same physical memory bank.

[0008] In some implementations, simultaneously accessing the plurality of separate physical memory banks includes accessing the plurality of separate physical memory banks without bank conflicts. In one aspect, a bank conflict occurs when two or more requestors request access to the same physical memory bank of the tiled memory during the same clock cycle. Each physical memory bank can include a plurality of rows, and the method includes, for each of the plurality of requests, i) obtaining a second subset of bits from the sequence of bits; ii) providing the second subset of bits as an input to a bank generation function; and iii) generating an individual row identifier that identifies a particular row within a particular bank among the plurality of physical memory banks of the tiled memory based on the bank generation function being applied to the second subset of bits.

[0009] In some implementations, each row of the plurality of rows comprises a width of 16 bytes, the tiled memory partition comprises 32 physical memory banks, and the memory access stride is row_width * Equal to num_banks. Obtaining a first subset of bits from the sequence of bits may include obtaining two or more bits that are among the least significant bits (LSBs) in the sequence of bits.

[0010] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs encoded on computer storage devices and configured to perform the actions of the methods. One or more computer systems may be so configured by software, firmware, hardware, or a combination thereof installed on the systems that, when operated, causes the systems to perform the actions. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the systems to perform the actions.

[0011] The subject matter described herein can be implemented in particular embodiments to achieve one or more of the following advantages: A bank generation function is disclosed that can be used to handle multiple requests for memory resources of a tiled memory without causing bank conflicts.

[0012] The bank generation function is configured to generate an individual bank ID for each request in a group of requests such that no two requests require access to the same physical memory bank of the tiled memory. The bank generation function provides an access pattern that enables parallel processing of each request in the group during the same clock cycle (e.g., a single clock cycle). The access pattern provided by the bank generation function enables simultaneous processing of multiple requests regardless of the stride value used for memory access.

[0013] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0014] [Figure 1A] FIG. 1 is a block diagram of an exemplary computing system for implementing a neural network machine learning model. [Figure 1B] FIG. 1 is a block diagram of an exemplary computing system for implementing a neural network machine learning model. [Figure 2] 1 illustrates an exemplary processing pipeline for routing inputs and outputs between memory and computational cells of a hardware integrated circuit. [Figure 3] 10 shows examples of memory scans for different banking generation functions. [Figure 4] 1 illustrates an exemplary bank generation function for reducing memory bank contention in a hardware integrated circuit. [Figure 5] 1 is an exemplary process for reducing memory bank contention in a hardware integrated circuit. [Figure 6] Examples of input tensors, parameter tensors, and output tensors are shown below. DETAILED DESCRIPTION OF THE INVENTION

[0015] Like reference symbols and designations in the various drawings indicate like elements. 1A is a block diagram of an exemplary computing system 100 for implementing a neural network model in a hardware integrated circuit, such as a machine learning hardware accelerator. Computing system 100 includes one or more computational tiles 101, a host 120, and a higher-level controller 125 (“controller 125”). As described in more detail below, host 120 and controller 125 cooperate to provide datasets and instructions to one or more computational tiles 101 of system 100.

[0016] In some embodiments, the host 120 and the controller 125 are the same device. The host 120 and the controller 125 can perform separate functions or can be integrated into a single device package. For example, the host 120 and the controller 125 can form a central processing unit (CPU) that interacts or cooperates with a hardware accelerator that includes multiple computational tiles 101. In some embodiments, the host 120, the controller 125, and the multiple computational tiles 101 are included on or formed on a single integrated circuit die. For example, the host 120, the controller 125, and the multiple computational tiles 101 can form a dedicated system-on-chip (SoC) optimized to run neural network models for processing machine learning workloads.

[0017] Each computational tile 101 generally includes a controller 103 that provides one or more control signals 105 to cause the inputs (or activations) of the input vectors 102 to be stored in or accessed from memory locations in a first memory 108 (“memory 108”). Similarly, the controller 103 may also provide one or more control signals 105 to cause the weights (or parameters) of the matrix structure of weights 104 to be stored in or accessed from memory locations in a second memory 110 (“memory 110”). In some implementations, the input vectors 102 are obtained from an input tensor, while the matrix structure of weights is obtained from a parameter tensor. Each of the input tensor and the parameter tensor may be a multidimensional data structure, such as a multidimensional matrix or tensor. This is described in more detail below with reference to FIG. 6.

[0018] Each memory location in the memories 108, 110 may be identified by a corresponding memory address, such as a logical address that has a corresponding mapping to a physical row of a physical memory bank of the memory. Referring to the example of Figure 1B, the computational tile 101 may derive a set of contiguous addresses (e.g., virtual / logical addresses) from a group of requests 130. For example, the set of contiguous addresses may be derived with reference to a logical memory that corresponds to the physical memory 108 of the computational tile 101. This is also described below with reference to the embodiments of Figures 4 and 5.

[0019] The logical memory has multiple logical ports 135, each potentially connected to or associated with a different requestor requesting access to the physical resources of memory 108. To process a given access request, for each port, tile 101 (or its controller 103) determines the bank to which the request is routed based on the address in the request. For each bank, computational tile 101 may include an arbiter 140 that arbitrates access to that bank from multiple ports according to a bank generation function uniquely configured to mitigate, for example, some (or all) of the requests from being routed to the same physical memory bank. The logical memory, its ports, and the arbiter may be implemented in software, hardware, or both. In some implementations, the logical memory and its ports, as well as the arbiter, are controlled based on control signals generated by controller 103.

[0020] Each of memories 108, 110 may be implemented as a series of physical banks, units, or any other related storage medium or device. Each of memories 108, 110 may include one or more registers, buffers, or both. In some implementations, memory 108 is an input / activation memory, while memory 110 is a parameter memory. In some other implementations, inputs or activations are stored in memory 108, memory 110, or both, and weights are stored in memory 110, memory 108, or both. For example, inputs and weights may be transferred between memory 108 and memory 110 to facilitate certain neural network calculations. In some implementations, each of memories 108 and 110 is referred to as a tiled memory.

[0021] Each computational tile 101 also includes an input activation bus 106, an output activation bus 107, and a computational unit 112, which has one or more hardware multiply-accumulate circuits (MACs) within each cell 114a / b / c. The controller 103 can generate control signals 105 to retrieve operands stored in the memory of the computational tile 101. For example, the controller 103 can generate control signals 105 to retrieve i) an example input vector 102 stored in memory 108 and ii) weights 104 stored in memory 110. Each input retrieved from memory 108 is provided to the input activation bus 106 for routing (e.g., direct routing) to a computational cell 114a / b / c within the computational unit 112. Similarly, each weight retrieved from memory 110 is routed to a cell 114a / b / c of ​​the computational unit 112.

[0022] As described below, each cell 114a / b / c performs a computation to obtain a partial sum or accumulation value to generate the output of a given neural network layer. An activation function may be applied to the set of outputs to generate a set of output activations for the neural network layer. In some implementations, the outputs or output activations are routed for storage and / or transmission via an output activation bus 107. For example, a set of output activations may be transmitted from a first computational tile 101 to a second, different computational tile 101 for processing as input activations for a different layer of the neural network in the second computational tile 101.

[0023] In general, each computational tile 101 and system 100 may include additional hardware structures to perform computations associated with multi-dimensional data structures such as tensors, matrices, and / or data arrays. In some implementations, inputs for input vectors (or tensors) 102 and parameter tensor weights 104 may be preloaded into memories 108, 110 of a computational tile 101. The inputs and weights are received as a set of data values ​​that arrive at a particular computational tile 101 from a host 120 (e.g., an external host) via a host interface or from a higher level control such as a controller 125.

[0024] Each of the computational tiles 101 and the controller 103 may include one or more processors, processing devices, and various types of memory. In some implementations, the processors of the computational tiles 101 and the controller 103 include one or more devices, such as a microprocessor or central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a combination of different processors. Each of the computational tiles 101 and the controller 103 may also include other computational and storage resources, such as buffers, registers, control circuits, etc. These resources cooperate to provide additional processing options for performing one or more of the decisions and calculations described herein.

[0025] In some implementations, the processing unit(s) of the controller 103 execute instructions stored and programmed in memory to cause the controller 103 and the computational tiles 101 to perform one or more functions described herein. The memory of the controller 103 may include one or more non-transitory machine-readable storage media. The non-transitory machine-readable storage media may include solid-state memory, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (e.g., EPROM, EEPROM, or flash memory), or any other tangible medium capable of storing information or instructions.

[0026] The system 100 receives instructions that define specific computational operations to be performed by the computational tiles 101. In some implementations, the host can generate a set of parameters (i.e., weights) and corresponding inputs for processing in a neural network layer. For example, the host can generate a set of compression parameters (CSPs) and a corresponding mapping vector, e.g., a non-zero map (NZM), that maps neural network inputs to non-zero parameters in the set of CSPs for a given operation. The host 120 sends the parameters to the computational tiles 101 via a host interface for further processing by that tile. The controller 103 can execute programmed instructions to analyze the data stream associated with the received weights and inputs, including the compression parameters and corresponding mapping vectors.

[0027] The controller 103 causes the data stream inputs and weights to be stored in the computational tiles 101. For example, the controller 103 may store mapping vectors and compressed sparse parameters in the local tile memory of the computational tiles 101, as described in more detail below. The controller 103 may also analyze the input data stream to detect operation codes (“opcodes”). The system 100 may support various types of opcodes, such as vector-matrix multiplication, element-wise vector operations, and opcode types that indicate an operation on whether a given operation uses compressed sparse parameters versus uncompressed parameters / weights.

[0028] Based on the one or more opcodes, the controller 103 may activate or execute a bank generation function (described below) to arbitrate requests for access to the tiled memory of the computing tile 101. For example, the controller 103 may utilize a bank generation function to arbitrate two or more requests such that each of the two or more requests is processed to a different physical bank of the tiled memory. The controller 103 may utilize a predetermined bank selection scheme programmed or encoded in the controller 103 before an inference decision is made in the computing tile 101.

[0029] In some implementations, a given computational operation involves multiple requestors / accessors, each requesting access to resources in memory 108. For example, a computational workload executing on a computing tile 101 may invoke memory access requests through tensor scan operations that require read and write access to individual address locations in memory 108. As described below, these address locations may correspond to elements of an input tensor that is processed as part of the workload.

[0030] In addition to tensor read / write operations, processing a workload may also include i) moving data (e.g., parameters) from memory 108 (narrow) to memory 110 (wide), and ii) processing read or write access requests to move data from memory 110 (wide) to memory 108 (narrow). In some cases, to process the exemplary workload, the first computational tile 101 arbitrates and executes access requests (e.g., read / write requests) for memory 108, where the requests are based on external data communications originating outside of the first computational tile 101.

[0031] These different types of access requests correspond to one or more computational threads that all require access to the same physical memory 108. In some cases, multiple requests may correspond to a single computational thread (or clock cycle). The disclosed bank generation function can be used to handle multiple requests for memory resources of a tiled memory without causing bank conflicts. For example, the bank generation function is configured to generate an individual bank ID for each request in a group of requests, such that no two requests require access to the same physical memory bank of a tiled memory.

[0032] In some implementations, based on the bank identifiers returned as an output of the bank generation function, the controller 103 can arbitrate requests by processing two or more requests to different physical memory banks of a tiled memory during the same clock cycle (e.g., a single clock cycle). The bank generation function also enables the controller 103 to achieve these benefits for striding for certain problems, such as memory accesses where the stride is a function of the number of banks and the number of bytes in a row of the bank, as will be described in more detail below.

[0033] Additionally, based on the opcode, the controller 103 may activate dedicated datapath logic associated with one or more computational cells 114a / b / c to perform a computation using the parameters and inputs / activations and a corresponding mapping vector used to map the inputs / activations to a subset of the parameters (e.g., a sparse computation). As used in this document, a sparse computation includes a neural network computation performed for a neural network layer using non-zero weight values ​​in a set of compressed sparse parameters generated from the set of weights for the neural network layer.

[0034] In some implementations, the opcode indicates details about operations related to the inputs and weights for a given layer, such as the sparsity of one or more parameter tensors associated with that layer. The controller 103 i) detects the opcode, including any relevant tensor sparsity information, ii) retrieves parameters from a tile memory (e.g., memory 108 or 110) using local read logic based on the opcode, and iii) wires or routes those parameters to cells 114a / b / c of ​​the computational tile 101. The controller 103 can also analyze the exemplary data stream and, based on that analysis, generate a set of compressed sparse parameters and corresponding mapping vectors that map discrete entries in the input vector to individual non-zero weight values ​​in the compressed sparse parameters. To the extent that operations and / or processes for generating the compressed sparse parameters and corresponding mapping vectors are described with reference to the controller 103, each of these operations and processes can also be performed by the host 120, the controller 125, or both.

[0035] In some implementations, some (or all) of the operations, such as analyzing tensor indices, performing direct memory access (DMA) operations to read address spaces in system memory (e.g., SRAM, DRAM, etc.) to obtain input and weight values, generating compressed sparse parameters, and generating corresponding mapping vectors, are performed on host 120, enabling reduced processing time at each computational tile 101 and increased data throughput at system 100. For example, by using controller 125 to perform these operations at host 120, an already-compressed set of parameters can be sent to a given tiled computation 101, thereby reducing the size and amount of data routed through system 100.

[0036] 2 illustrates an exemplary processing pipeline 200 for routing inputs and outputs between memory and computational cells of a hardware integrated circuit. In general, pipeline 200 uses input bus 106 to route inputs obtained from memory locations in memory 108 to one or more computational cells 114, and uses output bus 107 to route outputs generated from multiplications performed in one or more computational cells 114 to memory locations in memory 108.

[0037] The pipeline 200 utilizes a hardware architecture in which an input bus 106 is coupled (e.g., directly coupled) to each of multiple groupings of hardware computational cells of a dedicated integrated circuit. The system 100 may provide a first operand from a location in memory 108 and a second operand from a location in memory 110. The operands 204 are used in a computation performed in the cells 114, which generate an output corresponding to the result of the computation. In some implementations, the computation is for a machine learning operation, such as for processing an input through a neural network layer of an artificial neural network.

[0038] In this implementation, a computational tile 101 may provide a first operand corresponding to an input or activation (e.g., a0, a1, a2, etc.) of an input feature map to a subset of cells 114. For example, individual inputs of an input vector 102 may be provided to each MAC in the subset via the input bus 106 of the computational tile 101. The system 100 may perform this broadcasting operation across multiple computational tiles 101 to compute a product for a given neural network layer at each computational tile 101 using individual groupings of inputs and corresponding weights. At a given computational tile 101, the product is computed by multiplying an individual input (e.g., a1) by a corresponding weight (e.g., w1) at each MAC in the subset using the MAC's multiplication circuitry.

[0039] System 100 can generate a layer output based on an accumulation of multiple individual products computed at each MAC of cells 114a / b / c in a subset of cells 114a / b / c of ​​computation unit 112. As described below with reference to FIG. 6, a multiplication operation performed within computation tile 101 can require i) a first operand (e.g., an input or activation) stored in a memory location in memory 108 corresponding to an individual element of an input tensor, and ii) a second operand (e.g., a weight) stored in a memory location in memory 110 corresponding to an individual element of a parameter tensor.

[0040] 2, shift register 202 may provide a shifting function in which operand 204 inputs are broadcast onto input bus 106 and routed to one or more MACs 114 of cells 114. In some implementations, shift register 202 enables one or more input broadcast modes in computational tile 101. For example, shift register 202 may be used to broadcast inputs from memory 108 sequentially, e.g., one at a time (first broadcast mode), from memory 108 simultaneously, e.g., in parallel (second broadcast mode), or using some combination of these broadcast modes. Shift register 202 may be an integral feature of memory 108 and may be implemented in hardware, software, or both.

[0041] In some implementations, the weight (w3) of the operand 206 may have a weight value of zero. When the controller 103 determines that the weight (w3) has a zero value, the multiplication between the input (a2) and the weight (w3) may be skipped to conserve processing resources, so that the operands are not routed to or consumed by the cells 114a / b / c. The decision to skip that particular multiplication operation may be based on a mapping vector that maps the discrete inputs (a1) of the input vector to the individual weights (wn) of the parameter tensor, as described above.

[0042] 3 shows a scan table 300 illustrating example memory / data scans for different functions 302, 304 used to generate addresses for accessing physical memory banks of memory 108. In particular, a first memory scan is shown for a first function 302, while a second, different memory scan is shown for a second function 304. In some implementations, memory 108 includes one or more memory partitions.

[0043] In the example of FIG. 3, the partition of memory 108 includes 32 physical memory banks 312, shown as banks 0 through 31. Each bank can include a number of rows, e.g., 16, 24, etc. In some implementations, each row is 16 bytes (16B) wide, such that computational tiles 101 access data from memory 108 in 16B chunks (e.g., 128 bits). In some implementations, memory 108 can include more or fewer physical memory banks, and each row of a bank can have a width that is greater than 16B or less than 16B (e.g., 1B).

[0044] The scan table 300 includes a first address 306, a second address 308, and a third address 310. The memory scan distance between each of the first address 306, the second address 308, and the third address 310 can be based on a stride value. In the example of FIG. 3 , the stride value is 512. However, the stride value may be different in other examples. Accordingly, other stride values ​​are within the scope of this disclosure. A stride operation may be required when the computing system 100 executes the computation of a given task or machine learning workload. The stride operation may be based on a stride parameter or stride value. A given stride value may be programmable in the system 100. For example, the system 100 can program the stride for a given inference based on a stride for a particular problem known by a compiler of the system 100 as being specific to the particular inference operation.

[0045] In some implementations, the stride value is determined based on the hardware configuration of memory 108, the type of machine learning operation being performed, or both. For example, the computational tiles 101 of system 100 may be used to implement a neural network (e.g., a convolutional neural network) that is trained or used for compression and / or recognition of image and video content. The stride may be a component of the neural network. In this example, the machine learning operation involves processing an image through a layer of the neural network according to a filter corresponding to a set of weights / parameters for that layer.

[0046] An image, or pixel values ​​associated with an image, may be stored across physical rows and banks of memory 108 with reference to a hardware configuration of the physical rows and banks of that memory. In this example, the stride is a component or parameter of a filter (or kernel) of a neural network. The stride is used to modify the amount that the filter moves across the image or video. For example, if the stride is set to 1, the computational tile 101 moves the filter(s) across the area one pixel (or input) at a time. Similarly, if the stride is 2, the computational tile 101 moves the filter(s) across the area two pixels at a time.

[0047] Thus, the filter may be shifted based on the stride value of the layer, and in some implementations, the system 100 may perform this process repeatedly across multiple computational tiles 101 of different layers until the inputs of different regions of the image have corresponding dot products. Moving the filter across the inputs of the regions of the image based on the stride value may include retrieving, obtaining, or otherwise accessing the inputs from various locations in the memory 108 according to the stride value.

[0048] As described above, an image, or pixel values ​​associated with an image, is stored across physical rows and banks of memory 108. The example of FIG. 3 may be described with reference to a stride value of 512. In some examples, 512 represents a common stride associated with a particular image processing operation. Depending on the type of operation, computational tiles 101 of system 100 may process requests to access memory 108 according to a range of stride values. For example, a first tile 101 may process requests based on a first stride value, while a second, different tile 101 may process requests based on a second, different stride value.

[0049] As shown in scan table 300, when bank generation function 302 is used to generate a bank ID for processing a request, 512 access strides are repeated in the same physical memory bank 314 (e.g., bank 0) of the tiled memory. This indicates a bank conflict. As used herein, a bank conflict occurs when two or more requestors request access to the same physical memory bank of the tiled memory during the same clock cycle.

[0050] 4, bank generation function 304 is configured to process requests to access physical rows within memory banks of memory 108 so that they consistently, or at least substantially, result in access to distinct physical memory banks of memory 108. More specifically, for each request in a group of requests to access memory 108, bank generation function 304 uses bits from an address in the request to generate a bank ID that results in access to different physical memory banks when the group of requests are processed in parallel, e.g., during a single clock cycle.

[0051] 3, the bank generation function 304 allows two or more requests to be processed based on an access pattern that causes the computational tile 101 to access different physical memory banks of the memory 108. The bank generation function 304 is configured to generate bank IDs and row IDs that allow this access pattern regardless of the stride value used for the memory accesses.

[0052] In some implementations, based on the bank ID generated by the bank generation function 304, i) a first request causes the computational tile 101 to access 16B in a row of a first physical memory bank 316, e.g., bank “0”, ii) a second request causes the computational tile 101 to access 16B in a row of a second, different physical memory bank 318, e.g., bank “4”, and iii) a third request causes the computational tile 101 to access 16B in a row of a third, different physical memory bank 320, e.g., bank “8”.

[0053] Each of the first, second, and third requests may be different requests, may be from different request sources, or both. Based on the bank IDs and / or access patterns enabled by the bank generation function 304, there are no, or substantially no, cases where two or more request sources request access to the same physical memory bank of the tiled memory during the same clock cycle. Therefore, each of the first, second, and third requests can be processed simultaneously during the same clock cycle (e.g., a single clock cycle) without creating bank conflicts in the memory 108. Furthermore, each of the first, second, and third requests can be processed simultaneously during the same clock cycle (e.g., a single clock cycle) for a range of stride values ​​(e.g., a stride of 512) without creating bank conflicts.

[0054] In some implementations, the computational tiles 101 are operable to route and store outputs such that the memory 108 (e.g., activation memory) does not encounter bank contention when the stored output values ​​are taken as input activations to a second, different neural network layer. This is described in the following paragraphs with reference to activation values, but applies to other data types / values ​​that may be written to (or stored in) physical locations in the memory 108.

[0055] The computational tile 101 may include a non-linear unit that applies an activation function to accumulated values ​​resulting from the computations performed in the computational units 112. In one example, the non-linear unit may be a hardware circuit included within the multiplication and addition circuits of the computational tile 101. In another example, the non-linear unit is included in the computational tile 101 but external to the computational units 112.

[0056] The nonlinear unit applies its activation function to generate a set of activation values. The activation values ​​may be the output of a machine learning workload. For example, the activation values ​​may be the output of a first neural network layer that is routed to and stored in a memory bank of memory 108. These activation values ​​(e.g., the output of the first layer) may be retrieved from the memory bank of memory 108 and provided as input activations for processing through a second, different neural network layer.

[0057] Computational tiles 101 may issue access requests (e.g., write accesses) to memory 108 to store outputs, such as activation values, in physical rows and physical banks of memory 108. In some examples, these requests may be routed to a partition of memory 108 that is used as activation memory for storing activations. In some implementations, these write access requests may be handled using bank generation function 304, such that system 100 does not encounter bank conflicts at a particular computational tile 101 when output / activation values ​​are stored in memory 108 of that tile.

[0058] The bank generation function 304 may also be used to reduce or prevent bank contention when an access request (e.g., a read access) is processed to retrieve or fetch an output value, for example, an output value may be fetched and provided as an input activation to a second, different neural network layer.

[0059] 4 illustrates an example bank generation function 304 for reducing memory bank contention in a hardware integrated circuit. As described below, the bank generation function 304 is configured so that requests to access physical rows within a memory bank of memory 108 consistently, or at least substantially, result in access to separate physical memory banks of memory 108.

[0060] For example, given a group of requests to access memory 108, controller 103 of computational tile 101 can derive a set of addresses from the group of requests. In some cases, controller 103 may derive a set of contiguous addresses (e.g., virtual / logical addresses) from the group of requests. Given this set of contiguous addresses, bank generation function 304 is configured to generate a corresponding set of bank identifiers ("bank IDs"). For example, bank generation function 304 generates an individual bank ID for each request in the group of requests.

[0061] The system 100, or computational tile 101, can process a group of requests using a set of bank IDs over one or more clock cycles, or over the same clock cycle, without incurring bank contention in the memory 108. In other words, based on the set of bank IDs generated by the bank generation function 304, the computational tile 101 can process each request in the group of requests simultaneously (e.g., in parallel), such that no two requests require access to the same physical memory bank of the tiled memory 108 during the same clock cycle.

[0062] 4, the bank generation function 304 generates an individual bank ID for each request based on an algorithm 402. The algorithm 402 generates the bank ID based on an example sequence of address bits 404 included in the corresponding request to access the memory 108. For example, the computational tile 101 uses the sequence of address bits 404 to obtain individual values ​​for one or more variables provided as input to the algorithm 402. The input variables can correspond to different portions of the sequence of address bits 404.

[0063] A first variable A may be obtained from a first portion of bits, a second variable B may be obtained from a second portion of bits, a third variable row_id may be obtained from a third portion of bits, and a fourth variable byte_in_row may be obtained from a fourth portion of bits. For example, the sequence of bits 404 may be an input address that is divided into two parts: i) bytes_in_row (the number of bytes in a memory row) and ii) row_address. Using the bank generation function 304, the row_address may be divided into two parts: i) row_id (e.g., row_id within each bank) and ii) a conventional bank ID defined as variable A. The least significant bit (LSB) of row_id may be defined as variable B.

[0064] 4, for a given bit sequence 404, one or more bits in one portion of the sequence 404 may overlap with one or more bits in other portions of the sequence. In some implementations, the input variable may correspond to the least significant bit (lsb) in the request or the most significant bit (msb) in the request.

[0065] In some implementations, bank generation function 304 is configured to perform at least a first set of operations 412, a second set of operations 414, and a third set of operations 416. The first and / or second sets of operations can be used to establish specific input variables, such as row_address, row_id, A, and B. In some cases, row_address is used to extract a specific bit in sequence 404, such as bits 4 through 15, while row_id is used to extract a specific other bit, such as bits 8 through 15. In some implementations, these variables can be used to extract other ranges or combinations of bits.

[0066] A third operation (or set of operations) 416 may be used to rotate the bit vector based on a shift parameter or operation, such as a rotation_banking_shift. The bit vector may be derived from a sequence of address bits in a request to access memory 108 based on an operation performed using bank generation function 304. The bit vector may be derived from an msb in the sequence of address bits or an lsb in the sequence of address bits. For example, the bit vector may be a combination of variable A and variable B, and bank generation function 304 is configured to apply a rotation_banking_shift operation to this combination of variables.

[0067] The rotation_banking_shift operation can be applied with reference to the bank number in the memory of the computational tile 101. For example, the rotation_banking_shift operation can be used to i) shift the msb bit vector by the amount specified by the shift value of the operation, or ii) shift the lsb bit vector by the amount specified by the shift value of the operation. The system 100 performs the shift operation so that the bank ID of the new bank belongs to a specific group. Thus, the third operation 416 can obtain the rotated bank ID according to the bank number and any row, bank, and byte attributes associated with variables A and / or B.

[0068] In some implementations, system 100 determines the minimum number of bits to define an msb or lsb. For example, system 100 may determine the minimum number of bits based on the amount of physical memory banks in a given partition of memory 108. For example, if there are 32 physical memory banks, system 100 may determine that a minimum number of 5 bits is required to represent 32 numbers or 32 row IDs.

[0069] In some implementations, the bank generation function 304 is configured or coded as shown in example algorithm 402 of Figure 4. In some other implementations, a modified bank generation function can be configured or coded (using substitution commands) to generate a set of bank IDs that can be simultaneously processed by the computational tile 101 without causing bank conflicts in memory 108. For example, this modified bank generation function can be configured or coded for a tile memory that includes more or fewer physical memory banks and more or fewer rows compared to memory 108.

[0070] The bank generation function 304 may be an algorithm that receives one or more inputs and generates one or more outputs. For example, as described above, the inputs may be individual addresses for one or more rows ("row_address"), the number of rows in each bank ("rows_per_bank"), the least significant bit (lsb) in the request, a minimum amount of lsb bits, the most significant bit (msb) in the request, a minimum amount of msb bits, a bank rotation parameter, or a shift parameter ("msb_shift_minus_1"). The outputs may be individual bank IDs of physical memory banks of the memory 108 and individual row IDs of physical rows within the physical memory banks of the memory 108. More or fewer inputs are within the scope of this disclosure and may be used in connection with the bank generation function 304.

[0071] 5 is an exemplary process 500 for reducing memory bank contention in a tile memory of a hardware accelerator or dedicated hardware integrated circuit. In some implementations, process 500 is performed during computation for a neural network machine learning model implemented on the hardware accelerator. For example, the computation is performed to process neural network inputs, such as images or audio utterances, using a dedicated neural network processor. Such a processor may be represented by the integrated circuit or system 100 described with reference to FIG. 1A.

[0072] For example, the hardware integrated circuit may be configured to implement a CNN including multiple neural network layers. In some cases, the neural network layers may include group convolutional layers. The input may be an exemplary image, as described above, including various other types of digital images or related graphical data. In at least one example, the integrated circuit may implement an RNN for processing input derived from speech or other audio content. In some cases, process 500 is part of a technology that enables improved latency and throughput when accelerating neural network computations to generate image or audio processing output compared to other data processing techniques.

[0073] Process 500 can be implemented or performed using system 100, as described above. Accordingly, the description of process 500 may refer to the aforementioned computing resources of system 100. In some examples, the steps or actions of process 500 are enabled by programmed firmware instructions, software instructions, or both. Each type of instruction may be stored in a non-transitory machine-readable storage device and is executable by one or more processors of the devices and resources described herein.

[0074] In some implementations, the steps of process 500 are performed in a hardware circuit to generate layer outputs of the neural network layers. The outputs can be part of a computation for a machine learning task or an inference workload to generate image processing or image recognition outputs. As indicated above, the integrated circuit can be a dedicated neural network processor or a hardware machine learning accelerator configured to accelerate computations to generate various types of data processing outputs.

[0075] Referring again to process 500, system 100 receives 502 multiple requests to access memory resources of the system. For example, each request of the multiple requests may be used to access a first tile memory of an integrated circuit or hardware accelerator. In some implementations, system 100 processes multiple requests across multiple tiles. For example, system 100 may process a separate subset of the requests in each computational tile 101 of the integrated circuit.

[0076] For each request, the system 100 may identify within the request an individual logical address represented by a sequence of bits (504). As described above, the memory 108 may be row-addressable, such that a row of a memory bank can be identified or accessed based on an address in the corresponding request. Generally, each request specifies a logical address that is mappable to a corresponding physical address. For example, a compiler for the system 100 may determine a mapping of a set of logical addresses to a corresponding set of physical addresses, where the physical addresses specify a physical location within the memory, such as the location of a bank within the first memory 108 or the location of a row within a bank within the first memory 108.

[0077] For each request, the system 100 may obtain 506 a first subset of bits from the sequence of bits. For example, a logical address may be specified by a data structure including a sequence of bits, e.g., 8 bits, 16 bits, etc. In some implementations, the controller 103 of the computational tile 101 scans each data structure representing the logical address specified by the request. In response to scanning the bits forming the data structure, the controller identifies or determines a subset of bits that are usable as input to a bank generation function. In some implementations, the controller determines the subset of bits based on the configuration of the tile memory 108.

[0078] For each request, the system 100 may generate 508 an individual bank identifier (“bank ID”) based on the bank generation function 304 using the first subset of bits. Each individual bank ID identifies a specific physical bank among multiple physical memory banks of the memory 108. As described above, each physical memory bank may include multiple rows. For each request, the computational tile 101 may i) obtain a second subset of bits from the sequence of bits, ii) provide the second subset of bits as input to the bank generation function, and iii) generate an individual row ID that identifies a specific row within a specific bank among the physical memory banks of the tiled memory based on the bank generation function being applied to the second subset of bits.

[0079] The system 100 is configured to process each request using the individual bank ID generated for the request (510). In some implementations, the computational tile 101 processes multiple requests (e.g., simultaneously) and, in response to processing the requests, accesses two or more physical memory banks in parallel without generating bank conflicts (512). For example, the controller 103 of the computational tile 101 generates control signals to access physical banks or rows of the memory 108 to retrieve the input vector 102 from an address location in the first memory 108.

[0080] In some implementations, a computational tile 101 may simultaneously access multiple separate physical memory banks of the tile memory during a single clock cycle in response to processing each of a request to retrieve one or more input vectors. The input vectors 102 may correspond to an input feature map of an image, which may be a matrix structure of neural network inputs, such as activations generated by a previous neural network layer.

[0081] As described above, the bank generation function 304 allows the computational tile 101 to process two or more requests based on an access pattern that causes the computational tile 101 to access different physical memory banks of the memory 108. The bank generation function 304 is configured to generate bank IDs and row IDs that allow this access pattern regardless of the stride value used for the memory access.

[0082] Notably, the bank generation function can accommodate this access pattern even when the stride of the memory access is equal to the difference between specific rows of the same physical memory bank. For example, each row of a physical memory bank can have a width of 16 bytes, and a partition of tiled memory can have 32 physical memory banks. In this example, the bank generation function assumes that the memory access stride is equal to the row_width * This access pattern can also be accommodated when stride is equal to num_banks (i.e., stride=512).

[0083] Figure 6 shows an example of a tensor or multidimensional matrix 600 that includes an input tensor 604, a transformation of a parameter tensor 606, and an output tensor 608. In the example of Figure 6, each of the tensors 600 includes individual elements, where each element can correspond to an individual data value (or operand) for a computation performed at a given layer of a neural network.

[0084] For example, each input of the input tensor 604 may correspond to an individual element along a given dimension of the input tensor 604, each weight of the parameter tensor 606 may correspond to an individual element along a given dimension of the parameter tensor 606, and each output value or activation in the set of outputs may correspond to an individual element along a given dimension of the output tensor 608. Relatedly, each element may correspond to an individual memory location or address in the memory of the computational tile 101 that is assigned to operate on one or more dimensions of the given tensor 604, 606, 608.

[0085] The computations performed at a given neural network layer may include multiplying an input / activation tensor 604 by a parameter / weight tensor 606 over one or more processor clock cycles to obtain a layer output, which may include an output activation. Multiplying an activation tensor 604 by a weight tensor 606 includes multiplying activations from elements of tensor 604 by weights from elements of tensor 606 to obtain one or more partial sums. The example tensors 606 in FIG. 6 may be unmodified parameter tensors, modified parameter tensors, or a combination thereof. In some implementations, each parameter tensor 606 corresponds to a modified parameter tensor that includes non-zero CSP values ​​derived based on a particular sparsity exploitation technique.

[0086] The processor cores of the system 100 can operate on i) scalars corresponding to discrete elements in a multidimensional tensor 604, 606, ii) vectors of values ​​(e.g., input vector 102) that include multiple discrete elements 609 along the same or different dimensions of a multidimensional tensor 604, 606, or iii) combinations thereof. A discrete element 609, or each of multiple discrete elements 609 in a multidimensional tensor, can be represented using X, Y coordinates (2D) or X, Y, Z coordinates (3D), depending on the dimensionality of the tensor.

[0087] The system 100 can calculate multiple partial sums corresponding to products generated from multiplication of the batch inputs with corresponding weight values. As described above, the system 100 can perform the accumulation of products (e.g., partial sums) over many clock cycles. For example, the product accumulation can be performed in random access memory, shared memory, or scratchpad memory of one or more computational tiles based on the techniques described herein. In some implementations, the input weight multiplication can be written as a sum of products of discrete inputs of the input vector 102 multiplied by each weight element, such as a row or slice of the input tensor 604. The row or slice can represent a given dimension, such as a first dimension 610 of the input tensor 604 or a second, different dimension 615 of the input tensor 604.

[0088] In some implementations, a set of exemplary calculations can be used to calculate the output of a convolutional neural network layer. The calculation of a CNN layer can include performing a 2D spatial convolution between a 3D input tensor 604 and at least one 3D filter (weight tensor 606). For example, one 3D filter 606 can be convolved onto the 3D input tensor 604 to obtain a 2D spatial plane 620 or 625. The calculation can include calculating a sum of dot products for a particular dimension of an input volume that includes the input vector 102.

[0089] For example, spatial plane 620 may contain output values ​​of sum-of-products computed from inputs along dimension 610, while spatial plane 625 may contain output values ​​of sum-of-products computed from inputs along dimension 615. The computations to generate the sum-of-products of the output values ​​in each of spatial planes 620 and 625 may be performed i) in computation cells 114a / b / c, ii) directly in memory 110 using arithmetic units coupled to a shared bank of memory 110, or iii) both. In some implementations, various techniques for reducing accumulated values ​​may be used to streamline the reduction operation and perform it directly in memory cells (or locations) of memory 110.

[0090] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware containing the structures disclosed herein and structural equivalents thereof, or in one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by, or to control the operation of, a data processing apparatus.

[0091] Alternatively, or additionally, the program instructions may be encoded in an artificially generated propagated signal, such as a mechanically generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiving device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof.

[0092] The term "computing system" encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, computer, or multiple processors or computers. An apparatus can include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus can also include code that establishes an execution environment for a subject computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0093] A computer program (which may also be called or described as a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted, or declarative or procedural languages, and can be deployed in any form, such as as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0094] A computer program may, but need not, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple cooperating files, e.g., files that store one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communications network.

[0095] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array), or an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit).

[0096] A computer suitable for executing a computer program can include, and by way of example, can be based on, a general-purpose or special-purpose microprocessor or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory or a random-access memory, or both. Elements of a computer include a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to receive data from or transmit data to them, or both. However, a computer need not have such devices. Furthermore, a computer can be incorporated into other devices, such as a mobile phone, a personal digital assistant (PDA), a portable audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0097] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0098] To provide for interaction with a user, embodiments of the subject matter described herein can be implemented in a computer having a display device, such as an LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, that allows the user to provide input to the computer. Other types of devices can also be used to provide for interaction with a user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, such as acoustic input, voice input, or tactile input. Additionally, a computer can interact with a user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0099] Embodiments of the subject matter described herein can be implemented in a computing system including a back-end component, e.g., a data server, or a computing system including a middleware component, e.g., an application server, or a front-end component, e.g., a client computer having a graphical user interface or web browser to allow a user to interact with an embodiment of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks ("LANs") and wide area networks ("WANs"), e.g., the Internet.

[0100] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0101] While this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or what may be claimed, but rather as descriptions of features that may be unique to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as functioning in a particular combination and may initially be claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0102] Similarly, while operations are shown in the figures in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown or in sequential order, or that all of the illustrated operations be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated into a single software product or packaged into multiple software products.

[0103] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. 1. A computer-implemented method for simultaneously accessing memory banks of a hardware accelerator, comprising: receiving a plurality of requests, each used to access a tiled memory of the hardware accelerator; For each of the plurality of requests: identifying within said request a respective logical address represented by a sequence of bits; obtaining a first subset of bits from the sequence of bits; generating an individual bank identifier that identifies a particular bank from among a plurality of physical memory banks of the tiled memory based on a bank generation function that uses the first subset of bits, the method further comprising: processing each of the plurality of requests using the individual bank identifier generated for the request; and simultaneously accessing multiple separate physical memory banks of the tiled memory during a clock cycle in response to processing each of the multiple requests.

2. Simultaneously accessing the plurality of separate physical memory banks comprises:

2. The method of claim 1, comprising accessing the plurality of separate physical memory banks during a single clock cycle for a particular stride value.

3. 3. The method of claim 2, wherein the particular stride value is a memory access stride equal to the difference between particular rows of the same physical memory bank.

4. Simultaneously accessing the plurality of separate physical memory banks comprises:

4. The method of claim 3, further comprising accessing the plurality of separate physical memory banks without bank conflicts.

5. 5. The method of claim 4, wherein a bank conflict occurs when two or more requestors request access to the same physical memory bank of the tiled memory during the same clock cycle.

6. Each physical memory bank includes a plurality of rows, and the method comprises: For each of the plurality of requests: obtaining a second subset of bits from the sequence of bits; providing the second subset of bits as an input to the bank generation function; and generating an individual row identifier that identifies a particular row within the particular bank among the plurality of physical memory banks of the tiled memory based on the bank generation function being applied to the second subset of bits.

7. each row of said plurality of rows comprises a width of 16 bytes; the tiled memory partition includes 32 physical memory banks; The memory access stride is row_width ＊ 7. The method of claim 6, wherein the number of banks is equal to num_banks.

8. Obtaining the first subset of bits from the sequence of bits includes:

2. The method of claim 1, comprising obtaining two or more bits that are among the least significant bits (LSBs) in the sequence of bits.

9. 1. A system comprising: Hardware accelerators and a processing device; a non-transitory machine-readable storage medium for storing instructions, the instructions being executed by the processing device to: receiving a plurality of requests, each used to access a tiled memory of the hardware accelerator; For each of the plurality of requests: identifying within said request a respective logical address represented by a sequence of bits; obtaining a first subset of bits from the sequence of bits; generating an individual bank identifier that identifies a particular bank from among a plurality of physical memory banks of the tiled memory based on a bank generation function that uses the first subset of bits, wherein the instructions are further executable by the processing device to: processing each of the plurality of requests using the individual bank identifier generated for the request; and simultaneously accessing multiple separate physical memory banks of the tiled memory during a single clock cycle in response to processing each of the multiple requests.

10. Simultaneously accessing the plurality of separate physical memory banks comprises:

10. The system of claim 9, further comprising accessing the plurality of separate physical memory banks during a single clock cycle for a particular stride value.

11. 11. The system of claim 10, wherein the particular stride value is a memory access stride equal to the difference between particular rows of the same physical memory bank.

12. Simultaneously accessing the plurality of separate physical memory banks comprises:

12. The system of claim 11, further comprising accessing the plurality of separate physical memory banks without bank conflicts.

13. 13. The system of claim 12, wherein a bank conflict is when two or more requestors request access to the same physical memory bank of the tiled memory during the same clock cycle.

14. Each physical memory bank includes a plurality of rows, and the operation comprises: For each of the plurality of requests: obtaining a second subset of bits from the sequence of bits; providing the second subset of bits as an input to the bank generation function; 14. The system of claim 13, further comprising: generating an individual row identifier that identifies a particular row within the particular bank among the plurality of physical memory banks of the tiled memory based on the bank generation function being applied to the second subset of bits.

15. each row of said plurality of rows comprises a width of 16 bytes; the tiled memory partition includes 32 physical memory banks; The memory access stride is row_width ＊ 15. The system of claim 14, wherein the number of banks is equal to num_banks.

16. Obtaining the first subset of bits from the sequence of bits includes:

10. The system of claim 9, further comprising obtaining two or more bits that are among the least significant bits (LSBs) in the sequence of bits.

17. 1. A non-transitory machine-readable storage medium for storing instructions, the instructions being executed by a processing device of a hardware accelerator to: receiving a plurality of requests, each used to access a tiled memory of the hardware accelerator; For each of the plurality of requests: identifying within said request a respective logical address represented by a sequence of bits; obtaining a first subset of bits from the sequence of bits; generating, by the processing device of the hardware accelerator, an individual bank identifier that identifies a particular bank from among a plurality of physical memory banks of the tiled memory based on a bank generation function that uses the first subset of bits, wherein the instructions are further executable to cause the processing device of the hardware accelerator to perform operations including: processing each of the plurality of requests using the individual bank identifier generated for the request; and simultaneously accessing a plurality of separate physical memory banks of the tiled memory during a single clock cycle in response to processing each of the plurality of requests.

18. Simultaneously accessing the plurality of separate physical memory banks comprises:

20. The non-transitory machine-readable storage medium of claim 17, comprising accessing the multiple separate physical memory banks during a single clock cycle for a particular stride value.

19. 20. The non-transitory machine-readable storage medium of claim 18, wherein the particular stride value is a memory access stride equal to the difference between particular rows of the same physical memory bank.

20. Simultaneously accessing the plurality of separate physical memory banks comprises:

20. The non-transitory machine-readable storage medium of claim 19, comprising accessing the plurality of separate physical memory banks without bank conflicts.

Images