Reducing memory bank collisions in hardware accelerators

Abstract

Methods and systems, including computer readable media, are described for reducing or preventing memory bank collisions in a hardware accelerator to allow concurrent access to memory banks at the hardware accelerator. A computing chiplet of a hardware accelerator receives a request to access chiplet memory of the accelerator. For each of the requests: a logical address represented by a sequence of bits is identified in the request and a first subset of bits is obtained from the sequence. An identifier is generated based on a bank generation function using the first subset of bits. The identifier identifies a particular memory bank among physical memory banks of the chiplet memory. Each request is processed using a respective bank identifier generated for the request. In response to processing the request, a plurality of distinct memory banks are accessed concurrently during the same clock cycle.

Description

Background Art

[0001] This description generally relates to memory operations of hardware integrated circuits.

[0002] A neural network is a machine learning model that uses one or more layers of nodes to generate an output, such as a classification, for a received input. In addition to the output layer, some neural networks also include one or more hidden layers. Some neural networks may be convolutional neural networks (CNNs) configured for image processing or recurrent neural networks (RNNs) configured for speech and language processing. Different types of neural network architectures can be used to perform various tasks related to classification or pattern recognition, prediction involving data modeling, and information clustering.

[0003] A neural network layer may have a corresponding set of parameters or weights. The weights are used to process inputs (e.g., a batch of inputs) through a neural network layer to generate a corresponding output of the layer to compute a neural network inference. A batch of inputs and a set of kernels may be represented as a tensor, i.e., a multidimensional array, of inputs and weights. A hardware accelerator is a dedicated integrated circuit for implementing a neural network. The circuit includes memory whose locations correspond to elements of the tensor that can be traversed or accessed using the control logic of the circuit. Summary of the invention

[0004] This document describes techniques for reducing (or preventing) memory bank conflicts at chiplet (tile) memories of a hardware accelerator to allow concurrent access to these physical memory banks.

[0005] A compute chiplet of a hardware accelerator receives requests for accessing a chiplet memory of the accelerator. For each of the requests: i) identifying a logical address represented by a bit sequence in the request; ii) obtaining a first bit subset from the sequence; and iii) generating an identifier based on a memory bank generation function using the first bit subset.

[0006] The identifier identifies a specific memory bank among the physical memory banks of the chiplet memory. Each request is processed using a corresponding memory bank identifier ("memory bank ID") generated for the request. In response to processing the request, multiple distinct memory banks are accessed concurrently during the same clock cycle (e.g., a single clock cycle).

[0007] One aspect of the subject matter described in this specification can be embodied in a computer-implemented method for concurrently accessing a memory bank of a hardware accelerator. The method includes receiving a plurality of requests, each request being for accessing a chiplet memory of a hardware accelerator. For each of the plurality of requests, the method includes: identifying a corresponding logical address represented by a bit sequence in the request; obtaining a first bit subset from the bit sequence; and generating a corresponding memory bank identifier based on a memory bank generation function using the first bit subset, the corresponding memory bank identifier identifying a specific memory bank among a plurality of physical memory banks of the chiplet memory. The method further includes: i) processing each of the plurality of requests using a corresponding memory bank identifier generated for the request; and ii) concurrently accessing a plurality of distinct physical memory banks of the chiplet memory during a clock cycle in response to processing each of the plurality of requests.

[0008] These and other implementations can each optionally include one or more of the following features. For example, in some implementations, concurrently accessing multiple distinct physical memory banks includes accessing multiple distinct physical memory banks for a particular stride value during a single clock cycle. The particular stride value can be a memory access stride equal to a difference between particular rows of the same physical memory bank.

[0009] In some implementations, concurrently accessing multiple distinct physical memory banks includes: accessing multiple distinct physical memory banks without memory bank conflicts. In one aspect, a memory bank conflict is two or more requestors requesting access to the same physical memory bank of a chiplet memory during the same clock cycle. Each physical memory bank may include multiple rows, and the method includes: for each request in the multiple requests: i) obtaining a second subset of bits from the bit sequence; ii) providing the second subset of bits as input to a memory bank generation function; and iii) based on applying the memory bank generation function to the second subset of bits, generating a corresponding row identifier, the corresponding row identifier identifying a specific row in a specific memory bank among the multiple physical memory banks of the chiplet memory.

[0010] In some implementations, each row in the plurality of rows comprises a width of 16 bytes; the partitioning of the chiplet memory comprises 32 physical memory banks; and the memory access stride is equal to: row_width (row width) * num_banks (number of memory banks). Obtaining the first bit subset from the bit sequence may comprise: obtaining two or more bits of the least significant bit (LSB) in the bit sequence.

[0011] Other implementations of this and other aspects include corresponding systems, devices, and computer programs that are configured to perform the actions of these methods encoded on computer storage devices. The system of one or more computers may be so configured due to software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform actions during operation. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the device to perform actions.

[0012] The subject matter described in this specification can be implemented in certain embodiments to achieve one or more of the following advantages: A bank generation function is disclosed that can be used to handle multiple requests for memory resources of a chiplet memory without triggering bank conflicts.

[0013] The bank generation function is configured to generate a corresponding bank ID for each request in the request group such that no two requests will need to access the same physical memory bank of the chiplet memory. The bank generation function provides an access pattern that allows parallel processing of each request in the group during the same clock cycle (e.g., a single clock cycle). The access pattern provided by the bank generation function allows concurrent processing of multiple requests regardless of the stride value used for memory access.

[0014] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other potential features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1A is a block diagram of an example computing system for implementing a neural network machine learning model.

[0016] Figure 1B is a block diagram of an example computing system for implementing a neural network machine learning model.

[0017] Figure 2 An example processing pipeline for routing inputs and outputs between memory and compute cores of a hardware integrated circuit is shown.

[0018] Figure 3 Examples of memory traversal of different bank generation functions are shown.

[0019] Figure 4 An example memory bank generation function for reducing memory bank conflicts at a hardware integrated circuit is shown.

[0020] Figure 5 is an example process for reducing memory bank conflicts in a hardware integrated circuit.

[0021] Figure 6 Examples of input tensors, parameter tensors, and output tensors are illustrated.

[0022] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0023] Figure 1A 1 is a block diagram of an example computing system 100 for implementing a neural network model at a hardware integrated circuit such as a machine learning hardware accelerator. The computing system 100 includes one or more computing chiplets 101, a host 120, and a higher-level controller 125 ("controller 125"). As described in more detail below, the host 120 and the controller 125 cooperate to provide data sets and instructions to one or more computing chiplets 101 of the system 100.

[0024] In some implementations, the host 120 and the controller 125 are the same device. The host 120 and the controller 125 may also perform distinct functions but be integrated in a single device package. For example, the host 120 and the controller 125 may form a central processing unit (CPU) that interacts or collaborates with a hardware accelerator including a plurality of computing chiplets 101. In some implementations, the host 120, the controller 125, and the plurality of computing chiplets 101 are included or formed on a single integrated circuit die. For example, the host 120, the controller 125, and the plurality of computing chiplets 101 may form a dedicated system on chip (SoC) that is optimized for executing neural network models for processing machine learning workloads.

[0025] Each compute chiplet 101 typically includes a controller 103 that provides one or more control signals 105 to cause the input (or activation) of the input vector 102 to be stored at or accessed from a memory location of a first memory 108 ("memory 108"). Similarly, the controller 103 may also provide one or more control signals 105 to cause the weights (or parameters) of the matrix structure of weights 104 to be stored at or accessed from a memory location of a second memory 110 ("memory 110"). In some implementations, the input vector 102 is obtained from an input tensor, and the matrix structure of the weights is obtained from a parameter tensor. Each of the input tensor and the parameter tensor can be a multi-dimensional data structure, such as a multi-dimensional matrix or tensor. This is described below with reference to Figure 6 Describe in more detail.

[0026] Each memory location of the memories 108, 110 may be identified by a corresponding memory address, such as a logical address having a corresponding mapping to a physical row of a physical memory bank of the memory. Figure 1B For example, the computing chiplet 101 can derive a set of consecutive addresses (e.g., virtual / logical addresses) based on the request group 130. For example, the set of consecutive addresses can be derived with reference to a logical memory corresponding to the physical memory 108 of the computing chiplet 101. This will also be referred to below. Figure 4 and Figure 5 The embodiments are described below.

[0027] The logical memory has multiple logical ports 135, and each port can be connected to or associated with a different requestor requesting access to the physical resources of the memory 108. To process a given access request, for each port, the chiplet 101 (or its controller 103) determines the memory bank to which the request will be routed based on the address in the request. For each memory bank, the computing chiplet 101 may include an arbitrator 140 that arbitrates access to the memory bank from multiple ports, for example, based on a memory bank generation function that is uniquely configured to mitigate some (or all) of the requests from being routed to the same physical memory bank. The logical memory, the ports of the logical memory, and the arbitrator can be implemented in software, hardware, or both. In some implementations, the logical memory, its ports, and the arbitrator are controlled based on control signals generated by the controller 103.

[0028] Each of memories 108, 110 may be implemented as a series of physical storage bodies, units, or any other related storage medium or device. Each of memories 108, 110 may include one or more registers, buffers, or both. In some implementations, memory 108 is an input / activation memory and memory 110 is a parameter memory. In some other implementations, inputs or activations are stored at memory 108, memory 110, or both; and weights are stored at memory 110, memory 108, or both. For example, inputs and weights may be transferred between memory 108 and memory 110 to facilitate certain neural network calculations. In some implementations, each of memory 108 and memory 110 is referred to as a chiplet memory.

[0029] Each compute chiplet 101 also includes an input activation bus 106, an output activation bus 107, and a compute unit 112 that includes one or more hardware multiply accumulate circuits (MACs) in each cell 114 a / b / c. The controller 103 can generate a control signal 105 to obtain operands stored at a memory of the compute chiplet 101. For example, the controller 103 can generate a control signal 105 to obtain: i) an example input vector 102 stored at a memory 108 and ii) a weight 104 stored at a memory 110. Each input obtained from the memory 108 is provided to the input activation bus 106 for routing (e.g., direct routing) to a compute cell 114 a / b / c in the compute unit 112. Similarly, each weight obtained from the memory 110 is routed to a cell 114 a / b / c of ​​the compute unit 112.

[0030] As described below, each core 114a / b / c performs a computation that produces a partial sum or accumulated value to generate an output for a given neural network layer. An activation function may be applied to a set of outputs to generate a set of output activations for a neural network layer. In some implementations, the outputs or output activations are routed via an output activation bus 107 for storage and / or transmission. For example, the set of output activations may be transmitted from a first computing chiplet 101 to a different second computing chiplet 101 for processing at the second computing chiplet 101 as input activations for a different layer of a neural network.

[0031] In general, each compute chiplet 101 and system 100 may include additional hardware structures to perform computations associated with multi-dimensional data structures such as tensors, matrices, and / or data arrays. In some implementations, input vectors (or tensors) 102 inputs and weights 104 of parameter tensors may be preloaded into the memory 108, 110 of the compute chiplet 101. The inputs and weights are received as a set of data values ​​that arrive at a particular compute chiplet 101 from a host 120 (e.g., an external host) via a host interface or from a higher level control such as a controller 125.

[0032] Each of the compute chiplets 101 and controller 103 may include one or more processors, processing devices, and various types of memory. In some implementations, the processors of the compute chiplets 101 and controller 103 include one or more devices, such as a microprocessor or central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a combination of different processors. Each of the compute chiplets 101 and controller 103 may also include other computing and storage resources, such as buffers, registers, control circuit systems, etc. These resources cooperate to provide additional processing options for performing one or more determinations and calculations described in this specification.

[0033] In some implementations, the processing unit of the controller 103 executes programmed instructions stored in the memory to cause the controller 103 and the computing chiplet 101 to perform one or more functions described in this specification. The memory of the controller 103 may include one or more non-transitory machine-readable storage media. The non-transitory machine-readable storage medium may include a solid-state memory, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (e.g., EPROM, EEPROM, or flash memory), or any other tangible medium capable of storing information or instructions.

[0034] The system 100 receives instructions defining a particular computational operation to be performed by the compute chiplet 101. In some implementations, the host may generate a set of parameters (i.e., weights) and corresponding inputs for processing at a neural network layer. For example, the host may generate a set of compressed parameters (CSPs) and corresponding mapping vectors, such as non-zero mappings (NZMs), for a given operation that maps the neural network inputs to non-zero parameters in the CSP set. For example, the host 120 may send the parameters to the compute chiplet 101 via a host interface for further processing at the chiplet. The controller 103 may execute programmed instructions to analyze a data stream associated with the received weights and inputs, including compressed parameters and corresponding mapping vectors.

[0035] The controller 103 causes the inputs and weights of the data stream to be stored at the compute chiplet 101. For example, the controller 103 may store the mapping vectors and the compressed sparse parameters in a local chiplet memory of the compute chiplet 101. This is described in more detail below. The controller 103 may also analyze the input data stream to detect operation codes ("opcodes"). The system 100 may support various types of opcodes, such as opcode types that indicate vector-matrix multiplication operations, element-by-element vector operations, and whether a given operation uses compressed sparse parameters or uncompressed parameters / weights.

[0036] Based on one or more opcodes, the controller 103 may activate or execute a bank generation function (described below) to arbitrate requests for access to chiplet memory of the compute chiplet 101. For example, the controller 103 utilizes the bank generation function to arbitrate two or more requests such that each of the two or more requests is processed for a different physical bank of the chiplet memory. The controller 103 may employ a predetermined bank selection scheme that is programmed or encoded at the controller 103 prior to performing an inference determination at the compute chiplet 101.

[0037] In some implementations, a given compute operation involves multiple requesters / accessors, each of which requires access to resources of memory 108. For example, a compute workload executed at a compute chiplet 101 may trigger memory access requests due to tensor traversal operations that require read and write access to corresponding address locations of memory 108. As described below, these address locations may correspond to elements of input tensors processed as part of the workload.

[0038] In addition to tensor read / write operations, processing the workload may also involve processing read or write access requests to: i) move data (e.g., parameters) from memory 108 (narrow) to memory 110 (wide) and ii) move data from memory 110 (wide) to memory 108 (narrow). In some cases, to process the example workload, the first compute chiplet 101 arbitrates and executes access requests (e.g., read / write requests) to memory 108, where the request is based on external data communications originating from outside the first compute chiplet 101.

[0039] These different types of access requests correspond to one or more computing threads that all need to access the same physical memory 108. In some cases, multiple requests may correspond to a single computing thread (or clock cycle). The disclosed memory bank generation function may be used to handle multiple requests for memory resources of a chiplet memory without triggering a memory bank conflict. For example, the memory bank generation function is configured to generate a corresponding memory bank ID for each request in a request group so that no two requests will need to access the same physical memory bank of the chiplet memory.

[0040] In some implementations, based on the bank identifiers returned as output of the bank generation function, the controller 103 is able to arbitrate requests by processing two or more requests for different physical banks of chiplet memory during the same clock cycle (e.g., a single clock cycle). The bank generation function allows the controller 103 to achieve these advantages even for certain problematic strides, such as memory accesses in which the stride is a function of the number of banks and the number of bytes in a row of the banks. This is described in more detail below.

[0041] Additionally, based on the opcode, the controller 103 may activate dedicated data path logic associated with one or more of the compute cores 114 a / b / c to perform computations (e.g., sparse computations) using the parameters and inputs / activations and corresponding mapping vectors for mapping the inputs / activations to a subset of parameters. As used in this document, sparse computations include neural network computations performed for a neural network layer using non-zero weight values ​​in a compressed sparse parameter set generated from a set of weights for the neural network layer.

[0042] In some implementations, the opcode indicates details about the operation related to the inputs and weights of a given layer, such as the sparsity of one or more parameter tensors associated with the layer. The controller 103: i) detects the opcode including any relevant tensor sparsity information, ii) uses local read logic to obtain the parameters from a chiplet memory (e.g., memory 108 or 110) based on the opcode, and iii) wires or routes the parameters to the cores 114 a / b / c of ​​the compute chiplet 101. The controller 103 may also analyze the example data stream and, based on the analysis, generate a set of compressed sparse parameters and corresponding mapping vectors that map discrete inputs of the input vector to corresponding non-zero weight values ​​in the compressed sparse parameters. To the extent that operations and / or processes for generating compressed sparse parameters and corresponding mapping vectors are described with reference to the controller 103, each of these operations and processes may also be performed by the host 120, the controller 125, or both.

[0043] In some implementations, performing some (or all) of the operations at the host 120, such as analyzing tensor indices, performing direct memory access (DMA) operations to read address space in system memory (e.g., SRAM, DRAM, etc.) to obtain input and weight values, generating compressed sparse parameters, and generating corresponding mapping vectors, allows for reduced processing time at each compute chiplet 101 and increased data throughput at the system 100. For example, performing these operations at the host 120 using the controller 125 allows for sending already compressed sets of parameters to a given chiplet compute 101, which reduces the size and amount of data routed at the system 100.

[0044] Figure 2 An example processing pipeline 200 for routing inputs and outputs between a memory and a computing core of a hardware integrated circuit is shown. In general, the pipeline 200 uses an input bus 106 to route inputs obtained from a memory location of a memory 108 to one or more computing cores 114, and uses an output bus 107 to route outputs generated from multiplications performed at one or more computing cores 114 to a memory location of the memory 108.

[0045] The pipeline 200 utilizes a hardware architecture in which the input bus 106 is coupled (e.g., directly coupled) to a plurality of hardware computation cores in each grouping of the application specific integrated circuits. The system 100 may provide a first operand from a location in the memory 108 and a second operand from a location in the memory 110. The operand 204 is used for a computation performed at the core 114, and the core 114 generates an output corresponding to a result of the computation. In some implementations, the computation is used for a machine learning operation, such as for processing an input through a neural network layer of an artificial neural network.

[0046] In this implementation, the compute chiplet 101 may provide a first operand corresponding to an input or activation (e.g., a0, a1, a2, etc.) of an input feature map to a subset of cores 114. For example, a corresponding input of the input vector 102 is provided to each MAC in the subset via an input bus 106 of the compute chiplet 101. The system 100 may perform this broadcast operation across multiple compute chiplets 101 to compute products for a given neural network layer using corresponding input packets and corresponding weights at each compute chiplet 101. At a given compute chiplet 101, the product is computed by multiplying the corresponding input (e.g., a1) with the corresponding weight (e.g., w1) at each MAC in the subset using the multiplication circuitry of the MAC.

[0047] The system 100 may generate an output of a layer based on an accumulation of multiple corresponding products computed at each MAC of the cores 114a / b / c in the subset of cores 114a / b / c of ​​the computing unit 112. Figure 6 As explained, a multiplication operation performed within the compute chiplet 101 may involve: i) a first operand (e.g., input or activation) corresponding to a corresponding element of an input tensor stored at a memory location of the memory 108; and ii) a second operand (e.g., weight) corresponding to a corresponding element of a parameter tensor stored at a memory location of the memory 110.

[0048] exist Figure 2In the example of , shift register 202 may provide shift functionality in which inputs of operand 204 are broadcast onto input bus 106 and routed to one or more MACs of core cell 114. In some implementations, shift register 202 enables one or more input broadcast modes at compute chiplet 101. For example, shift register 202 may be used to broadcast inputs sequentially, e.g., one by one, from memory 108 (e.g., a first broadcast mode), concurrently, e.g., in parallel, from memory 108 (e.g., a second broadcast mode), or use some combination of these broadcast modes. Shift register 202 may be an integrated function of memory 108 and may be implemented in hardware, software, or both.

[0049] In some implementations, the weight (w3) of operand 206 may have a zero weight value. When the controller 103 determines that the weight (w3) has a zero value, in order to save processing resources, the multiplication between the input (a2) and the weight (w3) may be skipped so that these operands are not routed to or consumed by the core 114 a / b / c. As described above, the determination to skip this particular multiplication operation may be based on a mapping vector that maps discrete inputs (an) of the input vector to individual weights (wn) of the parameter tensor.

[0050] Figure 3 A traversal table 300 is shown that illustrates examples of memory / data traversals for different functions 302, 304 for generating addresses for accessing physical memory banks of the memory 108. Specifically, a first memory traversal is shown for a first function 302, while a second, different memory traversal is shown for a second function 304. In some implementations, the memory 108 includes one or more memory partitions.

[0051] exist Figure 3 In the example of FIG. 1 , the partitioning of the memory 108 includes 32 physical memory banks 312, which are shown as memory banks 0-31. Each memory bank can include a certain number of rows, such as 16, 24, etc. In some implementations, each row is 16 bytes (16B) wide, so that the compute chiplet 101 accesses data from the memory 108 in 16B blocks (e.g., 128 bits). In some implementations, the memory 108 can include more or fewer physical memory banks, and each row of the memory bank can have a width greater than 16B or less than 16B (e.g., 1B).

[0052] The traversal table 300 includes a first address 306, a second address 308, and a third address 310. The memory traversal distance between each of the first address 306, the second address 308, and the third address 310 may be based on a stride value. Figure 3In the example of , the stride value is 512. However, in other examples, the stride value may be different. Therefore, other stride values ​​are within the scope of the present disclosure. When computing system 100 performs calculations for a given task or machine learning workload, stride operations may be required. Stride operations may be based on stride parameters or values. A given stride value may be programmed at system 100. For example, system 100 may program the stride of a given inference based on certain problematic strides known by a compiler of system 100 to be unique to a particular inference operation.

[0053] In some implementations, the stride value is determined based on the hardware configuration of the memory 108, the type of machine learning operation being performed, or both. For example, the computing chiplet 101 of the system 100 may be used to implement a neural network (e.g., a convolutional neural network) that is tuned or used for compression and / or recognition of image and video content. The stride may be an integral part of the neural network. In this example, the machine learning operation involves processing an image through a layer of the neural network according to a filter corresponding to a set of weights / parameters of the layer.

[0054] An image or pixel values ​​associated with an image may be stored across physical rows and banks of memory 108 with reference to the hardware configuration of the physical rows and banks of memory. In this example, the stride is a component or parameter of a filter (or kernel) of a neural network. The stride is used to modify the amount by which the filter moves across an image or video. For example, when the stride is set to 1, the compute chiplet 101 moves the filter across the area one pixel (or input) at a time. Similarly, when the stride is 2, the compute chiplet 101 moves the filter across the area two pixels at a time.

[0055] Thus, the filter may be shifted based on the stride value of the layer, and in some implementations, the system 100 may repeatedly perform this process across multiple compute chiplets 101 at different layers until inputs to different regions of the image have corresponding dot products. Moving the filter across the inputs to a region of the image based on the stride value may include retrieving, obtaining, or otherwise accessing the inputs from various locations in the memory 108 according to the stride value.

[0056] As described above, images or pixel values ​​associated with images are stored across physical rows and banks of memory 108 . Figure 3 An example of can be described with reference to a stride value 512. In some examples, 512 represents a common stride associated with certain image processing operations. Depending on the type of operation, the compute chiplets 101 of the system 100 can process a request to access the memory 108 according to a range of stride values. For example, a first chiplet 101 can process a request based on a first stride value, while a different second chiplet 101 can process a request based on a different second stride value.

[0057] As shown at traversal table 300, when using bank generation function 302 to generate a bank ID for processing a request, access stride 512 is repeated at the same physical memory bank 314 (e.g., bank 0) of the chiplet memory. This indicates a bank conflict. As used in this document, a bank conflict occurs when two or more requesters request access to the same physical memory bank of a chiplet memory during the same clock cycle.

[0058] As referenced below Figure 4 As described, the bank generation function 304 is configured such that processing requests to access physical rows in a bank of the memory 108 consistently or at least substantially results in accessing a distinct physical bank of the memory 108. More specifically, for each request in a request group to access the memory 108, the bank generation function 304 uses bits from the address in the request to generate a bank ID that results in accessing a distinct physical bank when the request group is processed in parallel, for example, during a single clock cycle.

[0059] For example, Figure 3 As shown, the bank generation function 304 allows two or more requests to be processed based on an access pattern that causes the compute chiplet 101 to access different physical memory banks of the memory 108. The bank generation function 304 is configured to generate a bank ID and a row ID that allow the access pattern regardless of the stride value used for the memory access.

[0060] In some implementations, based on the bank ID generated by the bank generation function 304: i) a first request causes the compute chiplet 101 to access 16B in a row of a first physical memory bank 316, e.g., bank “0”; ii) a second request causes the compute chiplet 101 to access 16B in a row of a different second physical memory bank 318, e.g., bank “4”; and iii) a third request causes the compute chiplet 101 to access 16B in a row of a different third physical memory bank 320, e.g., bank “8”.

[0061] Each of the first, second, and third requests may be a different request, may be from a different requestor, or both. Based on the bank IDs and / or access modes enabled by the bank generation function 304, there is no or substantially no situation in which two or more requestors request access to the same physical bank of the chiplet memory during the same clock cycle. Therefore, each of the first, second, and third requests may be processed concurrently during the same clock cycle (e.g., a single clock cycle) without bank conflicts occurring at the memory 108. In addition, each of the first, second, and third requests may be processed concurrently for a range of stride values ​​(e.g., a stride of 512) during the same clock cycle (e.g., a single clock cycle) without bank conflicts occurring.

[0062] In some implementations, the compute chiplet 101 is operable to route and store outputs such that when the stored output values ​​are obtained as input activations of a different second neural network layer, the memory 108 (e.g., activation memory) does not experience bank conflicts. This is described in the following paragraphs with reference to activation values, but is applicable to other data types / values ​​that may be written to (or stored in) a physical location of the memory 108.

[0063] The compute chiplet 101 may include a nonlinear unit that applies an activation function to an accumulated value resulting from a computation performed at the compute unit 112. In one example, the nonlinear unit may be a hardware circuit included in the multiplication and addition circuits of the compute chiplet 101. In another example, the nonlinear unit is included at the compute chiplet 101 but is external to the compute unit 112.

[0064] The nonlinear unit applies its activation function to generate a set of activated values. The activated values ​​may be outputs of a machine learning workload. For example, the activated values ​​may be outputs of a first neural network layer that are routed to and stored at a memory bank of memory 108. These activated values ​​(e.g., outputs of the first layer) may be retrieved from the memory bank of memory 108 and provided as input activations for processing by a different second neural network layer.

[0065] The compute chiplets 101 may issue access requests (e.g., write accesses) to the memory 108 to store outputs, such as activation values, at physical rows and banks of the memory 108. In some examples, these requests may be routed to a partition of the memory 108 that is used as activation memory to store activations. In some implementations, a bank generation function 304 may be used to process these write access requests so that when output / activation values ​​are stored at the memory 108 of a particular compute chiplet 101, the system 100 does not experience bank conflicts at that chiplet.

[0066] The bank generation function 304 can also be used to reduce or prevent bank conflicts when processing access requests (e.g., read accesses) to retrieve or obtain output values. For example, the output values ​​can be obtained to be provided as input activations to a different second neural network layer.

[0067] Figure 4 An example bank generation function 304 is shown for reducing bank conflicts at a hardware integrated circuit. As described below, the bank generation function 304 is configured such that requests for accessing a physical row in a bank of memory 108 consistently or at least substantially result in accessing a distinct physical bank of memory 108.

[0068] For example, given a request group to access memory 108, controller 103 of compute chiplet 101 may derive a set of addresses from the request group. In some cases, controller 103 may derive a set of consecutive addresses (e.g., virtual / logical addresses) from the request group. Given the set of consecutive addresses, bank generation function 304 is configured to generate a corresponding set of bank identifiers ("bank IDs"). For example, bank generation function 304 generates a corresponding bank ID for each request in the request group.

[0069] The system 100 or the compute chiplet 101 may process the request group in one or more clock cycles or in the same clock cycle using the set of bank IDs without bank conflicts occurring at the memory 108. In other words, based on the set of bank IDs generated by the bank generation function 304, the compute chiplet 101 may process each request in the request group concurrently (e.g., in parallel), and no two requests will need to access the same physical memory bank of the chiplet memory 108 during the same clock cycle.

[0070] exist Figure 4In the example of , the bank generation function 304 generates a corresponding bank ID for each request based on the algorithm 402. The algorithm 402 generates the bank ID based on the example address bit sequence 404 included in the corresponding request to access the memory 108. For example, the compute chiplet 101 uses the address bit sequence 404 to obtain corresponding values ​​of one or more variables provided as input to the algorithm 402. The input variables can correspond to different parts of the address bit sequence 404.

[0071] A first variable A may be obtained from the first portion of bits, a second variable B may be obtained from the second portion of bits, a third variable row_id may be obtained from the third portion of bits, and a fourth variable byte_in_row may be obtained from the fourth portion of bits. For example, the bit sequence 404 may be an input address divided into two parts: i) bytes_in_row (the number of bytes in a memory row) and ii) row_address. Using the bank generation function 304, row_address may be divided into two parts: i) row_id (e.g., the row_id in each memory bank) and ii) a regular memory bank id defined as variable A. The least significant bit (LSB) of row_id may be defined as variable B.

[0072] like Figure 4 As indicated in the example of , for a given bit sequence 404, one or more bits in a portion of the sequence 404 may overlap with one or more bits in another portion of the sequence. In some implementations, the input variable may correspond to the least significant bit (lsb) in the request or the most significant bit (msb) in the request.

[0073] In some implementations, the bank generation function 304 is configured to perform at least a first set of operations 412, a second set of operations 414, and a third set of operations 416. The first and / or second set of operations may be used to establish certain input variables, such as row_address, row_id, A, and B. In some cases, row_address is used to extract certain bits, such as bits 4-15, in the sequence 404, while row_id is used to extract certain other bits, such as bits 8-15. In some implementations, the variables may be used to extract other ranges of bits or combinations of bits.

[0074] A third operation (or set of operations) 416 may be used to rotate a bit vector based on a shift parameter or operation such as rotation_banking_shift. The bit vector may be derived from an address bit sequence in a request to access the memory 108 based on operations performed using the bank generation function 304. The bit vector may be derived from the msb in the address bit sequence or the lsb in the address bit sequence. For example, the bit vector may be a combination of variables A and B, and the bank generation function 304 is configured to apply a rotation_banking_shift operation to the combination of variables.

[0075] The rotation_banking_shift operation may be applied with reference to a plurality of banks in the memory of the compute chiplet 101. For example, the rotation_banking_shift operation may be used to: i) shift the msb bit vector by an amount specified by the shift value of the operation; or ii) shift the lsb bit vector by an amount specified by the shift value of the operation. The system 100 performs the shift operation so that the bank ID of the new bank will belong to a particular group. Thus, the third operation 416 may generate a rotated bank ID based on the number of banks and any row, bank, and byte attributes associated with the variables A and / or B.

[0076] In some implementations, the system 100 determines the minimum number of bits used to define the MSB or LSB. For example, the system 100 may determine the minimum number of bits based on the number of physical memory banks in a given partition of the memory 108. For example, if there are 32 physical memory banks, the system 100 may determine that a minimum number of 5 bits is required to represent 32 numbers or 32 row IDs.

[0077] In some implementations, such as Figure 4 The bank generation function 304 may be configured or encoded as shown in the algorithm 402 in the example of . In some other implementations, the modified bank generation function may be configured or encoded (using alternative commands) to generate a set of bank IDs that can be processed concurrently at the compute chiplet 101 without triggering bank conflicts at the memory 108. For example, the modified bank generation function may be configured or encoded for a chiplet memory that includes more or fewer physical memory banks and more or fewer rows relative to the memory 108.

[0078] The bank generation function 304 may be an algorithm that receives one or more inputs and generates one or more outputs. For example, as described above, the inputs may be the corresponding addresses of one or more rows (“row_address”), the number of rows in each bank (“rows_per_bank”), the least significant bit (lsb) in the request, the minimum number of bits for the lsb, the most significant bit (msb) in the request, the minimum number of bits for the msb, a bank rotation parameter or a shift parameter (“msb_shift_minus_1”). The outputs may be the corresponding bank IDs of the physical banks of the memory 108 and the corresponding row IDs of the physical rows of the physical banks of the memory 108. More or fewer inputs are within the scope of the present disclosure and may be used in conjunction with the bank generation function 304.

[0079] Figure 5 is an example process 500 for reducing memory bank conflicts in a chiplet memory of a hardware accelerator or a dedicated hardware integrated circuit. In some implementations, process 500 is performed during computation of a neural network machine learning model implemented on a hardware accelerator. For example, a dedicated neural network processor may be used to perform computations to process neural network inputs such as images or speech utterances. Such a processor may be provided by reference to Figure 1A The integrated circuit or system 100 is depicted.

[0080] For example, a hardware integrated circuit may be configured to implement a CNN including multiple neural network layers. In some cases, the neural network layers may include grouped convolutional layers. The input may be an example image including various other types of digital images or related graphical data as described above. In at least one example, the integrated circuit may implement an RNN for processing input derived from speech or other audio content. In some cases, process 500 is part of a technique that allows improvements in latency and throughput to be achieved when accelerating neural network computations to generate image or speech processing outputs relative to other data processing techniques.

[0081] The process 500 may be implemented or performed using the system 100 described above. Therefore, the description of the process 500 may refer to the computing resources mentioned above of the system 100. In some examples, the steps or actions of the process 500 are implemented by programmed firmware instructions, software instructions, or both. Each type of instruction may be stored in a non-transitory machine-readable storage device and may be executed by one or more processors of the devices and resources described in this document.

[0082] In some implementations, the steps of process 500 are performed at a hardware circuit to generate a layer output of a neural network layer. The output can be part of a computation that generates an image processing or image recognition output for a machine learning task or inference workload. As described above, the integrated circuit can be a dedicated neural network processor or hardware machine learning accelerator configured to accelerate computations for generating various types of data processing outputs.

[0083] Referring again to process 500, system 100 receives multiple requests to access memory resources of the system (502). For example, each of the multiple requests may be used to access a first chiplet memory of an integrated circuit or a hardware accelerator. In some implementations, system 100 processes the multiple requests across multiple chiplets. For example, system 100 may process a corresponding subset of requests at each compute chiplet 101 of the integrated circuit.

[0084] For each request, the system 100 may identify a corresponding logical address represented by a bit sequence in the request (504). As described above, the memory 108 may be row-addressable such that a row of a memory bank may be identified or accessed based on an address in a corresponding request. In general, each request specifies a logical address that may be mapped to a corresponding physical address. For example, a compiler of the system 100 may determine a mapping of a set of logical addresses to a corresponding set of physical addresses, where the physical address specifies a physical location in a memory, such as a location of a memory bank in the first memory 108 or a location of a row in a memory bank in the first memory 108.

[0085] For each request, the system 100 may obtain a first bit subset from the bit sequence (506). For example, the logical address may be specified by a data structure that includes a bit sequence of, for example, 8 bits, 16 bits, etc. In some implementations, the controller 103 of the compute chiplet 101 scans each data structure representing the logical address specified by the request. In response to scanning the bits forming the data structure, the controller identifies or determines a bit subset that may be used as an input to the memory bank generation function. In some implementations, the controller determines the bit subset based on the configuration of the chiplet memory 108.

[0086] For each request, the system 100 may generate a corresponding memory bank identifier ("memory bank ID") (508) based on the memory bank generation function 304 using the first bit subset. Each corresponding memory bank ID identifies a specific physical memory bank among the multiple physical memory banks of the memory 108. As described above, each physical memory bank may include multiple rows. For each request, the compute chiplet 101 may: i) obtain a second bit subset from the bit sequence; ii) provide the second bit subset as input to the memory bank generation function; and iii) generate a corresponding row ID based on the memory bank generation function applied to the second bit subset, the row ID identifying a specific row in a specific memory bank among the physical memory banks of the chiplet memory.

[0087] The system 100 is configured to process each request using a corresponding memory bank ID generated for the request (510). In some implementations, the compute chiplet 101 processes multiple requests (e.g., concurrently) and, in response to processing the requests, accesses two or more physical memory banks in parallel without memory bank conflicts (512). For example, the controller 103 of the compute chiplet 101 generates control signals to access a physical memory bank or row of the memory 108 to retrieve the input vector 102 from an address location of the first memory 108.

[0088] In some implementations, the compute chiplet 101 may concurrently access multiple distinct physical memory banks of the chiplet memory during a single clock cycle in response to processing each of the requests to retrieve one or more input vectors. The input vector 102 may correspond to an input feature map of an image and may be a matrix structure of a neural network input such as an activation generated by a previous neural network layer.

[0089] As described above, the bank generation function 304 allows two or more requests to be processed based on an access pattern that causes the compute chiplet 101 to access different physical memory banks of the memory 108. The bank generation function 304 is configured to generate a bank ID and row ID that allows this access pattern regardless of the stride value used for the memory access.

[0090] It is worth noting that the bank generation function can provide this access pattern even when the memory access stride is equal to the difference between specific rows of the same physical memory bank. For example, each row of the physical memory bank may have a width of 16 bytes, and the partition of the chiplet memory may have 32 physical memory banks. In this example, the bank generation function can provide this access pattern even when the memory access stride is equal to: row_width * num_banks (i.e., stride = 512).

[0091] Figure 6 An example of a tensor or multi-dimensional matrix 600 is shown including an input tensor 604, a variation of a parameter tensor 606, and an output tensor 608. Figure 6 In the example of , each of tensors 600 includes respective elements, where each element may correspond to a respective data value (or operand) of a computation performed at a given layer of the neural network.

[0092] For example, each input of the input tensor 604 may correspond to a corresponding element along a given dimension of the input tensor 604, each weight of the parameter tensor 606 may correspond to a corresponding element along a given dimension of the parameter tensor 606, and each output value or activation in the output set may correspond to a corresponding element along a given dimension of the output tensor 608. Relatedly, each element may correspond to a corresponding memory location or address in the memory of the compute chiplet 101 assigned to operate on one or more dimensions of the given tensor 604, 606, 608.

[0093] The computations performed at a given neural network layer may include multiplying input / activation tensor 604 with parameter / weight tensor 606 over one or more processor clock cycles to produce a layer output, which may include an output activation. Multiplying activation tensor 604 with weight tensor 606 includes multiplying activations from elements of tensor 604 with weights from elements of tensor 606 to produce one or more partial sums. Figure 6 The example tensors 606 may be unmodified parameter tensors, modified parameter tensors, or a combination thereof. In some implementations, each parameter tensor 606 corresponds to a modified parameter tensor that includes a non-zero CSP value derived based on certain sparsity exploitation techniques.

[0094] The processor core of the system 100 can operate on: i) a scalar corresponding to a discrete element in a multidimensional tensor 604, 606; ii) a vector (e.g., input vector 102) comprising values ​​of multiple discrete elements 609 along the same or different dimensions of a multidimensional tensor 604, 606; or iii) a combination of these. Depending on the dimensionality of the tensor, a discrete element 609 in a multidimensional tensor or each discrete element in the multiple discrete elements 609 can be represented using X, Y coordinates (2D) or using X, Y, Z coordinates (3D).

[0095] The system 100 can calculate multiple partial sums corresponding to the products generated by multiplying a batch of inputs with corresponding weight values. As described above, the system 100 can perform accumulation of products (e.g., partial sums) over multiple clock cycles. For example, the accumulation of products can be performed in a random access memory, a shared memory, or a temporary storage of one or more computing chiplets based on the techniques described in this document. In some implementations, the input-weight multiplication can be written as a sum of products of each weight element multiplied by a discrete input (such as a row or slice of the input tensor 604) of the input vector 102. The row or slice can represent a given dimension, such as a first dimension 610 of the input tensor 604 or a different second dimension 615 of the input tensor 604.

[0096] In some implementations, the output of a convolutional neural network layer may be calculated using an example set of calculations. The calculations for the CNN layer may involve performing a 2D spatial convolution between a 3D input tensor 604 and at least one 3D filter (weight tensor 606). For example, convolution of a 3D filter 606 on the 3D input tensor 604 may produce a 2D spatial plane 620 or 625. The calculations may involve calculating the sum of dot products for a particular dimension of the input volume including the input vector 102.

[0097] For example, spatial plane 620 may include output values ​​of sums of products computed from inputs along dimension 610, while spatial plane 625 may include output values ​​of sums of products computed from inputs along dimension 615. The calculations of sums of products for generating output values ​​in each of spatial planes 620 and 625 may be performed i) at compute cores 114 a / b / c, ii) directly at memory 110 using arithmetic operators coupled to a shared memory bank of memory 110, iii) or both. In some implementations, the reduction operation may be simplified and performed directly at the memory cores (or locations) of memory 110 using various techniques for reduction of accumulated values.

[0098] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier, for execution by a data processing device or to control the operation of a data processing device.

[0099] Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode the information for transmission to an appropriate receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0100] The term "computing system" encompasses all kinds of equipment, devices and machines for processing data, including by way of example a programmable processor, a computer or multiple processors or computers. The equipment may include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the equipment may also include code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0101] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script or code) may be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0102] A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, such as files that store one or more modules, subroutines, or code portions. A computer program may be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0103] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit system, and the device can also be implemented as a dedicated logic circuit system, such as an FPGA (field programmable gate array), an ASIC (application-specific integrated circuit), or a GPGPU (general purpose graphics processing unit).

[0104] Computers suitable for executing computer programs include, by way of example, central processing units that can be based on general or special microprocessors or both, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. Some elements of a computer are a central processing unit for fulfilling or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from one or more mass storage devices or to transfer data to one or more mass storage devices or both. However, a computer does not have to have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0105] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including by way of example semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0106] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as an LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0107] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0108] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0109] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of a separate embodiment may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as acting in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may involve a sub-combination or a variation of the sub-combination.

[0110] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed, to achieve the desired result. In some contexts, multitasking and parallel processing can be advantageous. In addition, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0111] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions set forth in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous.

Claims

1. A computer-implemented method for concurrently accessing a memory bank of a hardware accelerator, the method comprising: receiving a plurality of requests, each of the plurality of requests for accessing a chiplet memory of the hardware accelerator; For each of the multiple requests: identifying in the request a corresponding logical address represented by the bit sequence; obtaining a first bit subset from the bit sequence; as well as generating a corresponding memory bank identifier based on a memory bank generation function using the first subset of bits, the corresponding memory bank identifier identifying a particular memory bank among a plurality of physical memory banks of the chiplet memory; processing each of the plurality of requests using the corresponding memory bank identifier generated for the request; as well as In response to processing each of the plurality of requests, a plurality of distinct physical memory banks of the chiplet memory are concurrently accessed during a clock cycle.

2. The method of claim 1 , wherein concurrently accessing the plurality of distinct physical memory banks comprises: For a particular stride value, the multiple distinct physical memory banks are accessed during a single clock cycle.

3. The method of claim 2, wherein the specific stride value is a memory access stride equal to a difference between specific rows of the same physical memory bank.

4. The method of claim 3, wherein concurrently accessing the plurality of distinct physical memory banks comprises: The plurality of distinct physical memory banks are accessed without bank conflicts.

5. The method of claim 4, wherein a bank conflict is two or more requestors requesting access to the same physical memory bank of the chiplet memory during the same said clock cycle.

6. The method of claim 5, wherein each physical memory bank comprises a plurality of rows, and the method comprises: For each of the multiple requests: obtaining a second subset of bits from the bit sequence; providing the second subset of bits as input to the memory bank generation function; as well as A corresponding row identifier is generated based on applying the bank generation function to the second subset of bits, the corresponding row identifier identifying a particular row in the particular bank among the plurality of physical memory banks of the chiplet memory.

7. The method of claim 6, wherein: Each of the plurality of rows comprises a width of 16 bytes; The chiplet memory is partitioned to include 32 physical memory banks; and The memory access stride is equal to: row_width * num_banks.

8. The method of claim 1, wherein obtaining a first subset from the bit sequence comprises: Two or more bits among the least significant bits (LSBs) in the bit sequence are obtained.

9. A system comprising: Hardware accelerators; Processing device; as well as A non-transitory machine-readable storage medium for storing instructions, the instructions being executable by the processing device to cause operations to be performed, the operations comprising: receiving a plurality of requests, each of the plurality of requests for accessing a chiplet memory of the hardware accelerator; For each of the multiple requests: identifying in the request a corresponding logical address represented by the bit sequence; obtaining a first bit subset from the bit sequence; and generating a corresponding memory bank identifier based on a memory bank generation function using the first subset of bits, the corresponding memory bank identifier identifying a particular memory bank among a plurality of physical memory banks of the chiplet memory; processing each of the plurality of requests using the corresponding memory bank identifier generated for the request; and In response to processing each of the plurality of requests, a plurality of distinct physical memory banks of the chiplet memory are concurrently accessed during a single clock cycle.

10. The system of claim 9, wherein concurrently accessing the plurality of distinct physical memory banks comprises: For a particular stride value, the multiple distinct physical memory banks are accessed during a single clock cycle.

11. The system of claim 10, wherein the specific stride value is a memory access stride equal to a difference between specific rows of the same physical memory bank.

12. The system of claim 11, wherein concurrently accessing the plurality of distinct physical memory banks comprises: The plurality of distinct physical memory banks are accessed without bank conflicts.

13. The system of claim 12, wherein a bank conflict is two or more requestors requesting access to the same physical memory bank of the chiplet memory during the same said clock cycle.

14. The system of claim 13, wherein each physical memory bank comprises a plurality of rows, and the operations further comprise: For each of the multiple requests: obtaining a second subset of bits from the bit sequence; providing the second subset of bits as input to the memory bank generation function; as well as A corresponding row identifier is generated based on applying the bank generation function to the second subset of bits, the corresponding row identifier identifying a particular row in the particular bank among the plurality of physical memory banks of the chiplet memory.

15. The system of claim 14, wherein: Each of the plurality of rows comprises a width of 16 bytes; The chiplet memory is partitioned to include 32 physical memory banks; and The memory access stride is equal to: row_width*num_banks.

16. The system of claim 9, wherein obtaining a first subset of bits from the bit sequence comprises: Two or more bits among the least significant bits (LSBs) in the bit sequence are obtained.

17. A non-transitory machine-readable storage medium for storing instructions, the instructions being executable by a processing device of a hardware accelerator to cause execution of operations comprising: receiving a plurality of requests, each of the plurality of requests for accessing a chiplet memory of the hardware accelerator; For each of the multiple requests: identifying in the request a corresponding logical address represented by the bit sequence; obtaining a first bit subset from the bit sequence; as well as generating a corresponding memory bank identifier based on a memory bank generation function using the first subset of bits, the corresponding memory bank identifier identifying a particular memory bank among a plurality of physical memory banks of the chiplet memory; processing each of the plurality of requests using the corresponding memory bank identifier generated for the request; as well as In response to processing each of the plurality of requests, a plurality of distinct physical memory banks of the chiplet memory are concurrently accessed during a single clock cycle.

18. The non-transitory machine-readable storage medium of claim 17, wherein concurrently accessing the plurality of distinct physical memory banks comprises: For a particular stride value, the multiple distinct physical memory banks are accessed during a single clock cycle.

19. The non-transitory machine-readable storage medium of claim 18, wherein the particular stride value is a memory access stride equal to a difference between particular rows of the same physical memory bank.

20. The non-transitory machine-readable storage medium of claim 19, wherein concurrently accessing the plurality of distinct physical memory banks comprises: The plurality of distinct physical memory banks are accessed without bank conflicts.

Images